Digital human driving method, digital human driving system, device, equipment and medium

Abstract

The invention provides a digital human driving method, a digital human driving system, a device, equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of natural language processing, computer vision, intelligent voice, multi-mode interaction and the like. According to the implementation scheme, real-time input information for the digital human is obtained; inputting the real-time input information into the large model to obtain a first lexical element flow output by the large model in real time; generating a first voice audio stream based on the first lexical element stream; acquiring a first video material stream based on the first lexical element stream; and generating a first video stream for driving the digital person based on the first voice audio stream and the first video material stream.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of natural language processing, computer vision, intelligent speech and multimodal interaction, and specifically to a digital human driving method, digital human driving system, digital human driving device, electronic device, computer-readable storage medium and computer program product. Background Technology

[0002] With the development of large-scale model capabilities, AI digital human capabilities have been applied to various scenarios, including digital human live streaming, digital human videos, digital human employees, and digital human assistants; digital human live streaming enables 24 / 7 live streaming, significantly improving broadcasting efficiency and reducing live streaming costs; digital human videos enable large-scale content production without real people appearing on camera, significantly increasing content production capacity; digital human employees and assistants have achieved the ability to receive customers at any time and solve user problems at any time.

[0003] As the application of digital humans expands, there are more and more business scenarios where people and digital humans can have real-time conversations and communication, including digital human interviews, live video chats, and emotional / legal consultations. Meanwhile, users' demands for services provided by digital humans are also increasing.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a method for driving a digital human, a system for driving a digital human, a device for driving a digital human, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this disclosure, a method for driving a digital human is provided, comprising: acquiring real-time input information for the digital human; inputting the real-time input information into a large model to obtain a first word stream output in real time by the large model; generating a first speech audio stream based on the first word stream; acquiring a first video material stream based on the first word stream; and generating a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

[0007] According to another aspect of this disclosure, a digital human driving system is provided, including an information acquisition module, a decision-making module, a response generation module, a speech synthesis module, a visual scheduling module, and a rendering module. The information acquisition module is configured to acquire real-time input information for the digital human. The decision-making module is configured to control the information acquisition module to input the real-time input information into the response generation module. The response generation module is configured to: process the real-time input information using a large model to obtain a first word stream output by the large model in real time; and distribute the first word stream in parallel to the speech synthesis module and the visual scheduling module. The speech synthesis module is configured to: synthesize a first speech audio stream in real time based on the first word stream; and send the first speech audio stream to the rendering module. The visual scheduling module is configured to: obtain a first video material stream based on the first word stream; and send the first video material stream to the rendering module. The rendering module is configured to: generate a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

[0008] According to another aspect of this disclosure, a driving device for a digital human is provided, comprising: a first acquisition unit configured to acquire real-time input information for the digital human; a first input unit configured to input the real-time input information into a large model to obtain a first word stream output in real time by the large model; a first generation unit configured to generate a first speech audio stream based on the first word stream; a second acquisition unit configured to acquire a first video material stream based on the first word stream; and a second generation unit configured to generate a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the digital human driving method of this disclosure.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to execute the driving method of the digital human of this disclosure.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the digital human driving method of this disclosure when executed by a processor.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram of the digital human interaction capability framework in related technologies is shown; Figure 2 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown; Figure 3 A flowchart illustrating a method for driving a digital human according to an embodiment of the present disclosure is shown; Figure 4 An architecture diagram of a digital human driving system according to an exemplary embodiment of the present disclosure is shown; Figure 5 A flowchart illustrating the acquisition of real-time input information according to an embodiment of the present disclosure is shown; Figure 6 A flowchart illustrating a method for driving a digital human according to an embodiment of the present disclosure is shown; Figure 7 A flowchart illustrating a method for driving a digital human according to an embodiment of the present disclosure is shown; Figure 8 A flowchart illustrating a method for driving a digital human according to an embodiment of the present disclosure is shown; Figure 9 A schematic diagram of the decision-making logic in a digital human interaction scenario according to an exemplary embodiment of the present disclosure is shown; Figure 10 A structural block diagram of a digital human driving system according to an embodiment of the present disclosure is shown; Figure 11 A structural block diagram of a driving device for a digital human according to an embodiment of the present disclosure is shown; Figure 12 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0017] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0018] Figure 1 A schematic diagram of the digital human interaction capability framework in related technologies is shown.

[0019] In related technologies, such as Figure 1 As shown, the commonly used human-digital human interaction framework mainly includes the following steps: First, an AI digital human avatar (usually a 2D or 3D static seated or standing public library avatar) is created in advance, and a real-time dialogue backend service capability for the digital human is built; Second, during the online dialogue stage, when a user converses with the digital human, the user's voice input is first converted into text through Automatic Speech Recognition (ASR), and after Voice Activity Detection (VAD), the text is input into a large model or FAQ library; Next, the large model generates a response text based on the user's question; Finally, the generated response text is processed by Text To Speech (TTS), and lip-sync is rendered based on the synthesized speech inference to ultimately achieve the digital human output.

[0020] However, this traditional interaction scheme suffers from significant real-time performance deficiencies. Because the various processing stages (speech recognition, text generation, speech synthesis, and lip-syncing) typically employ a serial processing architecture, downstream modules often need to wait for the upstream modules to fully output their results before they can begin working (e.g., they must wait for complete text generation before speech synthesis, and wait for speech synthesis before driving lip movements). This results in a long end-to-end latency in the entire chain. Currently, the response time of such general solutions is generally 4 seconds or more, which users can clearly perceive as a lag in the dialogue, failing to meet the demands for highly realistic, low-latency real-time interaction.

[0021] This disclosure provides a method for driving a digital human. It involves inputting real-time input information from an interactive object (such as a user) into a large language model to obtain a first token stream for real-time output. This first token stream is then distributed in parallel to a speech synthesis module and a visual scheduling module. Subsequently, based directly on this token stream, a real-time synthesized speech audio stream and a matched video material stream are simultaneously acquired, and a video stream output to drive the digital human is generated accordingly. This disclosure utilizes the token stream as a unified driving signal, breaking the traditional sequential dependency of "text first, then audio and video." It enables the parallel execution of audio synthesis and video material acquisition without waiting for a response text, significantly reducing end-to-end latency from input to video output and greatly improving the smoothness and responsiveness of real-time interaction with the digital human.

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] Figure 2 A schematic diagram of an exemplary system 200 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 2 The system 200 includes one or more client devices 201, 202, 203, 204, 205, and 206, a server 220, and one or more communication networks 210 coupling the one or more client devices to the server 220. The client devices 201, 202, 203, 204, 205, and 206 can be configured to execute one or more applications.

[0024] In embodiments of this disclosure, server 220 may run one or more services or software applications that enable the execution of the digital human driving methods of this disclosure.

[0025] In some embodiments, server 220 may also provide other services or software applications, which may include non-virtual environments and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, for example, to users of client devices 201, 202, 203, 204, 205 and / or 206 under a Software as a Service (SaaS) model.

[0026] exist Figure 2In the configuration shown, server 220 may include one or more components that implement the functions performed by server 220. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 201, 202, 203, 204, 205, and / or 206 can sequentially interact with server 220 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 200. Therefore, Figure 2 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0027] Users can interact with the digital human using client devices 201, 202, 203, 204, 205, and / or 206. The client devices provide an interface that allows users to interact with the client device. The client devices can also output information to the user through this interface. Although... Figure 2 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0028] Client devices 201, 202, 203, 204, 205, and / or 206 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0029] Network 210 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 210 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 220 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 220 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 220 may run one or more services or software applications that provide the functionality described below.

[0031] The computing unit in server 220 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 220 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 220 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 201, 202, 203, 204, 205 and / or 206. Server 220 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 201, 202, 203, 204, 205 and / or 206.

[0033] In some implementations, server 220 can be a server for a distributed system or a server integrated with blockchain. Server 220 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0034] System 200 may also include one or more databases 230. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 230 may be used to store information such as audio files and video files. Databases 230 may reside in various locations. For example, a database used by server 220 may be local to server 220, or it may be located remotely to server 220 and may communicate with server 220 via a network-based or dedicated connection. Databases 230 may be of different types. In some embodiments, the database used by server 220 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 230 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0036] Figure 2 The system 200 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0037] According to embodiments of this disclosure, such as Figure 3 As shown, a method for driving a digital human is provided, including: step S301, acquiring real-time input information for the digital human; step S302, inputting the real-time input information into a large model to obtain a first word stream output by the large model in real time; step S303, generating a first speech audio stream based on the first word stream; step S304, acquiring a first video material stream based on the first word stream; and step S305, generating a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

[0038] Therefore, by inputting real-time input information from interactive objects (such as users) to a large model to obtain a real-time output first token stream, this first token stream is distributed in parallel to the speech synthesis module and the visual scheduling module. Subsequently, based directly on this token stream, the real-time synthesized speech audio stream and the matched video material stream are simultaneously acquired, and a video stream output to drive the digital human is generated accordingly. The embodiments of this disclosure utilize the token stream as a unified driving signal, breaking the traditional serial dependency of "text first, then audio and video," enabling the parallel execution of audio synthesis and video material acquisition without waiting for a response text. This significantly reduces the end-to-end latency from input to video output, greatly improving the fluency and response speed of real-time interaction with the digital human (enabling a response speed of less than 2 seconds), achieving an effect comparable to real-person dialogue.

[0039] In some embodiments, the real-time input information acquired for the digital human can be plain text data or instruction data. Its source can be text directly entered by the user through input devices such as a keyboard or touchscreen (e.g., input in instant messaging software or web page dialog boxes), or control instructions or status text automatically pushed by a third-party business system (such as a bank inquiry system or a smart home control system) via an API interface. In this embodiment, the real-time input information is acquired by directly receiving the aforementioned text data or instruction data through a network interface (such as WebSocket or HTTP POST) or message queue, without processing by an ASR speech recognition or visual analysis module.

[0040] In some embodiments, the real-time input information acquired for the digital human may include dialogue information obtained from speech and audio recognition based on the input of the interactive object. Specifically, the system may use a microphone or microphone array to acquire the user's voice audio stream in real time as a speech signal. In some embodiments, the speech signal is input to the speech recognition module to obtain the dialogue information output by the speech recognition module. This can be achieved by processing the acquired audio stream using Automatic Speech Recognition (ASR) technology. The ASR module extracts and decodes features from the speech signal, converting the user's spoken speech into computer-processable text information (i.e., dialogue information). This dialogue information accurately records the specific content spoken by the user and can serve as the basic text input for subsequent large-scale models to understand the user's intentions.

[0041] In some embodiments, after acquiring the aforementioned real-time input information, the real-time input information can be directly input into the large model. Specifically, in some simplified interaction scenarios, the system can directly send the acquired information to the large model. For example, in the traditional question-and-answer mode, once valid real-time input information (such as a sentence sent by the user) is detected, the system defaults to the explanation trigger state and directly calls the large model to perform inference generation so that it outputs the first word stream.

[0042] In some embodiments, the aforementioned large model can be a Large Language Model (LLM), which refers to a generative artificial intelligence model built on a deep learning architecture (such as the Transformer architecture), pre-trained on massive amounts of text data, and fine-tuned for specific tasks. This model possesses powerful natural language understanding, logical reasoning, contextual memory, and text generation capabilities, and can handle long-distance semantic dependencies and generate coherent text that conforms to human language habits and logic.

[0043] In some embodiments, the above-mentioned large model can also be applied to a multimodal large model, and there are no restrictions on this.

[0044] In some embodiments, the large model can also be configured to output a word stream that conforms to the persona information.

[0045] In some embodiments, persona information can be a multi-dimensional data set used to define and shape the personalized characteristics of a digital human.

[0046] In some exemplary embodiments, persona information includes, but is not limited to, the following categories: 1) Basic and past experience settings, including the digital human's name, age, occupation, education, and unique life experiences (e.g., "was an English teacher"). This information provides the digital human with a memory background, enabling it to reference specific "personal" history when answering questions about "itself," thus enhancing realism. 2) Language style setting information, used to define the digital human's expression style and rhetorical habits, such as humorous, serious and academic, friendly and gentle, or sharp and direct; 3) Value orientation sets the information, defining the underlying logic and moral stance of digital personalities when expressing opinions. For example, for emotional counselors, the orientation might be "encouragement and comfort" rather than "criticism"; 4) Audience insight settings define the characteristics of the target dialogue group for the digital human. For example, if the audience is set to "middle-aged men aged 35-50 who are interested in current affairs", the digital human will automatically choose more in-depth and macro-level vocabulary when generating responses; if the audience is teenagers, it will use more colloquial and lively expressions. 5) Set the response angle information to define the preferred approach of the digital human when facing questions. For example, you can choose a "macro narrative perspective" or a "micro-level perspective" (such as stories of friends); you can also set it to "non-direct answer", that is, use humorous anecdotes or rhetorical questions to cleverly defuse tricky questions, rather than mechanically providing factual answers.

[0047] In some embodiments, using a large model, for real-time input information, outputting a lexical stream that conforms to the persona based on persona information can be achieved by: the system converting the aforementioned structured persona information into system prompts or contextual constraints described in natural language. When real-time input information (such as a user's question) is input into the large model, the system inputs the persona prompt as a pre-instruction. During autoregressive inference, the large model is constrained by this pre-instruction, calculating the probability distribution of each candidate lexical token. The model tends to select lexical tokens that conform to both semantic logic and persona characteristics (such as specific tone words or specific viewpoints) for output.

[0048] Thus, by configuring the large model to output a word stream that conforms to the character information, style constraints were imposed from the source text generation end, ensuring that the output language content matches the action style in the video material library, thus creating a unified and three-dimensional digital human image.

[0049] In some embodiments, the first token stream refers to the sequence of tokens representing the smallest semantic or functional units that are output sequentially over time during the autoregressive generation process of a large model. These tokens can be characters, words, or subwords in natural language, or they can be model-specific control characters. The first token stream has a streaming characteristic, meaning that it outputs one token as soon as it is generated, without waiting for the complete sentence to be generated.

[0050] In some embodiments, the digital human driving method according to this embodiment can be executed based on a data human driving system, the generation of the first speech audio stream based on the first word stream can be performed by the speech synthesis module in the system, and the acquisition of the first video material stream based on the first word stream can be performed by the visual scheduling module in the system.

[0051] In some embodiments, after each word output by the large model, it can simultaneously place it into the input buffer queues of both the speech synthesis module and the visual scheduling module, and concurrently send the first word stream to both modules. The speech synthesis module can employ streaming TTS (Text-to-Speech) technology to receive the word stream in real time, perform acoustic model inference, and output continuous audio data frames, which constitute the first speech audio stream. The visual scheduling module can use the words in the word stream as search keys to search for matching video clips (such as video frame sequences or skeletal animation data of specific actions or expressions) in a pre-set offline material library in real time. These retrieved and ordered video clips (including matched action clips or natural rhythmic clips used to fill in missing ones) constitute the first video material stream.

[0052] In some embodiments, generating a first video stream based on a first audio stream and a first video footage stream may include the following process: First, the rendering engine synchronously receives the first audio stream and the first video footage stream on the timeline; then, the rendering engine uses a lip-sync algorithm (such as audio feature-driven mesh deformation) to adjust the lip movements of the digital human face in the first video footage stream in real time according to the acoustic characteristics of the first audio stream, so as to achieve audio-visual synchronization; simultaneously, the rendering engine interpolates or smooths adjacent footage segments in the first video footage stream to eliminate abrupt transitions in actions and expressions. The continuous video frame synthesized after the above processing, containing synchronized sound, lip movements, expressions, and body movements, is the first video stream. This first video stream is pushed to the terminal display device, thereby driving the digital human image on the screen to perform real-time broadcasts and action performances that match the audio content.

[0053] In some embodiments, the above-described digital human driving method may further include: inputting real-time input information into a decision model before inputting real-time input information into a large model to obtain the interaction state output by the decision model after making a scene judgment based on the real-time input information; and inputting real-time input information into a large model to obtain the first word stream output by the large model in real time may include: in response to determining that the interaction state indicates an explanatory state, inputting real-time input information into a large model to obtain the first word stream.

[0054] Therefore, by inputting real-time input information into the decision model to obtain the interaction state before inputting it into the large model, and then executing the operation of inputting real-time input information into the large model when the interaction state indicates that it is in the explanatory state, the pre-judgment mechanism ensures that the call of the large model matches the current interaction scenario, avoids invalid generation, and improves the accuracy and logic of digital human interaction.

[0055] In some embodiments, the decision model described above can be a lightweight classification and decision neural network model independent of the larger model. Its core function is to act as the state machine controller or central scheduler of the entire interactive system. In some embodiments, supervised learning methods can be used to train the decision model. Specifically, a training dataset containing a large number of multimodal interaction scenarios can be constructed. Each sample in this dataset contains the input speech text, visual description information, and corresponding manually labeled correct interaction state labels (e.g., "explaining state," "listening state," "clarifying state," "initiating state," etc.). By learning from this dataset, the decision model can establish a mapping relationship from multimodal input features to interaction states, thereby enabling it to recognize user intent and determine the current timing of the dialogue.

[0056] Figure 4 An architecture diagram of a digital human driving system according to an exemplary embodiment of the present disclosure is shown.

[0057] In some exemplary embodiments, such as Figure 4 As shown, the acquired real-time input information can first be fed into the decision model. The decision model performs semantic analysis and intent recognition on the information and outputs an interaction state label accordingly to indicate the subsequent interaction state of the digital human. This interaction state can be, for example, a gating signal that triggers the system to route the real-time input information to the input of the large model. The large model responds to this input by performing real-time inference and streaming the first word stream, thereby initiating the subsequent parallel processes of speech synthesis and visual scheduling, driving the digital human to explain content with synchronized actions and voice.

[0058] In some embodiments, such as Figure 5 As shown, the real-time input information includes dialogue information and visual description information. Obtaining real-time input information for the digital human includes: step S501, receiving the voice signal and video image signal of the interactive object; step S502, performing speech recognition on the voice signal to obtain dialogue information; and step S503, inputting the video image signal into the multimodal large model to obtain the visual description information output by the multimodal large model.

[0059] Therefore, by introducing multimodal perception capabilities, digital humans can make decisions and generate data by integrating auditory and visual information, which greatly improves the depth and breadth of understanding interactive scenarios.

[0060] In some embodiments, the interaction object typically refers to a real user who interacts with the digital human system in real time. In some embodiments, the interaction object may also be other intelligent agents or digital humans, without limitation.

[0061] In some embodiments, see continue to see Figure 4Receiving voice and video signals from interactive objects can be achieved through hardware devices deployed on the interactive terminal or acquisition end. Specifically, the system can use a microphone or microphone array to acquire the user's voice audio stream in real time as the voice signal; simultaneously, the system uses a camera (such as a high-definition camera or depth camera) to capture real-time images of the user and their surrounding environment as the video signal. These two signals are usually acquired synchronously on the timeline to ensure the temporal consistency of subsequent multimodal understanding.

[0062] In some embodiments, the speech signal is input into the speech recognition module to obtain the dialogue information output by the speech recognition module. This can be achieved by processing the acquired audio stream using Automatic Speech Recognition (ASR) technology. The ASR module extracts and decodes features from the speech signal, converting the user's spoken speech into computer-processable text information (i.e., dialogue information). This dialogue information accurately records the specific content spoken by the user, serving as the basic text input for subsequent large-scale models to understand the user's intentions.

[0063] In some embodiments, the content of the video signal (or video image) may include, but is not limited to: the user's facial expressions (e.g., happy, serious), the user's clothing (e.g., wearing nice clothes), the user's body movements, and the user's environmental background (e.g., weather conditions, indoor / outdoor environment). The video signal containing the above information can be input into a multimodal large model to obtain the visual description information output by the multimodal large model, thereby achieving the technical effect of a digital human that "can both hear and see".

[0064] In some embodiments, a multimodal large model can be a deep learning model with cross-modal understanding capabilities (such as a general multimodal understanding model), which has the ability to understand the environment, emotions, and actions, and can convert visual signals into semantic text descriptions.

[0065] In some embodiments, the visual description information output by the multimodal large model can refer to the textual description output by the multimodal large model after analyzing the aforementioned visual images. For example, the model can output descriptive text such as "The user looks great today," "The user seems to be in a good mood," or "The weather is nice in the user's environment" based on the image. By inputting this visual description information along with dialogue information into the decision-making model, the model can make decisions in real time based on the user's current state, enabling the digital human to perceive the real world and thus make richer responses beyond just answering questions (such as proactively initiating small talk about clothing or mood), greatly enhancing the realism and human touch of the interaction.

[0066] In some embodiments, the multimodal large model can also be configured as an end-to-end audiovisual fusion perception model. In this embodiment, the input interface of the multimodal large model can simultaneously receive speech audio signals and video image signals from the interactive object. Internally, the multimodal large model employs a cross-modal attention mechanism or a multimodal joint encoder to align and fuse acoustic features (such as tone, volume, speech rate, and emotional tone) in the speech audio signals with visual features (such as facial micro-expressions, body posture, and environmental background) in the video image signals.

[0067] For example, when a user's facial expression is blank in a video frame, but the audio signal contains rapid breathing or a trembling tone, the multimodal big data model can comprehensively output visual descriptive information (or multimodal contextual descriptive information) indicating that "the user is exhibiting tension or anxiety." This fusion processing method can significantly improve the accuracy of recognizing complex emotions of interactive objects (such as wry smiles, irony, excited silence, etc.), thus enabling the output descriptive information to contain a deeper interpretation of the consistency or conflict between auditory emotions and visual behavior (e.g., "the user is laughing while saying something negative"), providing more accurate high-dimensional semantic input for subsequent decision-making models.

[0068] In some embodiments, obtaining a first video material stream based on a first word stream may include: using words in the first word stream as search keywords for matching and searching in a preset video material library; and in response to finding the first video material corresponding to the word, adding the first video material to the first video material stream.

[0069] The visual scheduling module receives and parses the first word stream from the large model in real time. Internally, the visual scheduling module maintains an inverted index or hash map, which establishes a correspondence between text words (or their corresponding text semantics) and video clip IDs. When a word is received, the visual scheduling module uses it as a search key in the map. If a corresponding record exists in the map (i.e., the word has explicit action semantics, such as "like," "laugh," or "wave"), the search is considered successful, and the system immediately retrieves the corresponding first video clip data from a pre-set video clip library based on the video clip ID in the record.

[0070] Therefore, by using the tokens in the first token stream as search keywords, matching and searching are performed in the pre-set video material library, and in response to the search of the corresponding first video material, it is added to the first video material stream. This realizes the direct use of tokens to drive the pre-set high-quality materials, replacing real-time generation with retrieval, which greatly reduces the computational load and latency, while ensuring accurate matching of actions and semantics.

[0071] In some embodiments, obtaining the first video material stream based on the first token stream may further include: in response to the first video material corresponding to the token not being detected, obtaining the video material corresponding to the current interaction state in the video material library based on the current interaction state of the digital human; and adding the video material corresponding to the current interaction state to the first video material stream.

[0072] If there is no corresponding record of the token in the mapping table (i.e., the token is a function word or a word without strong action semantics, such as "of", "is"), it is determined that the retrieval fails. In this case, according to the current interaction state of the digital human (such as "listening state" or "explanation state"), the video material corresponding to this interaction state can be obtained from the video material library and added to the first video material stream.

[0073] Thus, through the state filling mechanism, it can be ensured that the digital human still maintains a natural dynamic consistent with the current state during the semantic action blank period, realizing the smoothness and continuity of the video stream output.

[0074] In some embodiments, the construction of the video material library may include: obtaining the preset persona information of the digital human and multiple preset tags; using a visual generation large model to generate at least one candidate video material for each preset tag based on the persona information and the image material of the digital human to construct the video material library, where the preset tag corresponding to each candidate video material in the video material library is used to match the token or the interaction state.

[0075] In some embodiments, the above video material library may be a set of a large number of highly realistic and short-duration video clips (or a set of skeletal animation data) corresponding to a certain digital human. The material library can be divided into multiple subsets according to the interaction state, and each video material has a corresponding label, such as one or more semantic labels. For example, in the subset corresponding to the explanation state, the labels correspond to specific action semantics or language contents, such as: "like", "give a heart", "spread hands (indicating helplessness)", "laugh", "think", "emphasis (hand waving)", etc. In the listening material set, the labels correspond to feedback intentions or emotional states, such as: "nod (indicating approval)", "smile (indicating happiness)", "lean forward (indicating interest / attention)", "tilt the head (indicating doubt)", and "natural rhythm", etc.

[0076] In some embodiments, the generation of video footage and the construction of the video footage library can be completed offline before the digital human system goes online. Specific video footage generation can include: First, the system can acquire the digital human's persona information and basic image materials (such as a high-resolution frontal photo or a 3D model); then, it can acquire a predefined list of preset tags, and using a large visual generation model (such as a Sora-type model or a graph-based video model), the basic image materials are used as visual input, and the persona information combined with the currently traversed preset tags are used as text prompts, which are then input into the large visual generation model. The large visual generation model can then output video clips that are highly consistent with the tags and persona based on the above input, and the system can then store these clips in the footage library and establish a tag index.

[0077] Therefore, by utilizing a large visual generation model, candidate video materials are generated based on character information and image materials, targeting multiple preset tags to build a video material library. This achieves automated and batch construction of the material library, ensuring a high degree of consistency between the materials and specific character designs, as well as comprehensive tag coverage.

[0078] In some embodiments, such as Figure 6 As shown, the above-mentioned driving method for the digital human may further include: step S601, in response to determining that the interaction state indicates a listening state, and in response to determining that the digital human is currently in a state of outputting speech, interrupting the word stream output of the large model; step S602, acquiring a second video material stream, wherein the second video material stream includes video material corresponding to the listening state; and step S603, generating a second video stream for driving the digital human based on the second video material stream.

[0079] Therefore, by immediately interrupting the large model's word stream output when the decision model outputs the current interaction state as "listening" and the digital human is in the output speech state, the audio generation of the speech synthesis module is interrupted, and the visual scheduling module switches to sending a second video material stream containing video material corresponding to the listening state. This solution can respond to changes in the interaction state in real time, forcibly terminate the current output, and seamlessly switch to the listening behavior, achieving instant interruption and state transition, significantly improving the realism of the interaction.

[0080] In some embodiments, the decision model can also be configured to output an interaction feedback label corresponding to the preset interaction intent in response to detecting a preset interaction intent of the interactive object based on real-time input information. The above-mentioned digital human driving method can also include: in response to the decision model outputting the interaction feedback label, obtaining a second video material corresponding to the interaction feedback label; and adding the second video material to the second video material stream.

[0081] In some embodiments, the preset interaction intent can refer to the current emotional tendency, attitude, or potential communicative needs of the interaction object identified by the decision model after in-depth analysis of real-time input information (including the semantic content of dialogue information and the emotional characteristics of visual descriptive information). For example, when the system recognizes that the user says "You're right" or nods frequently, the preset interaction intent is "agreement / approval"; when the system recognizes that the user says "I don't understand" or frowns, the preset interaction intent is "confusion / bewilderment"; when the system recognizes that the user smiles or speaks in a cheerful tone, the preset interaction intent is "happiness / joy".

[0082] In some embodiments, the interactive feedback label can be a semantic index identifier output by the decision model to the visual scheduling module for retrieving specific video clips based on the identified preset interactive intent. This label corresponds to the semantic tags in the offline media library. For example, the output interactive feedback label for the "agree" intent could be "nodding"; the output label for the "interested" intent could be "leaning forward." By outputting these labels, the decision model can instruct the visual scheduling module to accurately retrieve corresponding nonverbal behavior segments (such as nodding or smiling expressions) from the media library without interrupting the user's speech (i.e., maintaining a listening state), thereby achieving a lifelike "active listening" effect.

[0083] This enables digital humans to provide real-time nonverbal feedback (such as nodding or shaking their heads) based on the user's intentions (such as agreement or doubt) while listening, enhancing the two-way nature and anthropomorphism of the interaction.

[0084] In some embodiments, such as Figure 7 As shown, the above-mentioned method for driving a digital human may further include: step S701, in response to determining that the interaction state indication is a clarification state, using a large model to generate a second lexical stream for confirming the intention of the interactive object based on the currently received input information; step S702, generating a second speech audio stream based on the second lexical stream; step S703, obtaining a third video material stream based on the second lexical stream; and step S704, generating a third video stream for driving the digital human based on the second speech audio stream and the third video material stream.

[0085] Thus, by actively truncating the input and generating clarifying statements, proactive control and guidance of chaotic dialogue scenarios are achieved, thereby improving the effectiveness of the interaction.

[0086] In some embodiments, the real-time input information may further include a silence detection signal for the interactive object, and the decision model may be configured to output the interaction state based on visual descriptive information in response to a silence detection signal indicating that no voice signal of the interactive object has been detected for a preset time. Figure 8As shown, the above-mentioned method for driving the digital human may further include: step S801, in response to determining that the interaction state indicates an active initiation state, inputting visual description information into a large model to obtain a third word stream for initiating a new topic, which is output in real time by the large model; step S802, generating a third speech audio stream based on the third word stream; step S803, obtaining a fourth video material stream based on the third word stream; and step S804, generating a fourth video stream for driving the digital human based on the third speech audio stream and the fourth video material stream.

[0087] Therefore, in response to the silence detection signal indicating a timeout, the decision model outputs an active initiation state based on visual description information and inputs the visual description information into the large model to obtain a lexical stream for initiating new topics. This enables the digital human to actively start topics based on what it "sees," effectively avoiding awkward silences and improving the user experience.

[0088] Figure 9 A schematic diagram of the decision-making logic in a digital human interaction scenario according to an exemplary embodiment of the present disclosure is shown.

[0089] In some embodiments, the interaction process of the digital human can achieve highly realistic switching between listening and explaining states, as well as proactive interaction and interruption control based on the above decision model (i.e., the "digital human brain").

[0090] In some exemplary embodiments, such as Figure 9 As shown, in addition to question-and-answer interaction scenarios with users, digital human interaction scenarios can also include interruption and return listening scenarios, clarification dialogue scenarios, and proactive communication scenarios.

[0091] In the context of interrupting and returning to listening, please continue to refer to Figure 9 During the digital human's speech output, the system continues to acquire real-time user input and feed it into the decision model. If the decision model outputs an interaction state indicating a listening state (i.e., judging that "the user has the intention to speak"), and the digital human is currently outputting speech, the generation and distribution of the lexical stream being generated by the large model can be immediately interrupted, and the audio output of the speech synthesis module can be stopped. Simultaneously, the visual scheduling module can be controlled to quickly switch to recalling video material from the subset of material corresponding to the listening state, such as second video material like "shutting up and nodding," "leaning forward," or "focused gaze," and send it to the rendering module as a video material stream for digital human video generation. Thus, not only is the sound abruptly stopped, but the visual transition to a listening posture is also seamless, simulating the natural reaction of a real person being interrupted.

[0092] In addition, when listening, if the system detects a change in the user's emotions (such as the user laughing happily), the decision model will output a corresponding interactive feedback label (such as "happy"), triggering the visual scheduling module to insert a second video clip of "smiling and nodding", thus achieving a realistic effect of "making facial expression feedback while hearing".

[0093] In clarifying the dialogue scenario, please refer to [link / reference]. Figure 9 When a user makes a lengthy, unclear statement (e.g., illogical, off-topic, or redundant), the decision model analyzes the semantic logic of the real-time input information and determines that "the user needs to clarify their intentions." In response, the decision model outputs an interaction state indicating clarification and triggers the larger model to generate guiding statements for clarifying the conversation based on previously received user input (e.g., "Excuse me, did you mean to say…?").

[0094] During this process, the system will pause the transmission of invalid voice information that the user continues to input (i.e., truncate the input), forcing the large model to generate the second word stream of the above-mentioned clarifying words based only on the received "chaotic text", and drive the digital human to interrupt the user, regain control of the conversation, and guide the interaction back on track.

[0095] In scenarios where communication is initiated proactively, please refer to [link / reference]. Figure 9 When a situation occurs where "the user has not spoken for a long time" (i.e., the VAD detects prolonged silence), the decision model, combined with the visual description information provided by the multimodal big model (such as "the user is looking at the screen but not speaking"), judges that the current situation is an awkward silence and that "initiating communication" is necessary. In response, the decision model outputs an interaction state indicating an initiative to initiate communication and triggers the big model to generate guiding words for breaking the ice (e.g., "How are you feeling today?").

[0096] In this process, the system inputs visual descriptive information (such as "the user looks worried") into the large model, which generates a context-appropriate third meta-stream (such as a heartwarming greeting) and drives the digital human to speak proactively, thereby breaking the silence and enhancing the agent's service initiative and human-like emotions.

[0097] In some embodiments, the above-described digital human driving method may further include: acquiring response text generated by a large model based on a first meta-stream, wherein the response text is used as subtitles when driving the digital human based on a first video stream.

[0098] Therefore, by directly generating response text as subtitles based on the first word stream from a large model, the word stream information from the generation source can be directly utilized without additional speech recognition steps. This ensures absolute consistency between the subtitle content and the speech content, eliminates recognition delay, and improves the real-time performance and accuracy of information display.

[0099] In some embodiments, such as Figure 10 A digital human driving system 1000 is provided, including an information acquisition module 1010, a decision-making module 1020, a response generation module 1030, a speech synthesis module 1040, a visual scheduling module 1050, and a rendering module 1060. The information acquisition module 1010 is configured to acquire real-time input information for the digital human; the decision-making module 1020 is configured to control the information acquisition module 1010 to input the real-time input information into the response generation module 1030; the response generation module 1030 is configured to process the real-time input information using a large model to obtain the first response from the real-time output of the large model. The first word stream is distributed in parallel to the speech synthesis module 1040 and the visual scheduling module 1050; the speech synthesis module 1040 is configured to: synthesize a first speech audio stream in real time based on the first word stream; and send the first speech audio stream to the rendering module 1060; the visual scheduling module 1050 is configured to: obtain a first video material stream based on the first word stream; and send the first video material stream to the rendering module 1060; and wherein the rendering module 1060 is configured to generate a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

[0100] The operations performed by the aforementioned digital human driving system 1000 and the effects it can achieve are similar to those of the digital human driving method described above, and will not be repeated here.

[0101] In some embodiments, the decision module can also be configured to: input real-time input information into the decision model deployed in the decision module before the control information acquisition module inputs real-time input information into the response generation module, so as to obtain the interaction state output by the decision model after making a scenario judgment based on the real-time input information; and in response to determining that the interaction state indicates an explanation state, execute the operation of the control information acquisition module to input real-time input information into the response generation module.

[0102] In some embodiments, the visual scheduling module may also be configured to perform the acquisition operation of the first video material stream by: using the words in the first word stream as search keywords to perform a matching search in a preset video material library; and in response to the retrieval of the first video material corresponding to the word, adding the first video material to the first video material stream.

[0103] In some embodiments, the visual scheduling module may also be configured to: in response to the first video material corresponding to the undetected word, obtain the video material corresponding to the current interaction state from the video material library based on the current interaction state of the digital human; and add the video material corresponding to the current interaction state to the first video material stream.

[0104] In some embodiments, the decision module may also be configured to: interrupt the lexical stream output of the response generation module and the speech audio generation of the speech synthesis module in response to determining that the interaction state indicates a listening state and in response to determining that the digital human is currently in a state of outputting speech; cause the visual scheduling module to acquire a second video material stream and send it to the rendering module, wherein the second video material stream includes video material corresponding to the listening state; and wherein the rendering module may also be configured to generate a second video stream for driving the digital human based on the second video material stream.

[0105] In some embodiments, the decision module may also be configured to, in response to detecting a preset interaction intent of an interactive object based on real-time input information through a decision model, output an interaction feedback tag corresponding to the preset interaction intent; and wherein, the visual scheduling module may also be configured to, in response to receiving the interaction feedback tag output by the decision module, acquire a second video material corresponding to the interaction feedback tag, and add the second video material to the second video material stream for sending to the rendering module.

[0106] In some embodiments, the decision module may also be configured to: in response to determining that the interaction state indicates a clarification state, control the information acquisition module to pause sending subsequently received real-time input information to the response generation module, so that the response generation module, based only on the currently received input information, calls the large model to generate a second word stream for confirming the intention of the interactive object, and distributes the second word stream in parallel to the speech synthesis module and the visual scheduling module; the speech synthesis module may also be configured to generate a second speech audio stream in real time based on the second word stream; the visual scheduling module may also be configured to obtain a third video material stream based on the second word stream; and wherein the rendering module may also be configured to generate a third video stream for driving the digital human based on the second speech audio stream and the third video material stream.

[0107] In some embodiments, the real-time input information may include dialogue information and visual description information. The information acquisition module may include a speech recognition module and a multimodal analysis module. The information acquisition module may be configured to: receive the voice signal and video image signal of the interactive object; input the voice signal into the speech recognition module to obtain the dialogue information output by the speech recognition module; and input the video image signal into the multimodal analysis module to analyze the video image signal using a multimodal large model to obtain the visual description information output by the multimodal large model.

[0108] In some embodiments, the real-time input information may further include a silence detection signal for the interactive object. The decision module may also be configured to: in response to the silence detection signal indicating that no voice signal of the interactive object has been detected for more than a preset time, output the interaction state based on visual description information through the decision model; and in response to determining that the interaction state indicates an active initiation state, cause the information acquisition module to input the visual description information into the response generation module, so that the response generation module, based on the visual description information, calls the large model to output a third word stream for initiating a new topic in real time, and distributes the third word stream in parallel to the speech synthesis module and the visual scheduling module; the speech synthesis module may also be configured to generate a third speech audio stream in real time based on the third word stream; the visual scheduling module may also be configured to obtain a fourth video material stream based on the third word stream; and wherein the rendering module may also be configured to generate a fourth video stream for driving the digital human based on the third speech audio stream and the fourth video material stream.

[0109] In some embodiments, the construction of a video material library may include: obtaining preset character information of a digital human and multiple preset tags; using a large visual generation model, based on the character information and the image material of the digital human, generating at least one candidate video material for each of the multiple preset tags to construct a video material library, wherein the preset tag corresponding to each candidate video material in the video material library is used to match lexical units or interactive states.

[0110] In some embodiments, the large model can also be configured to output a word stream that conforms to the persona information.

[0111] In some embodiments, the response generation module can also be configured to call a large model to generate response text based on the first word stream; and the rendering module can also be configured to display the response text as subtitles synchronously with the first video stream.

[0112] In some embodiments, such as Figure 11 As shown, a driving device 1100 for a digital human is also provided, comprising: a first acquisition unit 1110 configured to acquire real-time input information for the digital human; a first input unit 1120 configured to input the real-time input information into a large model to obtain a first word stream output in real time by the large model; a first generation unit 1130 configured to generate a first speech audio stream based on the first word stream; a second acquisition unit 1140 configured to acquire a first video material stream based on the first word stream; and a second generation unit 1150 configured to generate a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

[0113] The operations performed by units 1110 to 1150 in the aforementioned digital human driving device 1100 and the technical effects they can achieve are similar to steps S301 to S305 in the aforementioned digital human driving method, and will not be described in detail here.

[0114] In some embodiments, the driving device for the digital human may further include: a second input unit configured to input real-time input information into a decision model before inputting real-time input information into a large model, so as to obtain an interactive state output by the decision model after making a scene judgment based on the real-time input information; and a first input unit may be further configured to input real-time input information into a large model in response to determining that the interactive state indicates an explanatory state, so as to obtain a first word stream.

[0115] In some embodiments, the second acquisition unit may be further configured to: use the words in the first word stream as search keywords to perform matching searches in a preset video material library; and in response to the retrieval of the first video material corresponding to the word, add the first video material to the first video material stream.

[0116] In some embodiments, the second acquisition unit may be further configured to: in response to the first video material corresponding to the undetected word, acquire video material corresponding to the current interaction state from the video material library based on the current interaction state of the digital human; and add the video material corresponding to the current interaction state to the first video material stream.

[0117] In some embodiments, the driving device for the digital human may further include: an interruption unit configured to interrupt the lexical stream output of the large model in response to determining that the interaction state indicates a listening state and in response to determining that the digital human is currently in a state of outputting speech; a third acquisition unit configured to acquire a second video material stream, wherein the second video material stream includes video material corresponding to the listening state; and a third generation unit configured to generate a second video stream for driving the digital human based on the second video material stream.

[0118] In some embodiments, the decision model may also be configured to output an interaction feedback label corresponding to the preset interaction intent in response to detecting a preset interaction intent of the interactive object based on real-time input information. The driving device of the digital human may further include: a fourth acquisition unit configured to acquire a second video material corresponding to the interaction feedback label in response to the decision model outputting the interaction feedback label; and an adding unit configured to add the second video material to the second video material stream.

[0119] In some embodiments, the aforementioned driving device for the digital human may further include: a fourth generation unit configured to, in response to determining that the interaction state indicates a clarification state, generate a second lexical stream for confirming the intention of the interactive object based on currently received input information using a large model; a fifth generation unit configured to generate a second speech audio stream based on the second lexical stream; a fifth acquisition unit configured to acquire a third video material stream based on the second lexical stream; and a sixth generation unit configured to generate a third video stream for driving the digital human based on the second speech audio stream and the third video material stream.

[0120] In some embodiments, the real-time input information includes dialogue information and visual description information. The first acquisition unit can be configured to: receive the voice signal and video image signal of the interactive object; perform speech recognition on the voice signal to obtain dialogue information; and input the video image signal into the multimodal large model to obtain the visual description information output by the multimodal large model.

[0121] In some embodiments, the real-time input information may further include a silence detection signal for the interactive object, and the decision model may be configured to output an interaction state based on visual description information in response to the silence detection signal indicating that no voice signal of the interactive object has been detected for a preset time. The driving device for the digital human may further include: a third input unit configured to input visual description information into a large model in response to determining that the interaction state indicates an active initiation state, so as to obtain a third word stream for initiating a new topic in real time output by the large model; a seventh generation unit configured to generate a third speech audio stream based on the third word stream; a sixth acquisition unit configured to acquire a fourth video material stream based on the third word stream; and an eighth generation unit configured to generate a fourth video stream for driving the digital human based on the third speech audio stream and the fourth video material stream.

[0122] In some embodiments, the construction of a video material library may include: obtaining preset character information of a digital human and multiple preset tags; using a large visual generation model, based on the character information and the image material of the digital human, generating at least one candidate video material for each of the multiple preset tags to construct a video material library, wherein the preset tag corresponding to each candidate video material in the video material library is used to match lexical units or interactive states.

[0123] In some embodiments, the large model can also be configured to output a word stream that conforms to the persona information.

[0124] In some embodiments, the driving device for the digital human may further include: a seventh acquisition unit configured to acquire response text generated by the large model based on the first word stream, wherein the response text is used as subtitles when driving the digital human based on the first video stream.

[0125] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0126] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0127] refer to Figure 12 The present invention describes a structural block diagram of an electronic device 1200 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of the electronic device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0129] Multiple components in electronic device 1200 are connected to I / O interface 1205, including: input unit 1206, output unit 1207, storage unit 1208, and communication unit 1209. Input unit 1206 can be any type of device capable of inputting information to electronic device 1200. Input unit 1206 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 1207 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1208 may include, but is not limited to, a hard disk and an optical disk. The communication unit 1209 allows the electronic device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0130] The computing unit 1201 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the above-described digital human driving method. For example, in some embodiments, the above-described digital human driving method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the above-described digital human driving method can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the above-described digital human driving method by any other suitable means (e.g., by means of firmware).

[0131] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0136] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0137] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0138] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for driving a digital human, the method comprising: obtaining real-time input information for the digital human; inputting the real-time input information into a large model to obtain a first token stream output by the large model in real time; generating a first speech audio stream based on the first token stream; obtaining a first video material stream based on the first token stream; and generating a first video stream for driving the digital human based on the first speech audio stream and the first video material stream.

2. The method of claim 1, further comprising: before inputting the real-time input information into the large model, inputting the real-time input information into a decision model to obtain an interaction state output by the decision model after the decision model judges a scene based on the real-time input information; and the inputting the real-time input information into the large model to obtain a first token stream output by the large model in real time comprises: in response to determining that the interaction state indicates an explanation state, inputting the real-time input information into the large model to obtain the first token stream. the obtaining a first video material stream based on the first token stream comprises: taking a token in the first token stream as a retrieval keyword to perform a matching retrieval in a pre-stored video material library; and 3. The method of claim 2, wherein, in response to retrieving a first video material corresponding to the token, adding the first video material to the first video material stream. the obtaining a first video material stream based on the first token stream further comprises: in response to failing to detect a first video material corresponding to the token, obtaining a video material corresponding to a current interaction state of the digital human in the video material library based on the current interaction state; and 4. The method of claim 3, wherein, adding the video material corresponding to the current interaction state to the first video material stream.

5. The method of claim 4, further comprising: in response to determining that the interaction state indicates a listening state, and in response to determining that the digital human is currently in a state of outputting speech, interrupting the token stream output of the large model; obtaining a second video material stream, wherein the second video material stream comprises a video material corresponding to the listening state; and generating a second video stream for driving the digital human based on the second video material stream. the decision model is further configured to output an interaction feedback label corresponding to a preset interaction intention of an interaction object in response to detecting the preset interaction intention based on the real-time input information, and the method further comprises: in response to the decision model outputting the interaction feedback label, obtaining a second video material corresponding to the interaction feedback label; and adding the second video material to the second video material stream.

6. The method of claim 5, wherein, 7. The method of any one of claims 2 to 6, further comprising: in response to determining that the interaction state indicates a clarification state, generating a second token stream for confirming an intention of an interaction object based on currently received input information using the large model; generating a second speech audio stream based on the second token stream; obtaining a third video material stream based on the second token stream; and ​ ​ ​ ​ generate a third video stream for driving the digital person based on the second speech audio stream and the third video material stream.

8. The method of any one of claims 2 to 7, wherein, The real-time input information includes dialogue information and visual description information, and the obtaining the real-time input information for the digital person includes: receiving a speech signal and a video picture signal of an interactive object; performing speech recognition on the speech signal to obtain the dialogue information; and inputting the video picture signal into a multi-modal large model to obtain the visual description information output by the multi-modal large model.

9. The method of claim 8, wherein, The real-time input information further includes a silence detection signal for the interactive object, and the decision model is further configured to, in response to the silence detection signal indicating that no speech signal of the interactive object is detected for more than a preset time length, output the interaction state based on the visual description information, and the method further includes: in response to determining that the interaction state indicates an active initiation state, inputting the visual description information into the large model to obtain a third token stream for initiating a new topic that is real-time output by the large model; generating a third speech audio stream based on the third token stream; obtaining a fourth video material stream based on the third token stream; and generating a fourth video stream for driving the digital person based on the third speech audio stream and the fourth video material stream.

10. The method of any one of claims 3 to 9, wherein, The construction of the video material library includes: obtaining preset persona information of the digital person and a plurality of preset labels; using a visual generation large model, generating at least one candidate video material for each preset label in the plurality of preset labels based on the persona information and the image material of the digital person to construct the video material library, wherein each candidate video material in the video material library corresponds to a preset label for matching a token or an interaction state.

11. The method of claim 10, wherein, The large model is further configured to output a token stream consistent with the persona information based on the persona information.

12. The method of any one of claims 1-11, further comprising: obtaining a reply text generated by the large model based on the first token stream, wherein the reply text is used as a subtitle displayed when the digital person is driven based on the first video stream.

13. A digital person driving system, comprising an information acquisition module, a decision module, a reply generation module, a speech synthesis module, a visual scheduling module, and a rendering module, wherein the information acquisition module is configured to obtain real-time input information for the digital person; the decision module is configured to control the information acquisition module to input the real-time input information into the reply generation module; the reply generation module is configured to: process the real-time input information using a large model to obtain a first token stream that is real-time output by the large model; and distribute the first token stream to the speech synthesis module and the visual scheduling module in parallel; the speech synthesis module is configured to: real-time synthesize a first speech audio stream based on the first token stream; and send the first speech audio stream to the rendering module; the visual scheduling module is configured to: obtain a first video material stream based on the first token stream; and send the first video material stream to the rendering module; and wherein ​ ​ The rendering module is configured to generate a first video stream for driving the digital person based on the first speech audio stream and the first video material stream.

14. A driving apparatus of a digital person, the apparatus comprising: a first obtaining unit configured to obtain real-time input information for the digital person; a first input unit configured to input the real-time input information into a large model to obtain a first token stream output by the large model in real time; a first generation unit configured to generate a first speech audio stream based on the first token stream; a second obtaining unit configured to obtain a first video material stream based on the first token stream; and a second generation unit configured to generate a first video stream for driving the digital person based on the first speech audio stream and the first video material stream.

15. An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12. The computer instructions are used to enable a computer to perform the method of any one of claims 1-12.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-12.

17. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-12.

Images