Real-time interaction method and program product based on 2D digital human

Abstract

This application discloses a real-time interaction method and program product based on a 2D digital human, relating to the field of artificial intelligence technology. The method includes: acquiring a user audio stream collected and uploaded in real-time by a client; extracting valid speech segments from the user audio stream, performing speech recognition on the valid speech segments to obtain user question text, and generating response text based on the user question text; performing speech synthesis on the response text to obtain a response audio stream; driving a 2D digital human model to perform lip-reading inference based on the response audio stream to generate a digital human video frame sequence; time-synchronized encapsulating the response audio stream and the digital human video frame sequence, and pushing them to the client through a point-to-point real-time communication connection established with the client; and stopping the current digital human response generation task upon detecting valid speech during the generation and push of the digital human response. This application reduces the response latency of digital human interaction and improves the naturalness of digital human interaction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to real-time interaction methods and program products based on 2D digital humans. Background Technology

[0002] With the development of large language models, real-time rendering, and multimodal interaction technologies, digital human interaction, with its human-like performance, is being applied in customer service, marketing, and other scenarios. Currently, feasible forms of digital human interaction mainly include lightweight digital human advisors and digital human coaching platforms. Lightweight digital human advisors are typically integrated into mobile banking, WeChat service accounts, and other entry points via web pages. Digital human coaching platforms simulate customer questions and complaints by replicating business scenarios. Existing solutions mostly use a polling model, requiring users to wait for the digital human to complete a full response before proceeding to the next round of dialogue. This results in high latency, poor immersion and naturalness in the interaction, and a rigid, alternating speaking pattern between the user and the digital human. Summary of the Invention

[0003] The main purpose of this application is to provide a real-time interaction method and program product based on 2D (Two-Dimensional) digital humans, aiming to solve the technical problem of high response latency in digital human interaction, which leads to low naturalness of interaction.

[0004] To achieve the above objectives, this application proposes a real-time interaction method based on a 2D digital human, the method comprising: Acquire user audio streams that are collected and uploaded in real time by the client; The user audio stream is subjected to voice activity detection, valid voice segments are extracted from the user audio stream, the valid voice segments are subjected to speech recognition to obtain the user's question text, and a response text is generated based on the user's question text. The reply text is processed by speech synthesis to obtain a reply audio stream; Based on the response audio stream, a 2D digital human model is driven to perform lip-reading inference, generating a digital human video frame sequence synchronized with the lip-reading of the response audio stream. The response audio stream and the digital human video frame sequence are then time-synchronized and encapsulated, and pushed to the client through a point-to-point real-time communication connection established with the client, so that the client can play the digital human response. In response to the generation and push of the digital human response, if valid speech is detected from the user's audio stream, the current digital human response generation task is stopped, and the data queue that caches the intermediate data of the digital human response is cleared, so as to switch to the listening state of receiving user input.

[0005] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the real-time interactive method based on a 2D digital human as described above.

[0006] In this application, the following steps are taken: First, a user audio stream is acquired and uploaded in real-time by the client. Second, voice activity detection is performed on the user audio stream. Effective speech segments are extracted from the user audio stream. Third, speech recognition is performed on the effective speech segments to obtain a user question text. Fourth, a response text is generated based on the user question text. Fifth, speech synthesis is performed on the response text to obtain a response audio stream. Sixth, a 2D digital human model is driven to perform lip-syncing based on the response audio stream. A digital human video frame sequence synchronized with the response audio stream is generated. The response audio stream and the digital human video frame sequence are then time-synchronized and encapsulated. The response is pushed to the client via a point-to-point real-time communication connection established with the client, so that the client can play the digital human response. Seventh, in response to the generation and pushing of the digital human response, if effective speech is detected from the user audio stream, the current digital human response generation task is stopped, and the data queue caching the intermediate data of the digital human response is cleared to switch to a listening state that receives user input.

[0007] This application solves the technical problem of high latency in digital human interaction by employing user audio streaming acquisition and uploading, voice activity detection and speech recognition, streaming response text generation, streaming speech synthesis, real-time lip-reading inference of 2D digital human models, and direct push via point-to-point real-time communication connections. Compared with existing polling-based interaction technologies, it achieves low-latency output of digital human responses. By employing techniques to detect valid speech, stop the digital human response generation task, and clear the data queue during the generation and push of digital human responses, it solves the technical problem of not supporting real-time interruption during digital human responses, enabling users to interrupt and start new questions at any time during the digital human response process. By using a technique to drive the generation of digital human video frame sequences with lip closure using silent audio blocks, the digital human naturally transitions from a speaking state to a listening state after being interrupted, improving the naturalness of digital human interaction. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the first embodiment of the real-time interaction method based on 2D digital humans in this application; Figure 2 This is a flowchart illustrating the third embodiment of the real-time interaction method based on 2D digital humans in this application; Figure 3 This is a flowchart illustrating the fourth embodiment of the real-time interaction method based on 2D digital humans in this application; Figure 4 This is a simplified flowchart illustrating an embodiment of the real-time interaction method based on 2D digital humans in this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the real-time interaction method based on 2D digital humans in the embodiments of this application.

[0011] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0013] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0014] For ease of understanding, the following explains some key terms in this embodiment: The client is the terminal-side program through which users interact with the digital human. The client is responsible for collecting the user's audio, displaying the digital human's visuals, and playing the digital human's voice. The client is an interactive program running on the user's terminal; it can be a browser page, a mobile application, a WeChat service account page, an embedded page in mobile banking, etc.

[0015] A user audio stream is an audio data sequence that the client continuously captures the user's voice through a microphone and transmits it to the interactive server in chronological order. The user audio stream is audio sample data organized in chronological order; it can be a mono pulse-code modulation audio stream with a sampling rate of 16kHz and 16-bit quantization, or it can be a floating-point format audio sample sequence.

[0016] Speech activity detection is the process of determining whether there is a valid human voice in each segment of audio data. Speech activity detection is an audio classification process that distinguishes between human voices and non-human voices. Speech activity detection can be achieved through deep learning-based speech activity detection models, such as the Silero VAD (Silero Speech Activity Detector) model, or through threshold judgments based on acoustic features such as energy and zero-crossing rate.

[0017] A valid speech segment is an audio segment in the user's audio stream that is determined by speech activity detection to contain valid human voice. A valid speech segment is the signal interval in the audio stream where the human voice is located; a valid speech segment can be the audio segment corresponding to a complete question.

[0018] Speech recognition is the process of converting audio signals into text; it is also known as automatic speech recognition. Speech recognition is the conversion process from audio to text; it can be implemented using open-source speech recognition models such as Funasr (a speech recognition model).

[0019] User question text is the text content obtained after speech recognition of valid speech segments, used to represent user questions or instructions.

[0020] The response text is the text content generated from the user's question text, used by the digital human to answer the user. The response text is the text data to be synthesized; it can be a reply statement generated by a large language model based on the user's question text and knowledge base retrieval results.

[0021] Speech synthesis is the process of converting text content into speech waveforms, also known as text-to-speech. Speech synthesis is a text-to-audio conversion process; speech synthesis can be achieved through timbre cloning algorithms based on large models, or through text-to-speech models with preset timbres.

[0022] The response audio stream is a sequence of synthesized speech data that is output in chronological order after the response text is synthesized into speech.

[0023] 2D digital humans are virtual characters presented through two-dimensional images or two-dimensional video frames. 2D digital humans are flat, virtual figures; they can be digital clones generated from real-life video footage, or virtual customer service avatars driven by human image footage. Unlike 3D modeled and rendered digital humans, 2D digital human visuals consist of sequences of video frames, with lip movements and facial expressions inferred from audio by a 2D generative model.

[0024] A 2D digital human model is a generative model that generates digital human video frames based on input audio. Lip inference is the process by which the 2D digital human model generates lip shapes corresponding to the speech content of the input audio. The 2D digital human model is an audio-driven two-dimensional face generation model; it can be a DINet (DeformationInpainting Network) lip generation model, a Wav2Lip lip-sync model, etc.

[0025] A digital human video frame sequence is a collection of digital human images generated from a 2D digital human model and arranged in chronological order. The lip movements in each frame of the digital human video frame sequence are synchronized with the corresponding speech content.

[0026] Time-synchronized encapsulation is the process of aligning audio data and video frames according to a unified time base and organizing them into transmittable media data. Time-synchronized encapsulation is an audio-visual alignment process; it can be achieved by setting a unified time base and sample count for audio and video frames.

[0027] A peer-to-peer real-time communication connection is a direct communication connection established between an interactive server and a client for transmitting real-time media data. The media data is transmitted directly between the interactive server and the client without being forwarded by a central media server. Peer-to-peer real-time communication connections can be WebRTC (Web Real-Time Communication) connections.

[0028] Digital human responses are a combination of digital human voice and digital human visuals generated by the interactive server in response to user questions and played by the client.

[0029] The intermediate data of digital human response is the data that has been generated but not yet played by the client during the generation and push of digital human response. The intermediate data of digital human response includes the response text to be synthesized, the synthesized response audio stream, the generated digital human video frame sequence, and the audio data and video frames cached in each data queue.

[0030] The listening state is the state in which the interactive server waits for and receives user input after the digital human stops responding. In the listening state, the digital human appears with its lips closed, indicating to the user that it is currently in input-receiving mode.

[0031] A data queue is a storage structure that caches data in a first-in, first-out (FIFO) order. It is used to transfer data between two processing stages; the writer writes data sequentially, and the reader reads data in the same order. The data queue is a buffered storage structure; it can be a memory queue, a cross-process message queue, etc. The data queues involved in this embodiment include: a user audio input queue for caching user audio streams received from the client; a response text queue for caching generated response text; a synthesized audio queue for caching response audio streams generated by the speech synthesis process; a raw video frame queue for caching raw video frames containing only the digital human subject; an overlay composite video frame queue for caching video frames composited after background overlay; and push audio and push video frame queues, respectively used to cache audio data and video frames to be sent to the media stream track of the point-to-point real-time communication connection.

[0032] An independent process is a program instance that runs independently within an operating system. Each independent process has its own execution space, and they exchange data through data queues. The independent processes involved in this embodiment include: a speech synthesis process, used to read the response text and generate a response audio stream; a digital human generation and rendering process, used to read the response audio stream and drive a 2D digital human model to generate original video frames; a video frame overlay process, used to overlay and synthesize the original video frames with a background image; and an audio / video stream push process, used to synchronously encapsulate audio data and video frames and push them to the media stream track of a peer-to-peer real-time communication connection.

[0033] Global dialogue state variables are state identifiers maintained by the interaction server and visible to each processing stage. The first state variable indicates whether the user is in a talking state, and the second state variable indicates whether the interaction server is in the state of generating and outputting digital human responses. Global dialogue state variables are Boolean state identifiers; the first state variable can be the `user_talking` variable, and the second state variable can be the `responsing` variable.

[0034] The start-of-speech threshold is the effective duration limit for determining when a user begins to speak, and the stop-of-speech threshold is the effective duration limit for determining when a user finishes speaking. Both the start-of-speech and stop-of-speech thresholds are time-limit parameters; the start-of-speech threshold can be set to 0.2 seconds, and the stop-of-speech threshold can be set to 0.1 seconds.

[0035] A silence audio block is an audio data block that does not contain human voice energy. A silence audio block is zero-energy or low-energy audio data; a silence audio block can be a 1.28-second, 16kHz all-zero sampled audio block.

[0036] Large language models are neural network models with a large number of parameters capable of text understanding and text generation. Retrieval-enhanced generation involves retrieving relevant knowledge from a knowledge base before the large language model generates text, and then providing the retrieved knowledge along with the input to the large language model, thus incorporating specialized knowledge into the generated text. Incremental text generation refers to the newly added text fragments added to the output of the large language model each time it generates response text in a streaming output mode.

[0037] The large-model timbre cloning algorithm is an algorithm that synthesizes speech with the target timbre based on a large model and a reference audio file containing the target timbre. The large-model timbre cloning algorithm can be an indexTTS (Text to Speech) algorithm. An asynchronous streaming interface is an interface that returns data in segments after generation, without waiting for all data to be generated; an asynchronous streaming interface can be a StreamingResponse interface based on the FastAPI (a web framework). A chunked transmission encoding mechanism is an encoding mechanism that divides response data into several data blocks and transmits them sequentially.

[0038] The original video frame is a video frame generated during the digital human generation and rendering process, containing only the digital human subject. Areas outside the digital human subject in the original video frame are transparent or areas to be replaced. The human mask is a mask image used to identify the pixels containing the digital human subject in the video frame; each human mask corresponds one-to-one with a video frame. The background image is the content of the scene other than the digital human subject; the background image can be a preset static image or a dynamically changing image. Visual elements are additional display content superimposed on the screen; visual elements can be subtitles, logos, decorative patterns, etc. The target video frame sequence is a sequence of video frames that can be directly used for display, obtained by superimposing and synthesizing the original video frames, background image, human mask, and visual elements.

[0039] Audio frame objects and video frame objects are data encapsulation units compatible with real-time communication protocols. An audio frame object encapsulates a segment of audio sample data, while a video frame object encapsulates a frame of video data. Audio track objects and video track objects are media tracks used to send audio and video respectively in point-to-point real-time communication connections. An audio resampler is a processing device used to adjust the audio sampling rate and audio block duration.

[0040] The executing entity in this embodiment can be a server with data processing, network communication, and program execution functions, such as a standalone server, cloud server, or server cluster, or a computing service device capable of performing the above functions. The following description uses an interactive server as an example to illustrate this embodiment and the subsequent embodiments. The interactive server is a server-side device that deploys a real-time interactive method based on 2D digital humans. The interactive server and the client are connected via a network.

[0041] Based on this, this embodiment provides a real-time interaction method based on a 2D digital human, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the real-time interaction method based on a 2D digital human. In this embodiment, the real-time interaction method based on a 2D digital human includes steps S10-S50: Step S10: Obtain the user audio stream that the client collects and uploads in real time.

[0042] Real-time capture refers to the client capturing and uploading audio data simultaneously with the user's speech, rather than uploading the entire recording all at once after the user finishes speaking. The interactive server continuously receives the audio data uploaded by the client, forming a user audio stream. Because the user audio stream continuously arrives at the interactive server in a streaming manner, subsequent voice activity detection can be performed synchronously during the user's speech, avoiding additional delays introduced by waiting for the entire recording to be uploaded.

[0043] In one feasible implementation, the client is a browser page. After obtaining microphone access, the browser page creates an audio processing link through the audio context interface, converts the floating-point audio stream captured by the microphone into a 16-bit integer audio byte stream, segments it according to a fixed buffer size, and uploads the data packet containing the sampling rate and audio byte stream to the interactive server in real time via a WebSocket connection. The interactive server listens for audio events and receives data packets, parses the data packets to obtain the sampling rate and audio byte stream, and restores the audio byte stream to a floating-point audio signal normalized to the [-1,1] range to obtain the user audio stream. In another feasible implementation, the client is a mobile application. The mobile application encapsulates the captured audio data into real-time transmission data packets and uploads them to the interactive server via a real-time communication channel. The above are only two feasible implementations of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S10.

[0044] Step S20: Perform voice activity detection on the user audio stream, extract valid voice segments from the user audio stream, perform speech recognition on the valid voice segments to obtain the user's question text, and generate a response text based on the user's question text.

[0045] In addition to the user's voice, the user's audio stream also includes environmental noise and silent segments. Speech activity detection is performed first, followed by speech recognition. Environmental noise and silent segments are excluded from the speech recognition process; the input for speech recognition is only the signal range containing the valid human voice. Response text is generated based on the user's question, ensuring that the digital human's response is relevant to the user's question.

[0046] In one feasible implementation, the interactive server uses a speech activity detection model to detect the user's audio stream, outputs the start and end positions of speech segments, and extracts and concatenates valid speech segments from the user's audio stream according to the start and end positions. An automatic speech recognition model recognizes the valid speech segments to obtain the user's question text. A large language model understands the user's question text and generates a response text. For example, the automatic speech recognition model can use a Funasr model, and the inference block size parameter of the Funasr model can be set to [0, 10, 5]. The large language model can combine retrieval enhancement generation technology to generate the response text. For specific implementations of retrieval enhancement generation, please refer to the sixth embodiment below. In another feasible implementation, the interactive server uses an end-to-end speech understanding model. The speech understanding model directly understands the user's audio stream and outputs the user's question text. Speech activity detection and speech recognition are performed by the same model. The above are only two feasible implementations of step S20 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S20.

[0047] Since speech activity detection filters out environmental noise and silent segments, the computational cost and false recognition rate of speech recognition are reduced; since the response text is generated by a large language model, the digital human can perform open-domain question and answer, breaking through the answer range limitations of the preset question and answer database.

[0048] Step S30: Perform speech synthesis on the reply text to obtain the reply audio stream.

[0049] Speech synthesis is the process of converting a response text into a speech waveform sentence by sentence. Streaming speech synthesis generates and outputs the response audio stream simultaneously, without waiting for the entire response text to be synthesized.

[0050] In one feasible implementation, the interactive server performs speech synthesis on the reply text based on a large-model timbre cloning algorithm to generate a reply audio stream with a target timbre, which is the timbre corresponding to the digital human image, such as the timbre of a real person corresponding to the digital human image. For a specific implementation of the streaming speech synthesis based on the large-model timbre cloning algorithm, please refer to the seventh embodiment below.

[0051] In another feasible implementation, the interactive server uses a text-to-speech model with preset timbres to synthesize the reply text, and the preset timbres are selected from a timbre library. The above are merely two feasible implementations of step S30 provided in this embodiment; this embodiment does not specifically limit the specific implementation of step S30.

[0052] Because the response audio stream is generated in a streaming manner, subsequent lip-reading inference can be initiated simultaneously with speech synthesis, reducing the first packet latency of the digital human's response; and because timbre cloning is used, the timbre of the digital human's speech is consistent with the digital human character, improving the expressiveness of the digital human's speech and its fit with the character.

[0053] Step S40: Based on the response audio stream, drive the 2D digital human model to perform lip-reading inference, generate a digital human video frame sequence synchronized with the lip-reading of the response audio stream, and encapsulate the response audio stream and the digital human video frame sequence in time synchronization. Push the sequence to the client through a point-to-point real-time communication connection established with the client so that the client can play the digital human response.

[0054] The response audio stream serves as both an auditory output signal and a visual driving signal: one path of the response audio stream, after time-synchronized encapsulation, becomes the sound output of the digital human's response; the other path inputs to the 2D digital human model. The 2D digital human model infers the lip morphology of each video frame based on the speech content of the response audio stream, generating a sequence of digital human video frames. Time synchronization encapsulation establishes a unified time reference for both the response audio stream and the digital human video frame sequence, ensuring that the sound and lip movements remain aligned during client playback. Point-to-point real-time communication connects directly between the interactive server and the client, transmitting media data without retransmission through a central media server.

[0055] In one feasible implementation, the 2D digital human model employs the DINet lip-syncing model. The DINet lip-syncing model takes the digital human base model material and the response audio stream as input, and outputs a sequence of digital human video frames synchronized with the lip movements and speech content. The video frame rate is set to 25fps, the audio sampling rate is set to 16kHz, and the audio-visual synchronization error is less than 120 milliseconds. The point-to-point real-time communication connection uses a WebRTC connection. The WebRTC connection is established by a signaling service relaying session description information and candidate network information between the client and the interactive server; the signaling service itself does not transmit media data. For specific implementation methods of overlay synthesis and push, please refer to the eighth and ninth embodiments below. In another feasible implementation, the 2D digital human model uses the Wav2Lip lip-syncing model to generate the digital human video frame sequence.

[0056] Because the digital human video frame sequence is generated in real time by inferring from the 2D digital human model based on the response audio stream, the digital human's lip movements and speech content are accurately matched; because the media data is directly transmitted to the client via point-to-point real-time communication, there is no forwarding delay from the central media server on the transmission link, the end-to-end delay of the digital human's response can be controlled within the range required for real-time dialogue, and because the transmission process combines network adaptation and forward error correction mechanisms, the picture remains stable.

[0057] In step S50, in response to detecting valid speech from the user's audio stream during the generation and push of the digital human response, the current digital human response generation task is stopped, and the data queue of the intermediate data of the cached digital human response is cleared to switch to the listening state of receiving user input.

[0058] The generation of digital human responses and the detection of user audio streams occur in parallel: while generating and pushing digital human responses, the interaction server continuously detects speech activity in the user's audio stream. Intermediate data from the digital human responses is cached in various data queues, awaiting consumption by subsequent stages. When the user begins to speak during a digital human response, valid speech is detected in the user's audio stream. The interaction server then stops the current digital human response generation task; the response text is no longer synthesized into speech, and the response audio stream no longer drives lip-reading inference. Simultaneously, intermediate data cached in each data queue belonging to interrupted responses is cleared and discarded, and interrupted responses are no longer output. The interaction server then enters a listening state, ready to receive new questions from the user.

[0059] If the digital human's response is not interrupted, the digital human will continue to speak while the user asks a question, causing the user's question and the digital human's response to overlap and impair the interactive experience. By stopping the generation of tasks and clearing the data queue, the digital human's response stops immediately after valid voice is detected, realizing an interactive method that simulates real-person dialogue with real-time interruption.

[0060] In one feasible implementation, the interactive server can broadcast interruption events to each media generation process via a global dialogue state variable. Each media generation process listens to the global dialogue state variable and performs stop and clear operations. In another feasible implementation, the interactive server sends interrupt commands to each media generation process via an interruption control unit. Upon receiving the interrupt command, each media generation process stops its current data generation task and clears the corresponding output data queue.

[0061] This embodiment overcomes the technical problem of high latency in digital human interaction response by adopting end-to-end streaming processing and point-to-point direct push technology, thereby achieving the technical effect of low-latency output of digital human responses; by adopting interruption detection and data queue clearing technology, it overcomes the technical problem of not supporting real-time interruption during the digital human response process, thereby achieving the technical effect that users can interrupt the digital human response at any time.

[0062] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter. In this embodiment, the real-time interaction method based on 2D digital humans is executed collaboratively by multiple independent processes. The independent processes communicate with each other through a first-in-first-out data queue. The data queue includes at least a user audio input queue, a response text queue, a synthesized audio queue, an original video frame queue, an overlay synthesized video frame queue, a push audio queue, and a push video frame queue. In the digital human response generation chain, the processing speed of each link is different: the speed of generating response text by the large language model, the speed of speech synthesis, the speed of lip-reading inference, and the speed of push differ. If each link calls directly, the slower link will block the faster link. The independent processes communicate with each other through the first-in-first-out data queue. The writer and the reader do not wait for each other, and each process runs at its own speed. The data queue buffers the difference in processing speed.

[0063] In this embodiment, after step S10: obtaining the user audio stream collected and uploaded by the client in real time, the method further includes: step S60, writing the user audio stream into the user audio input queue for reading by voice activity detection and speech recognition processing. The user audio input queue buffers the user audio stream received from the client, and voice activity detection and speech recognition processing read audio data from the user audio input queue in a first-in-first-out order.

[0064] Following step S20, which generates response text based on the user's question text, the process further includes step S70: writing the response text into a response text queue, having the speech synthesis process read the response text from the response text queue and generate a response audio stream, and then writing the response audio stream into a synthesis audio queue. The response text queue caches the response text generated by the large language model, serving as the data source for the speech synthesis process; the synthesis audio queue caches the response audio stream generated by the speech synthesis process, serving as the data source for the digital human generation and rendering process.

[0065] Step S40: The step of generating a digital human video frame sequence lip-synced with the response audio stream includes: Step S401, where the digital human generation and rendering process reads the response audio stream from the composite audio queue, generates original video frames containing only the digital human subject, and writes them into the original video frame queue; the video frame overlay process retrieves the original video frames from the original video frame queue and overlays them with a background image, writing the resulting composite video frame into the overlay composite video frame queue. The original video frame queue buffers digital human video frames without background overlay, and the overlay composite video frame queue buffers the composite video frames after background overlay.

[0066] Step S40: The step of time-synchronized encapsulation of the response audio stream and the digital human video frame sequence includes: Step S402, where the audio and video stream push process sequentially retrieves audio data and video frames from the synthesized audio queue and the overlay synthesized video frame queue, performs audio-visual synchronization encapsulation, and then sends them to the media stream track of the point-to-point real-time communication connection via the push audio queue and the push video frame queue, respectively. The dequeue order and timestamp of the push audio queue and the push video frame queue are synchronized, and the audio and video are sent to the media stream track according to a unified time base.

[0067] In one feasible implementation, each data queue is a first-in-first-out (FIFO) queue in memory, with each independent process running on the same interactive server. Writing and reading from the queues are implemented through intra-process or inter-process memory data structures. In another feasible implementation, each data queue is a distributed message queue, with each independent process distributed across multiple servers. Writing and reading from the queues are implemented through a message middleware. These are merely two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the specific implementation form of the data queues.

[0068] In this embodiment, the production and consumption stages of each independent process are decoupled through a data queue, so that fluctuations in the processing speed of any process will not be directly transmitted to other processes; the unidirectional data flow avoids competition for the same data by multiple processes, thus avoiding deadlock; and when an interruption event occurs, clearing the data queue can discard all intermediate data of the interrupted response, providing a unified operation object for interruption handling.

[0069] In one feasible embodiment, step S20: the step of detecting speech activity in the user audio stream and extracting valid speech segments from the user audio stream, includes steps S201~S203: Step S201: Continuously acquire audio data blocks from the user audio input queue and cache them. When the cached audio duration exceeds the preset detection duration, input the cached audio data into the speech activity detection model to obtain a timestamp list containing the start and end positions of speech segments.

[0070] Speech activity detection models require processing audio data of a certain length to obtain stable detection results. Audio data blocks are continuously retrieved from the user's audio input queue and cached. Once the cached audio duration reaches a preset detection duration, the cached audio data is used as input for a single detection. For example, the preset detection duration is 0.6 seconds. The speech activity detection model employs the Silo VAD model, which outputs the start and end times of each speech segment in the cached audio data, forming a timestamp list.

[0071] Step S202: Map the timestamp list to the cached audio data, extract each valid speech segment and splice them into a continuous signal, and calculate the total effective speech duration of the spliced ​​continuous signal.

[0072] The start and end positions in the timestamp list correspond to the sampling point positions in the cached audio data. Each speech segment is retrieved from the cached audio data according to its start and end positions, and the speech segments are concatenated into a continuous signal in chronological order. The total duration of the continuous signal is the total effective speech duration.

[0073] Step S203: When the total effective speech duration exceeds the preset start-of-speech threshold, it is determined that the user has started speaking, and the audio data thereafter is cached; when the total effective speech duration is less than the preset stop-of-speech threshold, it is determined that the user has finished speaking, and the cached audio data is used as a valid speech segment.

[0074] If the total effective speech duration exceeds the start-of-speech threshold, it indicates the presence of continuous human voice rather than transient background noise in the current audio. Audio data from this point onward is continuously cached, forming the user's speech content for this round. If the total effective speech duration is less than the stop-of-speech threshold, it indicates the human voice has ended, and the cached audio data is output as a valid speech segment. After each detection process, the cached audio data is cleared, and it is re-cached for the next detection round.

[0075] In one feasible implementation, the speech activity detection model employs a deep learning-based Silo VAD model; in another feasible implementation, speech activity detection is achieved by calculating the short-time energy and zero-crossing rate of audio data and comparing them with a threshold. These are merely two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the specific implementation method of speech activity detection.

[0076] This embodiment detects audio by caching fixed-length audio and extracts spliced ​​speech segments by timestamp, ensuring accurate boundary positioning of effective speech. By using dual threshold determination of start and stop speaking thresholds, transient noise is filtered out, and the start and end determination of user speech is stable, providing reliable input for interruption handling and speech recognition.

[0077] In one feasible embodiment, step S20: the step of generating reply text based on the user's question text, includes steps S204~S205: Step S204: Based on retrieval enhancement generation technology, the input of a large language model is constructed by combining professional knowledge retrieved from the knowledge base with the user's question text. The large language model then generates incremental text of the response text in a streaming output mode.

[0078] A knowledge base is a pre-built collection of professional knowledge, and the knowledge content in the knowledge base can be updated according to business needs. For example, a knowledge base could be a financial product knowledge base, storing information such as financial product introductions and business processing rules. The retrieval-enhanced generation process is as follows: based on the user's query text, relevant knowledge content is retrieved from the knowledge base. The retrieved knowledge content, along with the user's query text, is used as input to the large language model. The large language model then combines this professional knowledge to generate the response text. The streaming output mode means that the large language model outputs a piece of text as it generates it, rather than generating all the response text at once. Each newly added text fragment is considered an incremental text.

[0079] Step S205: When receiving incremental text, use regular expressions to perform semantic segmentation on the incremental text, identify complete sentences and output them to the response text queue in real time, and temporarily store the contents of unclosed sentences in the buffer.

[0080] The boundaries of incremental text segments are often not the same as sentence boundaries; a sentence may be split into multiple incremental texts for output. Regular expressions are used to match sentence-ending markers, such as periods, question marks, exclamation marks, and other punctuation marks. After receiving incremental text, the portions of the received text that match sentence-ending punctuation are considered complete sentences, and these complete sentences are output to the response text queue in real time for the speech synthesis process to read. The content after the last sentence-ending punctuation mark is considered an unclosed sentence and is temporarily stored in a buffer, to be concatenated with subsequent incremental texts until a sentence-ending punctuation mark is matched.

[0081] In one feasible implementation, regular expressions are used to match punctuation marks at the end of sentences for semantic segmentation; in another feasible implementation, a semantic segmentation model is used to determine the completeness of the sentence. These are merely two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the specific implementation method of semantic segmentation.

[0082] In this embodiment, the response text is generated and output to the response text queue at the granularity of complete sentences. The speech synthesis process can start synthesis as soon as the complete sentence is output, without waiting for the entire response text to be generated. The first packet delay of the digital human's response is reduced. At the same time, professional knowledge in the knowledge base participates in the generation of the response text. The digital human's answer breaks through the scope limitation of the preset question and answer database. Moreover, the knowledge base can be updated according to business needs, and the knowledge maintenance cost is low.

[0083] Based on the embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 In this embodiment, the real-time interaction method based on 2D digital humans further includes: Step S80: Maintain global dialogue state variables, wherein the global dialogue state variables include at least a first state variable indicating whether the user is in a speaking state, and a second state variable indicating whether the user is in a digital human response generation and output state.

[0084] The first and second state variables are Boolean variables. The first state variable takes the value True, indicating that the user is speaking; the fourth state variable takes the value False, indicating that the user has finished speaking or has not spoken. The second state variable takes the value True, indicating that the interaction server is in the state of generating and outputting digital human responses, and media generation processes such as speech synthesis, lip-reading inference, and overlay are running; the third state variable takes the value False, indicating that the digital human response has stopped, and the interaction server is in a listening and waiting state. For example, the first state variable can be a user_talking variable, and the second state variable can be a responding variable.

[0085] In this embodiment, step S50: In response to detecting valid speech from the user's audio stream during the generation and push of the digital human response, stopping the current digital human response generation task and clearing the data queue of the intermediate data of the cached digital human response, includes steps S501~S503: Step S501: When the effective speech duration detected from the user's audio stream exceeds the preset threshold for starting to speak, the first state variable is set to a first value representing that the user is speaking, the user's audio is buffered, and the second state variable is switched from a second value representing that the user is responding to a third value representing that the user is stopping responding.

[0086] If the effective speech duration exceeds the threshold for initiating speech, it indicates that the user's speech is not merely brief ambient noise but a statement with a clear intention. At this point, it is determined that the user intends to ask a question or interrupt. After the first state variable is set to its first value, subsequent user audio is continuously cached, and the cached audio constitutes the speech content for the user's next round of questions. The second state variable switches to its third value; this change in the second state variable acts as a broadcast instruction, affecting all media generation processes.

[0087] In step S502, each media generation process listens to the second state variable in its own main loop. When the second state variable changes from the second value to the third value, the current data generation task is stopped, and the corresponding output data queue is cleared, discarding the intermediate data belonging to the interrupted digital human response.

[0088] The media generation process includes speech synthesis, digital human generation and rendering, and video frame overlay. Each media generation process reads the value of a second state variable in each task loop: when the second state variable is at the second value, the media generation process executes the data generation task normally; when the second state variable changes from the second value to the third value, the media generation process immediately terminates the currently executing data generation task and clears the corresponding output data queue. For example, the speech synthesis process clears the synthesized audio queue, the digital human generation and rendering process clears the original video frame queue, and the video frame overlay process clears the overlay synthesized video frame queue. Intermediate data belonging to interrupted responses is discarded, releasing system resources, and outdated data will not continue to flow to the client.

[0089] Step S503: When the effective speech duration detected from the user audio stream is less than the preset stop speaking threshold, the first state variable is set to the fourth value representing the user's speech has ended, and the cached user audio is sent as an effective speech segment to the speech recognition and reply text generation process.

[0090] If the effective speech duration is less than the stop-speak threshold, it indicates that the user's speech has ended, and the cached user audio is the complete speech content of the user's question in this round. After the cached user audio is sent as a valid speech segment to the speech recognition and response text generation processing, the interaction server generates a new digital human response for the next round of questions, and the second state variable is reset to the second value.

[0091] In one feasible implementation, the start-of-speech threshold is set to 0.2 seconds, and the stop-of-speech threshold is set to 0.1 seconds. A shorter start-of-speech threshold ensures that interruption detection is triggered quickly after the user speaks, and a shorter stop-of-speech threshold ensures that speech recognition starts as soon as the user finishes speaking. In one feasible implementation, each media generation process monitors the second state variable via polling; in another feasible implementation, changes in the global dialogue state variable are pushed to each media generation process via an event publish-subscribe mechanism. These are merely two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the monitoring method.

[0092] This embodiment coordinates the start and stop of each media generation process through a global dialogue state variable. Interruption events do not require the transmission of processing instructions level by level. Each media generation process responds synchronously based on the same state identifier, resulting in fast response speed for interruption processing and consistent state among all processes.

[0093] In one feasible embodiment, after step S501: switching the second state variable from a second value representing a response to a third value representing a cessation of response, the method further includes steps S504-S505: Step S504: When the second state variable is the third value, continuously generate silent audio blocks of a preset duration and write the silent audio blocks into the synthesized audio queue that caches the synthesized audio data.

[0094] After the digital human stops responding, if no more data is pushed to the media stream track, the digital human's screen on the client will freeze on the last frame, appearing as a static, unresponsive figure. To maintain screen continuity, when the second state variable reaches the third value, the speech synthesis process does not synthesize reply text but continuously generates silent audio blocks. The duration of the silent audio block is a preset duration; for example, the duration of the silent audio block is 1.28 seconds, the sampling rate is 16kHz, and all sample values ​​of the silent audio block are zero.

[0095] Step S505: Based on the silent audio block, drive the 2D digital human model to perform lip shape inference, generate a digital human video frame sequence with closed lips, and push the silent audio block and the digital human video frame sequence with closed lips to the client after time synchronization and encapsulation.

[0096] The lip-reading inference of the 2D digital human model is driven by the input audio: when the input audio contains human voice, the 2D digital human model generates lip opening and closing shapes corresponding to the human voice; when the input audio is a silent audio block, there is no speech content in the silent audio block, and in the digital human video frame sequence generated by the 2D digital human model, the digital human's lips are in a closed state. After the lip-closed digital human video frame sequence is time-synchronized and encapsulated with the silent audio block, it is continuously pushed to the client via the media stream track, and the client's screen continues to play continuously, presenting the digital human as a listening image with closed lips. The natural movements of the digital human's body, limbs, and head are provided by the images in the digital human base model material; the difference between the listening state and the speaking state lies in whether the lips open and close with speech.

[0097] In one feasible implementation, the silence audio block is generated cyclically by the speech synthesis process according to a preset duration. Each generated silence audio block is written to the synthesized audio queue until the second state variable is reset to the second value. In another feasible implementation, the interactive server pre-generates a video segment of the digital human's listening state. After an interruption occurs, the server directly reads the video segment of the listening state and encapsulates it with the silence audio before pushing it to the client. The above are only two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the method of generating the listening state.

[0098] In this embodiment, after the interruption occurs, the digital human naturally transitions from a speaking lip shape to a listening state with closed lips. The image remains continuous and the digital human image maintains natural movement, clearly indicating to the user that the system has switched to the mode of receiving user input. This avoids the stiffness caused by the image freezing, and improves the naturalness of the interaction during the interruption transition.

[0099] Based on the embodiments of this application, in the fourth embodiment of this application, the content that is the same as or similar to the embodiments described above can be referred to the above description, and will not be repeated hereafter. In this embodiment, please refer to Figure 3 Step S30: The step of performing speech synthesis on the reply text to obtain the reply audio stream includes steps S301~S302: Step S301: The reply text is segmented into short sentences, and the segmented short sentences are synthesized using a large model timbre cloning algorithm to generate audio data with the target timbre.

[0100] Sentence granularity refers to the text granularity based on complete sentences or short sentences. After segmenting the response text into sentence granularity, the large-scale model voice cloning algorithm independently synthesizes speech for each sentence, and the synthesis of each sentence can be performed sequentially in a pipeline manner. The large-scale model voice cloning algorithm synthesizes speech with the target voice timbre based on the reference audio of the target voice timbre. For example, the large-scale model voice cloning algorithm uses the indexTTS algorithm, and the target voice timbre is the voice corresponding to the digital human character, such as the voice of a real person corresponding to the digital human character.

[0101] Step S302: The audio data is transmitted in the original pulse code modulation format through an asynchronous streaming interface, in audio blocks of a preset duration range, combined with a block transmission encoding mechanism, to obtain the response audio stream.

[0102] Raw pulse code modulation (RCM) is an audio sample data format without a file header. For example, RCM can be the audio / L16 format. Compared to waveform file formats, RCM eliminates the header encapsulation process, reducing initial packet latency and transmission overhead. The asynchronous streaming interface allows audio data to be generated and transmitted simultaneously: the audio data synthesized from each short text phrase is divided into audio blocks of a preset duration, and these blocks are transmitted sequentially using a chunked transmission encoding mechanism.

[0103] In one feasible implementation, audio blocks are segmented according to a fixed duration; in another feasible implementation, audio blocks are dynamically segmented according to short sentence boundaries, with each audio block corresponding to a synthesized audio of a short sentence. These are merely two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the segmentation method of audio blocks.

[0104] In this embodiment, timbre cloning enables the digital human's voice to present a timbre consistent with the digital human's role, improving the expressiveness of the digital human's voice and its fit with the role; streaming transmission enables the synthesized audio data to enter subsequent stages in real time, which is suitable for continuous voice output scenarios such as long text and multi-turn dialogue, and further reduces the first packet delay of the digital human's response.

[0105] In one feasible embodiment, after step S40: generating a sequence of digital human video frames lip-synced with the response audio stream, the method further includes: Step S90: The generated original video frame containing only the digital human subject is superimposed and synthesized with a preset or dynamic background image, a person mask for each frame, and visual elements to obtain a target video frame sequence for display; In this embodiment, step S40: the step of time-synchronized encapsulation of the response audio stream and the digital human video frame sequence includes: step S403: time-synchronized encapsulation of the response audio stream and the target video frame sequence.

[0106] In the original video frames generated during the digital human generation and rendering process, only the pixels containing the digital human subject are valid pixels, while the areas outside the digital human subject are transparent. A human mask is used to identify the pixel position of the digital human subject in the original video frame. For example, the human mask is a binary image of the same size as the original video frame, with the pixel containing the digital human subject set to 1 and the remaining pixels set to 0. The overlay compositing process is as follows: for each original video frame, the pixels of the digital human subject are overlaid onto the corresponding positions of the background image according to the human mask, and then visual elements such as subtitles and logos are overlaid to obtain a target video frame. The background image can be a preset static image, such as a fixed scene image; or it can be a dynamically changing image, such as a scene image that switches according to the interactive scene.

[0107] In one feasible implementation, the overlay synthesis employs a mask-weighted fusion method, where the pixels of the digital human subject are weighted and fused according to the human mask and the pixels of the background image, resulting in a smooth edge transition for the digital human subject. In another feasible implementation, the overlay synthesis employs a direct replacement method, where the pixel positions marked by the human mask are directly replaced by pixels of the digital human subject with pixels of the background image. These are merely two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the specific implementation method of the overlay synthesis.

[0108] In this embodiment, the digital human subject and the background image are generated separately and then superimposed and synthesized. The digital human image and the background image can be replaced independently. The same digital human image can be adapted to different display scenarios, and the digital human assets have a high degree of reusability. The human mask code identifies the position of the digital human subject frame by frame, and the edges of the superimposed and synthesized image are accurate.

[0109] In one feasible embodiment, step S, which involves time-synchronized encapsulation of the response audio stream and the digital human video frame sequence and pushing them to the client via a peer-to-peer real-time communication connection, includes steps S404-S405: In step S404, the audio and video stream push module starts a background worker thread to sequentially retrieve audio data blocks and video frames from the data queue of cached audio data and video frames, and encapsulate the audio data blocks and video frames into audio frame objects and video frame objects that are compatible with the real-time communication protocol.

[0110] The audio / video stream push module is the output module that interfaces with the interactive server and the point-to-point real-time communication connection. For example, the audio / video stream push module can be a StreamPusher module, and the background worker thread can be a stream_worker_thread. In a specific implementation, video frame objects are encapsulated in BGR (Blue-Green-Red) 24 format, and audio frame objects are processed in 16-bit mono pulse code modulation format. The sampling rate and duration are unified through an audio resampler. BGR24 format is an image format where each pixel is represented by 8 bits for each of the blue, green, and red color components; 16-bit mono pulse code modulation format is a mono audio format quantized with 16 bits per sample point. The audio resampler unifies the sampling rate and duration of audio data blocks to target parameters, such as a sampling rate of 16kHz. The duration of each audio data block is adapted to the video frame interval to ensure the stability of continuous audio playback.

[0111] In step S405, the encapsulated audio frame object and video frame object are pushed to the audio track object and video track object of the point-to-point real-time communication connection respectively through an asynchronous queue, so as to be sent to the client through the underlying transmission channel.

[0112] Audio track objects and video track objects are media tracks that carry audio and video respectively in a point-to-point real-time communication connection. For example, the audio track object and video track object can be PlayStreamTrack track objects in a WebRTC connection, and the asynchronous queue can be an asyncio.Queue queue (a class name). After the encapsulated audio frame object and video frame object are sent to the track object via the asynchronous queue, the track object sends them to the client via the underlying transmission channel, which can be a transmission channel based on a real-time transmission protocol.

[0113] In one feasible implementation, audio frame objects and video frame objects are timestamped using a unified time base and sample count: the video frame rate is set to 25fps, the audio sampling rate is set to 16kHz, and the number of audio samples per frame is adapted to the video frame interval, i.e., 640 audio samples per 40 milliseconds; simultaneously, the queue length of audio track objects and video track objects is controlled to not exceed a preset number of frames, for example, not exceeding 10 frames. When the queue length reaches the preset number of frames, the incoming data is rate-limited or discarded to prevent data backlog from increasing playback latency. The background worker thread continues to run until a stop signal is received. In another feasible implementation, the timestamps of audio frame objects and video frame objects are uniformly generated by the system clock. The above are only two feasible implementation methods provided in this embodiment, and this embodiment does not specifically limit the specific implementation method of timestamp alignment.

[0114] In this embodiment, audio and video are encapsulated into protocol-compatible frame objects on the interactive server and pushed on separate tracks. Audio-video synchronization is uniformly guaranteed by the interactive server, reducing audio-video synchronization errors. Controlling the track queue length prevents data backlog, and the playback latency of the digital human's audio and video streams remains stable.

[0115] For example, to help understand the implementation process of the real-time interaction method based on 2D digital humans obtained by combining this embodiment with the above embodiment one, please refer to... Figure 4 , Figure 4 A simplified flowchart of a real-time interaction method based on 2D digital humans is provided, specifically: Step 1: Connection Establishment and Audio Acquisition. The client establishes a connection with the signaling service. The signaling service relays session description information and candidate network information between the client and the interactive server. A WebRTC peer-to-peer media data channel is established between the client and the interactive server. The signaling service does not transmit media data. The client acquires user audio in real time and uploads it via a WebSocket connection (a computer network protocol). The interactive server writes the user audio stream into the user audio input queue.

[0116] Step two, speech detection and recognition. The interactive server continuously retrieves and caches audio data blocks from the user's audio input queue. When the cached audio duration exceeds 0.6 seconds, it is input into the Silero VAD model to obtain a timestamp list. Valid speech segments are extracted and concatenated, and the total duration of valid speech is calculated. When the total duration of valid speech exceeds the 0.2-second threshold for starting to speak, the user is considered to have started speaking; when it falls below the 0.1-second threshold for stopping to speak, the user is considered to have finished speaking. The cached audio data is then used as a valid speech segment, and the Funasr model identifies the user's question text.

[0117] Step 3: Response Text Generation. Based on retrieval-enhanced generation technology, the input of a large language model is constructed by combining professional knowledge retrieved from the knowledge base with the user's question text. The large language model streams incremental text, which is then semantically segmented using regular expressions to obtain complete sentences, which are then output to the response text queue in real time.

[0118] Step four: Speech synthesis and lip-reading inference. The speech synthesis process reads the response text from the response text queue, segments it into short sentences, and synthesizes audio data with the target timbre using the indexTTS timbre cloning algorithm. This data is then streamed into the synthesized audio queue in 0.5- to 3-second audio blocks using the original pulse code modulation format. The digital human generation and rendering process reads the response audio stream from the synthesized audio queue, drives the DINet model to generate original video frames containing only the digital human subject, and writes them into the original video frame queue. The video frame overlay process overlays the original video frames with the background image, the human mask, and visual elements, and writes the composite video frame into the overlay queue.

[0119] Step 5: Synchronous Encapsulation and Push. The audio and video stream push process sequentially retrieves audio data and video frames from the synthesized audio queue and the overlay synthesized video frame queue, encapsulates them into audio frame objects and video frame objects, aligns the timestamps using a unified time base and sample count, and sends them to the audio track and video track of the WebRTC connection via the push audio queue and push video frame queue respectively, and pushes them to the client to play the digital human response.

[0120] Step Six: Interruption Handling. During the generation and delivery of the digital human's response, voice activity detection continues. When the total effective voice duration exceeds the threshold for starting to speak, the first state variable is set to its first value, the second state variable switches to its third value, and each media generation process stops its current data generation task and clears its respective output data queue. The speech synthesis process continuously generates a 1.28-second silent audio block, and the 2D digital human model generates a sequence of digital human video frames with lip closure based on the silent audio block, transitioning the digital human into a listening state. After the user finishes speaking, the cached user audio is sent as a valid voice segment to the speech recognition and response text generation process to begin a new round of interaction.

[0121] The above examples are only for understanding this embodiment and do not constitute a limitation on the real-time interaction method based on 2D digital humans in this embodiment. Many simple modifications based on this technical concept are within the protection scope of this embodiment. It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the real-time interaction method based on 2D digital humans in this application. Many simple modifications based on this technical concept are within the protection scope of this application.

[0122] This application also provides a real-time interactive device based on a 2D digital human. The real-time interactive device based on a 2D digital human provided in this application, employing the real-time interactive method based on a 2D digital human in the above embodiments, can solve the technical problem of high response latency in digital human interaction, leading to low naturalness of interaction. Compared with the prior art, the beneficial effects of the real-time interactive device based on a 2D digital human provided in this application are the same as the beneficial effects of the real-time interactive method based on a 2D digital human provided in the above embodiments, and other technical features in the real-time interactive device based on a 2D digital human are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0123] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the real-time interactive method based on a 2D digital human as described in Embodiment 1 above.

[0124] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0125] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0126] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0127] The electronic device provided in this application employs the real-time interaction method based on 2D digital humans in the above embodiments, which can solve the technical problem of high response latency in digital human interaction, resulting in low naturalness of interaction. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the real-time interaction method based on 2D digital humans provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0128] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0129] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0130] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the real-time interactive method based on a 2D digital human in the above embodiments.

[0131] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0132] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0133] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to implement the various embodiments of the aforementioned real-time interaction method based on 2D digital humans.

[0134] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0137] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described real-time interaction method based on 2D digital humans. This solves the technical problem of high response latency in digital human interaction, leading to low naturalness of interaction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the real-time interaction method based on 2D digital humans provided in the above embodiments, and will not be repeated here.

[0138] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the real-time interactive method based on a 2D digital human as described above.

[0139] The computer program product provided in this application can solve the technical problem of high response latency in digital human interaction, resulting in low naturalness of interaction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the real-time interaction method based on 2D digital humans provided in the above embodiments, and will not be repeated here.

[0140] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A real-time interaction method based on 2D digital humans, characterized in that, The real-time interaction method based on 2D digital humans includes: Acquire user audio streams that are collected and uploaded in real time by the client; The user audio stream is subjected to voice activity detection, valid voice segments are extracted from the user audio stream, the valid voice segments are subjected to speech recognition to obtain the user's question text, and a response text is generated based on the user's question text. The reply text is processed by speech synthesis to obtain a reply audio stream; Based on the response audio stream, a 2D digital human model is driven to perform lip-reading inference, generating a digital human video frame sequence synchronized with the lip-reading of the response audio stream. The response audio stream and the digital human video frame sequence are then time-synchronized and encapsulated, and pushed to the client through a point-to-point real-time communication connection established with the client, so that the client can play the digital human response. In response to the generation and push of the digital human response, if valid speech is detected from the user's audio stream, the current digital human response generation task is stopped, and the data queue that caches the intermediate data of the digital human response is cleared, so as to switch to the listening state of receiving user input.

2. The real-time interaction method based on 2D digital humans as described in claim 1, characterized in that, The real-time interaction method based on 2D digital human is executed collaboratively by multiple independent processes. The independent processes communicate with each other through a first-in-first-out data queue. The data queue includes at least a user audio input queue, a reply text queue, a synthesized audio queue, a raw video frame queue, an overlay synthesized video frame queue, a push audio queue, and a push video frame queue. After the step of acquiring the user audio stream collected and uploaded by the client in real time, the method further includes: writing the user audio stream into the user audio input queue for reading by voice activity detection and voice recognition processing; After the step of generating reply text based on the user's question text, the method further includes: writing the reply text into the reply text queue, having the speech synthesis process read the reply text from the reply text queue and generate a reply audio stream, and writing the reply audio stream into the synthesized audio queue; The step of generating a sequence of digital human video frames that is lip-synced with the response audio stream includes: the digital human generation and rendering process reading the response audio stream from the synthesized audio queue, generating an original video frame containing only the digital human subject and writing it into the original video frame queue; the video frame overlay process retrieving the original video frame from the original video frame queue and overlaying it with a background image, and writing the synthesized video frame into the overlay synthesized video frame queue. The step of time-synchronized encapsulation of the response audio stream and the digital human video frame sequence includes: the audio and video stream push process sequentially extracts audio data and video frames from the synthesized audio queue and the overlay synthesized video frame queue, performs audio-visual synchronization encapsulation, and sends them to the media stream track of the point-to-point real-time communication connection through the push audio queue and the push video frame queue, respectively.

3. The real-time interaction method based on 2D digital humans as described in claim 2, characterized in that, The step of performing voice activity detection on the user audio stream and extracting valid voice segments from the user audio stream includes: Audio data blocks are continuously acquired from the user audio input queue and cached. When the cached audio duration exceeds the preset detection duration, the cached audio data is input into the speech activity detection model to obtain a timestamp list containing the start and end positions of speech segments. The timestamp list is mapped to the cached audio data, each valid speech segment is extracted and spliced ​​into a continuous signal, and the total effective speech duration of the spliced ​​continuous signal is calculated. When the total duration of the effective speech exceeds the preset threshold for starting to speak, it is determined that the user has started speaking, and the audio data thereafter is cached; when the total duration of the effective speech is less than the preset threshold for stopping to speak, it is determined that the user has finished speaking, and the cached audio data is used as the effective speech segment.

4. The real-time interaction method based on 2D digital humans as described in claim 2, characterized in that, The step of generating reply text based on the user's question text includes: Based on retrieval-enhanced generation technology, the input of a large language model is constructed by combining professional knowledge retrieved from the knowledge base with the user's question text. The large language model then generates incremental text of the response text in a streaming output mode. When receiving the incremental text, regular expressions are used to perform semantic segmentation on the incremental text, identify complete sentences and output them to the response text queue in real time, and temporarily store the contents of unclosed sentences in a buffer.

5. The real-time interaction method based on 2D digital humans as described in claim 1, characterized in that, The real-time interaction method based on 2D digital humans also includes: Maintain global dialogue state variables, wherein the global dialogue state variables include at least a first state variable indicating whether the user is in a speaking state, and a second state variable indicating whether the user is in a digital human response generation and output state. The step of responding to the generation and push of the digital human response by detecting valid speech from the user's audio stream, stopping the current digital human response generation task, and clearing the data queue that caches the intermediate data of the digital human response includes: When the effective speech duration detected from the user's audio stream exceeds a preset threshold for starting to speak, the first state variable is set to a first value representing that the user is speaking, the user's audio is buffered, and the second state variable is switched from a second value representing that the user is responding to a third value representing that the user is stopping responding. Each media generation process listens to the second state variable in its own main loop. When the second state variable switches from the second value to the third value, the current data generation task is stopped, and the corresponding output data queue is cleared, discarding the intermediate data belonging to the interrupted digital human response. When the effective speech duration detected from the user audio stream is less than the preset stop speaking threshold, the first state variable is set to the fourth value representing the user's speech has ended, and the cached user audio is sent as the effective speech segment to the speech recognition and reply text generation process.

6. The real-time interaction method based on 2D digital humans as described in claim 5, characterized in that, After the step of switching the second state variable from a second value representing a response to a third value representing a cessation of response, the method further includes: When the second state variable is the third value, a silent audio block of preset duration is continuously generated, and the silent audio block is written into the synthesized audio queue of the cached synthesized audio data; Based on the silent audio block, the 2D digital human model is driven to perform lip shape inference, generate a sequence of digital human video frames with closed lips, and then the silent audio block and the sequence of digital human video frames with closed lips are time-synchronized, encapsulated, and pushed to the client.

7. The real-time interaction method based on 2D digital humans as described in claim 1, characterized in that, The step of performing speech synthesis on the reply text to obtain a reply audio stream includes: The reply text is segmented into short sentences, and the segmented short sentences are synthesized using a large model timbre cloning algorithm to generate audio data with the target timbre. The audio data is transmitted in the original pulse code modulation format through an asynchronous streaming interface, in audio blocks of a preset duration range, combined with a block transmission encoding mechanism, to obtain the response audio stream.

8. The real-time interaction method based on 2D digital humans as described in claim 1, characterized in that, After the step of generating a sequence of digital human video frames that is lip-synced with the response audio stream, the method further includes: The generated original video frames containing only the digital human subject are superimposed and synthesized with preset or dynamic background images, the human mask for each frame, and visual elements to obtain the target video frame sequence for display. The step of time-synchronizing and encapsulating the response audio stream with the digital human video frame sequence includes: The response audio stream is time-synchronized and encapsulated with the target video frame sequence.

9. The real-time interaction method based on 2D digital humans as described in claim 1, characterized in that, The step of time-synchronizing and encapsulating the response audio stream with the digital human video frame sequence, and pushing it to the client through a peer-to-peer real-time communication connection established with the client, includes: The background worker thread is started by the audio and video stream push module. It retrieves audio data blocks and video frames sequentially from the data queue of cached audio data and video frames. The audio data blocks and video frames are encapsulated into audio frame objects and video frame objects that are compatible with the real-time communication protocol. The video frame objects are encapsulated in BGR24 format, and the audio frame objects are processed in 16-bit mono pulse code modulation format. The sampling rate and duration are unified by the audio resampler. The encapsulated audio frame object and video frame object are pushed to the audio track object and video track object of the point-to-point real-time communication connection respectively through an asynchronous queue, so as to be sent to the client via the underlying transmission channel.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the real-time interactive method based on a 2D digital human as described in any one of claims 1 to 9.

Images