Digital human interaction system, method, device, storage medium and program product

Abstract

Embodiments of the present application provide a digital human interaction system, method, device, storage medium and program product. The digital human interaction system is based on a hierarchical architecture design, including a scheduling layer, a coordination layer, a processor layer, a data interaction layer and a channel layer, etc. Each layer cooperates with each other to form a processing pipeline for interaction requests, thereby improving the processing efficiency and response speed of the system. On this basis, the system encapsulates the reply audio and the lip feature data corresponding to the interaction request into the same binary data and sends it to the client, thereby avoiding the arrival time difference of different packet sending and the long time-consuming problem of analysis. In the binary data packet, the lip feature data is located before the reply audio and is aligned according to a preset storage granularity, thereby facilitating the client to preferentially analyze the lip feature data to render the lip animation, and solving the problem of audio and picture out of sync. The digital human interaction system in the present application improves the real-time interaction of the digital human, reduces the end-to-end response delay, and enhances the user interaction experience.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to a digital human interaction system, method, device, storage medium and program product. Background Technology

[0002] In digital human application scenarios, users can initiate one-on-one (1V1) real-time interaction with the digital human through the client. The server can send pre-generated explanatory audio and digital human video to the client, which will then render the digital human explaining the product based on the audio and output it to the user.

[0003] During the product demonstration by the digital human, users can interact with it. The server can generate response audio and lip-shape feature data (required for the digital human to output the response audio) based on the user's voice or text, and send both to the client. Upon receiving the response audio and lip-shape feature data, the client can render a video of the digital human based on the lip-shape feature data and simultaneously play the response audio, achieving a natural and realistic effect where the digital human responds to the user according to the audio.

[0004] However, in the above methods, the response audio and lip-shape feature data are usually large in size, and the transmission and parsing take a long time, resulting in high end-to-end interaction latency. In addition, network fluctuations may cause the response audio and lip-shape feature data to arrive at different times, causing audio-visual desynchronization problems in digital humans. Summary of the Invention

[0005] This application provides a digital human interaction system, method, device, storage medium, and program product to improve the real-time interaction of digital humans, reduce end-to-end response latency, effectively solve the problem of audio-visual asynchrony, enhance user interaction experience, and improve the real-time interaction effect of digital humans.

[0006] This application provides a digital human interaction system, comprising: a scheduling layer configured to, in response to a connection request sent by a client, create a digital human instance for the client and load the narration audio and digital human video corresponding to the digital human instance, wherein the digital human instance corresponds to a unique session identifier; a coordination layer configured to, based on the session identifier, route interaction requests sent by the client during the playback of the narration audio and digital human video by the digital human instance to the digital human instance; a processor layer configured to, in response to the interaction request, execute an interaction task to generate response audio and lip-sync feature data corresponding to the interaction request; a data interaction layer configured to, encapsulate the response audio and lip-sync feature data into a binary data packet; wherein the lip-sync feature data is located before the response audio in the binary data packet and is aligned according to a preset storage granularity; and a channel layer configured to, establish a session channel between the digital human instance and the client, and send the narration audio, digital human video, and binary data packet to the client through the session channel, so that the client drives the digital human instance to play the narration audio and digital human video, and in the process, drives the digital human instance to interact with the user in real time during the narration based on the response audio and lip-sync feature data.

[0007] This application provides a digital human interaction method, the method comprising: receiving an interaction request sent by a client, the interaction request being sent during the playback of explanatory audio and digital human video by a digital human instance, for requesting real-time interaction with the digital human instance; executing an interaction task in response to the interaction request to generate response audio and lip-sync feature data corresponding to the interaction request; encapsulating the response audio and the lip-sync feature data into a binary data packet, wherein the lip-sync feature data is located before the response audio in the binary data packet and aligned according to a preset storage granularity; and sending the binary data packet to the client through a pre-established session channel, so that the client drives the digital human instance to interact with the user in real-time during the playback of the explanatory audio and digital human video by means of the response audio and lip-sync feature data.

[0008] This application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program in the memory to implement the steps in the digital human interaction method.

[0009] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the digital human interaction method.

[0010] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, enable the processor to implement the steps in the digital human interaction method.

[0011] In this embodiment, the digital human interaction system is based on a layered architecture design, including multiple layers such as a scheduling layer, coordination layer, processor layer, data interaction layer, and channel layer. These layers collaborate to form a processing pipeline for interaction requests, improving the overall processing efficiency and response speed of the system. Based on this layered architecture, the digital human interaction system can encapsulate the response audio and lip-sync feature data corresponding to the interaction request into the same binary data for transmission to the client, avoiding the arrival time difference and long parsing time issues associated with sending different packets. Furthermore, in this binary data packet, the lip-sync feature data precedes the response audio and is aligned according to a preset storage granularity, facilitating the client's priority parsing of the lip-sync feature data for rendering lip-sync animation, thus solving the problem of audio-visual asynchrony. In summary, the digital human interaction system based on this embodiment improves the real-time interaction of the digital human, reduces end-to-end response latency, enhances the user interaction experience, and improves the real-time interaction effect of the digital human. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram of the structure of a digital human interaction system provided as an exemplary embodiment of this application; Figure 2 A schematic diagram of the structure of a binary data packet provided as an exemplary embodiment of this application; Figure 3 A schematic diagram of the structure of another digital human interaction system provided as an exemplary embodiment of this application; Figure 4 A flowchart illustrating a digital human interaction method provided in an embodiment of this application; Figure 5 A schematic diagram of a digital human interaction method provided for an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0015] Additionally, it should be noted that when user interaction operations or triggering operations are involved in the embodiments of this application, these operations include, but are not limited to, various interaction methods such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations. Touch operations include, but are not limited to, click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight-line swipes and curved-line swipes.

[0016] In digital human application scenarios, users can initiate one-on-one (1V1) real-time interaction with a digital human through a client. The server can send pre-generated explanatory audio and digital human video to the client, which then renders the digital human explaining the product based on the audio and outputs it to the user. During the product explanation, the user can interact with the digital human. The server can generate response audio and lip-sync feature data required for the digital human to output the response audio based on the user's voice or text, and send the response audio and lip-sync feature data to the client respectively. After receiving the response audio and lip-sync feature data, the client can render the digital human video based on the lip-sync feature data and simultaneously play the response audio to achieve a natural and realistic effect of the digital human responding to the user according to the response audio. However, in the above method, the response audio and lip-sync feature data are usually large in size, and the transmission and parsing are time-consuming, resulting in high end-to-end interaction latency. In addition, network fluctuations may cause the response audio and lip-sync feature data to arrive at different times, causing audio-visual desynchronization problems in the digital human.

[0017] In addition, the scheduling layer in the digital human interaction system in related technologies usually adopts a single digital human instance or simple load balancing, lacking a mechanism for pre-building digital human instances and pre-connecting channels. The response time of the first frame often exceeds 1 second, which is difficult to meet the needs of real-time dialogue scenarios.

[0018] To address the aforementioned technical issues, this application provides a digital human interaction system. This system utilizes two key technologies—a layered architecture design and an optimized binary data packet structure—to synergistically improve the real-time interaction performance of the digital human, reduce end-to-end response latency, ensure audio-visual synchronization, and ultimately enhance the user experience and improve the real-time interaction effect of the digital human. Specifically, the layered architecture includes a scheduling layer, a coordination layer, a processor layer, a data interaction layer, and a channel layer. The scheduling layer can respond to connection requests sent by clients, create digital human instances, and load their corresponding narration audio and video. The coordination layer can route interaction requests sent by clients to digital human instances based on session identifiers. The processor layer can respond to interaction requests and execute interaction tasks while the digital human instance is playing narration content, outputting response audio and lip-sync feature data. The data interaction layer can encapsulate the response audio and lip-sync feature data into binary data packets and send binary data packets to the client through the session channel established by the channel layer between the digital human instance and the client. This allows the client to render the lip-sync animation of the digital human instance based on the lip-sync feature data and simultaneously play the response audio, enabling the digital human to support real-time interaction during narration.

[0019] In this embodiment, on the one hand, a layered design allows each layer to focus on a single responsibility, achieving decoupling of responsibilities. Specifically, the scheduling layer creates digital human instances on demand and preloads resources to avoid runtime blocking; the coordination layer accurately routes requests based on session identifiers to ensure low interference and low latency under multi-user concurrency; the processor layer can respond to interactive requests in real time during the explanation and playback process, supporting "playing and answering simultaneously" without interrupting the main process; the data interaction layer and the channel layer separate data encapsulation and transmission logic, facilitating independent optimization to improve processing efficiency and concurrency capabilities. The cooperation between the layers forms a real-time processing pipeline, providing conditions from the architectural level for improving the overall processing throughput and response speed of the system.

[0020] On the other hand, based on the layered architecture, binary data packets are further integrated to ensure low-latency transmission and precise synchronization of audio and video data. Specifically, since the digital human interaction system can encapsulate the response audio and lip-sync data into the same binary data packet and send this binary data packet to the client, it avoids the inconsistency in the timing of the client receiving the response audio and lip-sync data caused by cross-packet transmission. Moreover, in this binary data packet, the lip-sync data is placed before the response audio, allowing the client to parse the lip-sync data first and render the lip-sync animation, thus achieving audio-visual synchronization. Furthermore, since the response audio and lip-sync data are in binary format and aligned according to a preset storage granularity, the client does not need to perform text parsing and deserialization or additional copying when parsing the data, reducing the time spent parsing the data and thus reducing the latency of the digital human responding to the user.

[0021] In summary, the digital human interaction system provided in this application achieves an efficient and scalable real-time processing pipeline through a layered architecture, and ensures low-latency transmission and accurate synchronization of audio and video data through a carefully designed binary data packet structure. These two aspects complement each other, not only solving the core pain points of high latency and audio-visual asynchrony in traditional digital human systems, but also constructing a high-performance, highly reliable, and highly immersive real-time interaction platform, significantly improving the natural interaction experience between users and digital humans.

[0022] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of the structure of a digital human interaction system provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 1 The digital human interaction system 100 may include: a scheduling layer 101, a coordination layer 102, a processor layer 103, a data interaction layer 104, and a channel layer 105.

[0024] The digital human interaction system 100 can be applied in a variety of scenarios that require human-like interaction. For example, it can be used in e-commerce for digital human shopping guides, virtual live streamers, financial services for wealth management, online teaching assistants and homework help in education, health consultants, and psychological counseling in healthcare. This embodiment does not impose any limitations on these applications.

[0025] The Digital Human Interaction System 100 can be deployed on a cloud server or a traditional server, without any limitation.

[0026] In this embodiment of the application, the scheduling layer 101 can be used to create a digital human instance for the client in response to a connection request sent by the client, and load the explanatory audio and digital human video corresponding to the digital human instance.

[0027] The client can refer to a user's terminal device or the digital human application software on the user's terminal device. In this embodiment, the user's terminal device is used as an example for illustration.

[0028] Optionally, the connection request can be sent by the client when it detects an activation operation on the digital human instance. For example, the activation operation can be, but is not limited to, any of the following: the user clicks on the digital human live stream page, the user long-presses on a digital human video area, or the user swipes into the digital human interactive zone on the application interface.

[0029] After a user activates the digital human instance on the client, the client can detect the user's activation of the digital human instance and send a connection request to the cloud server, that is, send a connection request to the digital human interaction system.

[0030] It is important to emphasize that the connection request is sent before the user interacts with the digital human instance. The digital human interaction system 100 can respond to the connection request through the scheduling layer 101 to create a digital human instance for the client, and establish a session channel between the digital human instance and the client through the channel layer 105.

[0031] Optionally, the digital human instance can be associated with explanatory audio and digital human video. The scheduling layer 101 can load the explanatory audio and digital human video corresponding to the digital human instance from the storage space (located in a local server or a remote server) into the digital human interaction system, so that the channel layer 105 can push the explanatory audio and digital human video to the client after establishing a session channel between the digital human instance and the client.

[0032] It should be noted that for any client, the scheduling layer 101 can respond to the connection request sent by that client and create a digital human instance for that client. Therefore, the scheduling layer 101 supports the creation of multiple digital human instances, which can run in parallel in independent processes without interfering with each other.

[0033] In this embodiment of the application, in conjunction with process isolation and multi-instance strategy (i.e., supporting multiple digital human instances in parallel), the concurrency capability of a single node is increased by more than 1 times, effectively improving resource utilization.

[0034] After the scheduling layer 101 creates a digital human instance, it can also assign a unique session identifier to the digital human instance.

[0035] In response to a connection request from the client, the channel layer 105 creates a digital human instance in the scheduling layer 101, assigns a session identifier to the digital human instance through the coordination layer 102, establishes a session channel between the digital human instance and the client, and associates the session identifier with this session channel. Then, the channel layer 105 can send the explanatory audio and video corresponding to the digital human instance to the client through the session channel. In other words, before the user interacts with the digital human instance, the channel layer 105 has already sent the explanatory audio and video corresponding to the digital human instance to the client through the session channel.

[0036] Alternatively, the audio and video of the digital human can be sent to the client in a binary data packet format or in a non-binary text format, such as JSON (JavaScript Object Notation) or XML (Extensible Markup Language).

[0037] After receiving the audio and video explanations, the client can drive the digital human instance to play the audio and video explanations.

[0038] Optionally, the audio explanation can be generated based on the explanatory text of the target object, and the digital human video can be a video showcasing and explaining the target object. The digital human video and the audio explanation can be played synchronously to achieve the demonstration and explanation of the target object.

[0039] In this embodiment, the target object to be explained can be, but is not limited to, any of the following: commodity objects, cultural and tourism resource objects, digital artworks, etc. The explanatory text for the target object can describe the basic attribute information, functions, usage methods, etc. of the target object in a conversational, logically clear, and engaging manner, and belongs to natural language description text.

[0040] In practical applications, explanatory texts for the target object can be written manually, or they can be automatically generated using generative AI (Artificial Intelligence) models; there are no restrictions on this. Generative AI models are algorithmic models that automatically generate text, images, audio, video, code, and other content using artificial intelligence technology. This application does not limit the parameter scale of the generative AI model used. For example, the generative AI model can be a Large Language Model (LLM) or a Multimodal Large Language Model (MLLM).

[0041] In some optional embodiments of this application, after obtaining the explanatory text for the target object, a content security review can be performed on the explanatory text. For example, the content security review may include, but is not limited to, sensitive word filtering and compliance verification to ensure that the content of the explanatory text is safe, legal, and compliant.

[0042] Taking a product livestreaming scenario as an example, when using an LLM model to generate explanatory text for product objects, relevant personnel can input the anchor's image, product image, and explanatory text requirements into the front-end interface.

[0043] For example, the requirements for the explanatory text may include, but are not limited to: target audience characteristics, desired language style, key product selling points to be emphasized, text length limits, content structure requirements, and compliance constraints. This explanatory text requirement information can guide the LLM model to generate personalized explanatory content that meets the needs of the live streaming scenario. The LLM model can perform content generation tasks based on the host's image, product images, and the explanatory text requirement information to obtain personalized explanatory text for the product.

[0044] In practical applications, the text of the explanation of the target object can be processed by TTS (Text-to-Speech) to obtain the audio of the explanation of the target object.

[0045] In practical applications, S2V (Speech-to-Video) processing can be performed on the audio of the target object's explanation to obtain a digital human video of the target object. The digital human video mainly consists of video footage of a digital human explaining the target object. Specifically, the digital human video obtained through audio-to-video processing can dynamically adjust the digital human's posture, pointing gestures, or explanation focus based on the target object's attribute information (e.g., type), user attention points, or interactive commands. The digital human's explanation footage is highly synchronized with the target object, and its body movements and facial expressions are closely aligned with the current explanation context. The digital human's movements are more diverse and natural, achieving a visual effect that simulates the micro-movements and random behaviors of real humans.

[0046] In this embodiment of the application, the digital human video includes not only video footage in which a digital human image explains the target object, but also video footage related to the target object but without a digital human. This video footage can highlight key information such as the detailed features (e.g., material, structure) of the target object.

[0047] The coordination layer 102 can be used to manage the session channels corresponding to the digital human instances, assign session identifiers to each digital human instance, and route the interaction requests sent by the client to the digital human instance based on the session identifiers.

[0048] Since the scheduling layer 101 can create multiple digital human instances, and each digital human instance corresponds one-to-one with a client, there are multiple session channels corresponding to multiple digital human instances. The coordination layer 102 can manage each session channel, including but not limited to: binding or unbinding a session channel with its corresponding digital human instance, and monitoring the channel status of each session channel. Optionally, the channel status can include connected status, disconnected status, idle status, busy status, etc.

[0049] For any given session channel, there is a corresponding session identifier. For example, the session identifier for session channel 1 could be S001.

[0050] Since the session channel between the client and the digital human instance is unique, and the session identifier corresponding to the session channel is unique, there is a one-to-one correspondence between the digital human instance and the session identifier, meaning that each digital human instance corresponds to a unique session identifier. The coordination layer 102 can assign a corresponding session identifier to each digital human instance.

[0051] During the playback of narration audio and video by the digital human instance, any client can initiate an interaction request to the digital human interaction system 100 through the corresponding session channel. Therefore, the coordination layer 102 can receive multiple interaction requests. For any given interaction request, the coordination layer 102 can route the interaction request to the corresponding digital human instance based on the session identifier.

[0052] Optionally, the session identifier can be carried in the interaction request or it can be marked by the coordination layer 102 based on the session channel that receives the interaction request.

[0053] In this embodiment, the interaction request can be generated by the client in response to user input of voice or text. The interaction request can carry input data, which can be user-inputted voice or text data. The interaction request can be used to request real-time interaction with the digital human instance. For example, if the digital human instance is explaining target object A, and the user wants to see the material of target object A, the user can input the voice message "What is the material?" on the digital human instance interface. The client can then respond to the user's voice input and generate an interaction request, which can include the voice data "What is the material?".

[0054] Since the digital human instance and session channel are pre-established before the user interacts with the digital human, when the user interacts with the digital human (i.e., when the client responds to the user's input of voice or text to generate an interaction request and sends an interaction request to the digital human interaction system 100), the digital human instance has already been created and the session channel is ready. The interaction request can directly trigger the processing pipeline of the digital human interaction system 100 without waiting for resource initialization. The response time of the first frame can be compressed to within 1 second, which effectively meets the needs of real-time interaction scenarios and solves the problem of slow cold start.

[0055] The processor layer 103 can be used to perform interactive tasks in response to an interactive request to generate response audio and lip feature data corresponding to the interactive request.

[0056] Optionally, processor layer 103 can call the LLM model to generate response text, and then use a TTS algorithm to synthesize speech from the response text to generate response audio. Optionally, the TTS algorithm can generate response audio with a style consistent with the narration audio, based on the role characteristics of the digital human instance (such as speech rate, voice style, etc.). The generated response audio is a personalized answer to the user's input voice or text.

[0057] Processor layer 103 can call the lip-driven feature engine to generate lip shape feature data corresponding to the response text. The lip shape feature data can accurately describe the opening and closing state of the digital human's lips at every moment, and determine the synchronization between lip movements and response audio.

[0058] In the embodiments of this application, generating response audio and generating lip shape feature data can be performed in parallel to reduce the latency of the interaction.

[0059] For example, if the interaction request includes the voice data "What is the material?", then the processor layer 103 can call the LLM model to generate the response text "The material is cotton and linen", call the TTS algorithm to generate the response audio corresponding to the response text, and call the lip feature engine to generate the lip shape feature data corresponding to the response text.

[0060] The data interaction layer 104 can be used to encapsulate response audio and lip-sync feature data into binary data packets.

[0061] Below, in conjunction with Figure 2 The structure of binary data packets is explained.

[0062] Figure 2 This is a schematic diagram illustrating the structure of a binary data packet provided for an exemplary embodiment of this application. Please refer to... Figure 2 In this embodiment of the application, the binary data packet may include a fixed header and a dynamic payload portion. Both the fixed header and the dynamic payload portion are in binary format.

[0063] The fixed head includes at least one of the following: lip feature data length, response audio length, timestamp, digital human instance information, and data encoding metadata.

[0064] Optionally, the fixed header may also include, but is not limited to, at least one of the following: transport protocol version number, extended flags, reserved fields, sequence number, etc.

[0065] like Figure 2 As shown, the fixed header can be 128 bits long (i.e., bits 0-127). Here, V represents the transport protocol version number, occupying bits 0-2; X represents the extension flag, occupying bit 3; LDL (Lip Data Length) indicates the number of bytes occupied by the lip-sync data, occupying bits 4-7; RAL (Reply Audio Length) indicates the number of bytes occupied by the reply audio, occupying bits 8-11; F represents a reserved field, occupying bits 12-15; the sequence number occupies bits 16-31; the timestamp occupies bits 32-63; the digital human instance information occupies bits 64-95; and the data encoding metadata occupies bits 96-127. Starting from bit 128 is the dynamic payload section, which begins with the lip-sync data and then the reply audio.

[0066] The reserved X extension bits and F reserved fields in the binary data packet can support future functional expansion without breaking the existing parsing logic. For example, adding QoS (Quality of Service) levels or semantic tags.

[0067] The extension flag can be used to indicate whether extension space is used. For example, if X is 0, it means that extension space is not used; if X is 1, it means that extension space is used.

[0068] Sequence numbers can provide packet loss retransmission and out-of-order reassembly capabilities, ensuring the integrity of key frames.

[0069] A timestamp is the moment a data packet is sent and can be used to calibrate end-to-end latency.

[0070] Digital human instance information can be used to describe the attributes and status of a digital human instance. For example, digital human instance information may include, but is not limited to: digital human instance identifier, role type, language style identifier, etc.

[0071] Data encoding metadata may include, but is not limited to: audio encoding metadata corresponding to the response audio, lip-shape encoding metadata corresponding to the lip-shape feature data, synchronization and packaging data, compression and quantization data, etc.

[0072] The audio encoding metadata may include audio encoding format, sampling rate, number of channels, etc.; the lip-coding metadata may include lip-shape parameter type, feature dimension, frame rate, etc.; the synchronization and packing data may include whether the audio and lip shape are aligned in the same frame, the number of samples contained in each frame, etc.; the compression and quantization data may include whether differential coding is enabled, whether Huffman coding is used, etc.

[0073] The dynamic payload may include lip-sync feature data and response audio, and the payload is aligned according to a preset storage granularity. For example, the preset storage granularity may be any of the following, but not limited to: 32 bits, 64 bits, 128 bits, etc.

[0074] Optionally, the preset storage granularity can be determined based on the client's CPU (Central Processing Unit) type and / or model. For example, when the client's CPU is a 32-bit architecture, the preset storage granularity can be 32 bits to match its natural word length and memory access characteristics.

[0075] In this embodiment, the preset storage granularity can be 32 bits. The fixed header of the binary data packet has a total length of 128 bits, which is an integer multiple of 32 bits; the dynamic payload is aligned to 32 bits, so that the entire data packet is stored continuously in memory without padding bytes, which can adapt to the memory access optimization of mainstream processor architectures. The client can directly access each field through pointer offset, significantly reducing parsing overhead and improving mobile device compatibility.

[0076] In the dynamic payload section, lip-sync data can precede the response audio. This allows the client to prioritize decoding the lip-sync data to drive the rendering pipeline, improving visual responsiveness. The response audio follows immediately after the lip-sync data, ensuring audio playback continuity.

[0077] Binary data packets can adopt a compact encoding format, that is, while ensuring information integrity, (1) necessary fields are retained and redundant data is removed (e.g., no redundant delimiters); (2) each field is represented by the minimum necessary number of bits, and multiple fields are arranged tightly by bits. Based on the collaborative design of simplifying the field structure and efficient encoding (i.e., minimum bit representation) in binary data packets, compact encoding of binary data packets is realized, which improves transmission efficiency and parsing speed. In the embodiments of this application, the design of binary data packets avoids the overhead of text formats such as JSON / XML, reduces network bandwidth occupation and decoding latency, and is particularly suitable for mobile terminals and real-time inference and rendering scenarios. Moreover, the design of this data packet breaks through the traditional audio and video separation transmission mode, significantly reduces the number of network round trips, improves the smoothness of interaction, and has technical practicality.

[0078] Optionally, the binary data packet can be adapted to various transport protocols, such as WebSocket or HTTP (Hypertext Transfer Protocol).

[0079] After the data interaction layer 104 encapsulates the fixed head and dynamic payload (including response audio and lip-sync feature data) to obtain a binary data packet, the channel layer 105 can send the binary data packet to the client through the session channel corresponding to the digital human instance based on the transmission protocol. This enables the client to drive the digital human instance to interact with the user in real time during the explanation process by driving the digital human instance to play the explanation audio and digital human video, based on the response audio and lip-sync feature data.

[0080] After receiving the binary data packet, the client can extract lip shape feature data and response audio from the binary data packet, and render the lip shape animation of the digital human instance based on the lip shape feature data, and play the response audio synchronously to enable interaction during the explanation.

[0081] In an optional embodiment, the client can adopt a full-frame rendering response mode to render the lip animation of the digital human instance and realize interaction. That is, after receiving binary data, the client can read the lip feature data length and response audio length from the binary data packet, and read the lip feature data from the beginning position of the dynamic payload according to the number of bytes indicated by the lip feature data length. The lightweight 3D rendering engine is started to render the lip animation of the digital human instance in real time according to the lip feature data, and the response video stream is directly generated. The client can read the response audio from the end position of the lip feature data according to the number of bytes indicated by the response audio length, and call the corresponding audio decoder according to the audio encoding format in the data encoding metadata to restore the response audio to PCM (Pulse Code Modulation) data and store the PCM data in the audio playback buffer queue.

[0082] The client can determine the target playback start time based on the timestamp and preset time offset in the binary data packet, and based on the target playback start time, pause the playback of the original digital human video and narration audio, and synchronously play the response video stream and response audio to achieve a natural connection with the digital human's narration content, thereby enabling real-time interaction with the user during the narration process.

[0083] The target playback start time is calculated as: timestamp + preset time offset. The preset time offset can be a fixed value, a negotiated value, or dynamically determined by the digital human interaction system based on the interaction context.

[0084] In another optional embodiment, the client can employ a lip overlay enhancement mode to render the lip animation of the digital human instance, enabling interaction. Specifically, after receiving a binary data packet sent by the channel layer, the client can pause the playback of the current digital human video and narration audio. The client can parse the binary data packet, extracting lip shape feature data and response audio. The lip shape feature data can be a time-aligned facial deformation weight sequence (such as a mixed deformation coefficient sequence), with each frame corresponding to a playback moment. Subsequently, the client can launch an audio-visual synchronization player, mapping the lip shape feature data to dynamic lip deformation parameters while playing the response audio. A transparent lip mask layer is then generated based on these parameters and overlaid in real-time onto the facial area of ​​the digital human video. Pixel-level fusion is performed through the GPU (Graphics Processing Unit) rendering pipeline, dynamically replacing the lip area in the original digital human video with lip animation synchronized with the speech content. The overlay process can be frame-aligned based on the timestamp embedded in the binary data packet, ensuring strict synchronization between the start time of the lip animation and the audio playback, achieving audio-visual synchronization. In this way, real-time lip-sync enhancement of pre-recorded digital human videos can be achieved without regenerating the complete video stream, significantly reducing server-side computing load and network bandwidth consumption, while maintaining a highly natural interactive experience and enabling real-time interaction with users during the explanation process.

[0085] In this embodiment, the digital human interaction system includes a scheduling layer, a coordination layer, a processor layer, a data interaction layer, and a channel layer. Since the digital human interaction system can respond to a connection request sent by a client when it detects an activation operation on a digital human instance, it creates a digital human instance for the client through the scheduling layer and establishes a session channel between the digital human instance and the client through the channel layer. This provides a pre-creation mechanism for digital human instances and a pre-connection mechanism for channels, reducing the first-frame response time and effectively meeting the needs of real-time interaction scenarios. Because the data interaction layer can encapsulate the response audio and lip-sync feature data into the same binary data packet and send this binary data packet to the client, it avoids inconsistent timing of the client receiving the response audio and lip-sync feature data due to heterogeneous packet transmission, significantly reducing network round trips and improving interaction smoothness. Furthermore, in this binary data packet, the lip-sync feature data is placed before the response audio, allowing the client to prioritize parsing the lip-sync feature data and rendering the lip-sync animation. Since the response audio and lip-sync feature data are in binary format and aligned according to a preset storage granularity, the client avoids the overhead of text parsing and deserialization during data parsing, and does not require additional copying, reducing parsing time and thus reducing the latency of the digital human responding to the user. In summary, the digital human interaction system provided in this application improves the real-time interaction of digital humans, reduces end-to-end response latency, effectively solves the problem of audio-visual asynchrony, enhances user interaction experience, and improves the real-time interaction effect of digital humans.

[0086] Below, in the above Figure 1 Based on the implementation examples, combined with Figure 3 The above-mentioned digital human interaction system will be further explained.

[0087] Figure 3 A schematic diagram of another digital human interaction system provided as an exemplary embodiment of this application. Please refer to... Figure 3 The scheduling layer 101 may include an instance scheduler and an instance manager. The instance scheduler can be used to determine the resource configuration of the digital human instance in response to a connection request sent by the client, and send an instance creation instruction to the instance manager.

[0088] Optionally, the connection request may include a user identifier and an instance type identifier for the digital human instance. The instance scheduler can determine the resource configuration of the digital human instance based on the user identifier, instance type identifier, and system load status. Optionally, the resource configuration may include, but is not limited to: computing resource quotas, digital human role models, speech synthesis parameters, lip-syncing models, interactive knowledge bases, and security isolation policies. Among these, computing resource quotas can be used to limit the CPU, memory, or GPU usage of the digital human instance, and security isolation policies can be used to bind user identities and restrict the scope of data access.

[0089] The instance manager can create digital human instances in a separate process based on instance creation instructions, and load explanatory audio and digital human videos.

[0090] The instance manager can also manage the entire lifecycle of at least one digital human instance. For example, the instance manager can monitor the running status of at least one digital human instance and destroy digital human instances. The instance manager can publish the online status, network address, and service capabilities of digital human instances to the service registry center through service registration and discovery protocols (such as IP polling mechanisms). Clients can establish communication connections with target digital human instances based on registration information and be notified of status changes in a timely manner when the instance is offline.

[0091] At least one digital human instance runs in an independent process. Creating digital human instances in independent processes ensures that each instance is independent in terms of memory space, computing resources, and security context. This not only achieves fault and resource isolation but also supports data security, coexistence of heterogeneous models, and fine-grained lifecycle management, thereby ensuring service quality and system stability in high-concurrency interaction scenarios.

[0092] Optionally, the scheduling layer 101 may also include a process manager and an IPC (Inter-Process Communication) manager.

[0093] The process manager can be used to manage the entire lifecycle of operating system-level processes running on a digital human instance. This includes: upon receiving an instance creation command, starting an independent process to host the runtime environment of the digital human instance; continuously monitoring its resource usage (such as CPU utilization, memory consumption, and GPU memory usage) and health status (such as whether it responds to heartbeats and whether there are abnormal exits) during instance operation; and safely terminating and reclaiming all system resources (including memory, file handles, network connections, etc.) occupied by the process when preset conditions are met (such as instance idle timeout, user-initiated disconnection, system resource shortage, or process crash), thereby preventing resource leaks and ensuring the long-term stability and high availability of the system.

[0094] The IPC Manager can establish secure, low-latency, and ordered data exchange channels between multiple independent processes within the scheduling layer (such as the digital human service process and the audio processing process). Based on the inter-process communication mechanisms provided by the operating system (such as shared memory or message queues), it achieves efficient transmission of binary data packets and ensures data consistency and security during concurrent read and write operations of different processes through sequence number verification, access control, and memory boundary protection. In particular, when transmitting synchronization packets containing lip-sync data and response audio, the IPC Manager can complete cross-process transmission with microsecond-level latency, thereby supporting the client to achieve precise synchronized playback of lip-sync animation.

[0095] The coordination layer 102 may include a task manager, a message manager, and a channel manager.

[0096] The task manager can monitor the status of interactive tasks. Optionally, when a user triggers an interactive request (such as a voice question, text input, or click operation), the digital human interaction system can generate a corresponding interactive task and assign a unique task identifier. The task manager can continuously track the status changes of the task in the processing flow, including but not limited to "pending processing," "generating a response," "audio and lip-sync data ready," "transmitting," "playback complete," or "abnormal termination." At the same time, the task manager can also record the task's creation time, processing time, associated digital human instance, and resource consumption information, and perform retry, degradation, or alarm operations on timed-out or failed tasks based on preset policies, thereby ensuring the reliability and traceability of the interaction process.

[0097] The message manager can monitor events generated by at least one functional processor in the processor layer and trigger the next functional processor to execute the corresponding task based on the event.

[0098] At least one functional processor includes, but is not limited to, an audio processor, a text processor, a semantic understanding processor, a speech synthesis processor, and a lip-syncing processor. When any functional processor completes its subtask, it publishes a structured event to the message bus (such as "TTS completed, audio ID=audio_123" or "lip-syncing feature generated, sequence number=456"). After the message manager captures the event, it automatically triggers the next dependent functional processor to execute subsequent processing steps according to predefined event-task mapping rules or state machine logic (such as the TTS completion event triggering the lip-syncing processor to start), thereby achieving loose coupling, asynchronous operation, and automated collaboration of multimodal tasks, and improving system throughput and response efficiency.

[0099] The channel manager can maintain multiple channel types and determine the target channel type among multiple channel types based on client type, network quality, and task type. The target channel type is then provided to the channel layer 105 so that the channel layer 105 can create a session channel based on the target channel type.

[0100] Multiple channel types can include RTQ (Real-Time Queue) channels and RTC (Real-Time Communication) channels. RTQ channels can achieve ordered, persistent transmission based on reliable message queues, suitable for low-latency real-time audio streaming, and support high-concurrency transmission of response audio and lip-sync data. RTC channels can achieve millisecond-level end-to-end delivery based on low-latency transmission protocols.

[0101] The channel manager can dynamically determine the target channel type from multiple channel types based on client type, network quality, and task type. In other words, the channel manager can dynamically select a data transmission channel that meets quality requirements (e.g., a data transmission channel with low packet loss rate or low network latency) based on the current session context to adapt to different client capabilities, network environments, and task priorities.

[0102] For example, the channel manager can collect client type (such as mobile or web), current network quality indicators (such as packet loss rate and round-trip latency) and the type of task to be transmitted (such as high-priority real-time Q&A or low-priority explanation playback) in real time, and intelligently determine the target channel type from multiple channel types based on the above multi-dimensional parameters.

[0103] The channel manager can provide the target channel type to the channel layer. Channel layer 105 can then create a session channel between the digital human instance and the client based on the target channel type.

[0104] Optionally, the channel manager can also dynamically switch channels during a session (such as switching from an RTC channel to an RTQ channel when the network deteriorates), thereby maximizing transmission efficiency and system robustness while ensuring user experience.

[0105] Optionally, the coordination layer 102 may also include a processing unit manager. The processing unit manager can uniformly manage at least one processing unit in the processor layer 103, that is, manage at least one functional processor. Optionally, the processing unit manager can perform lifecycle management on each functional processor, including on-demand creation and initialization, runtime status monitoring, resource allocation and reclamation, anomaly detection and restart, and maintain the dependencies and execution contexts between processing units, thereby ensuring that multimodal interaction tasks flow and execute efficiently, orderly, and reliably within the coordination layer.

[0106] The processor layer 103 may include a task processing unit, a semantic response unit, and a multimodal data generation unit. The semantic response unit may include an audio processor, a text processor, and a semantic understanding processor. The multimodal data generation unit may include a speech synthesis processor and a lip-syncing processor.

[0107] The interactive tasks generated by the processor layer 103 in response to the interactive request sent by the client may include semantic response generation tasks and multimodal data generation tasks.

[0108] Optionally, in processor layer 103, a task processing unit can generate a semantic response generation task and a multimodal data generation task based on the interaction request, and distribute the semantic response generation task and the multimodal data generation task to the corresponding processing unit, that is, distribute the semantic response generation task to the semantic response unit and the multimodal data generation task to the multimodal data generation unit. The semantic response generation task and the multimodal data generation task can have a higher priority than the explanation task.

[0109] After receiving a semantic response generation task, the semantic response unit can generate response text based on the speech data to execute the semantic response generation task. Since the semantic response unit may include an audio processor, a text processor, and a semantic understanding processor, and the semantic response generation task may include a speech recognition subtask and a response text generation subtask, if the interaction request includes speech data, the audio processor can perform speech activity detection and speech recognition processing on the speech data to generate target text data, which is then written to a text queue to execute the speech recognition subtask. The text processor can retrieve the target text data from the text queue and trigger the semantic understanding processor to retrieve the target text data. The semantic understanding processor can perform semantic understanding on the target text data to generate response text to execute the response text generation subtask.

[0110] In this embodiment, the digital human interaction system may further include an algorithm layer 106. The algorithm layer 106 may include an ASR (Automatic Speech Recognition) algorithm, a VAD (Voice Activity Detection) algorithm, an LLM (Large Language Model) algorithm, a TTS algorithm, and a lip-sync engine. The TTS algorithm can support multiple timbres, multiple languages, and prosody control, and output compressed audio with timing alignment information to meet the transmission requirements of both low bandwidth and high fidelity.

[0111] In processor layer 103, the audio processor can call the VAD algorithm to detect speech activity in the speech data and the ASR algorithm to perform speech recognition processing to generate target text data. Specifically, the audio processor can call the VAD algorithm to analyze the energy characteristics, spectral patterns, and temporal changes in the speech data in real time to accurately identify speech segments containing valid human voices and filter out silence segments and background noise. After detecting speech activity, the audio processor can call the ASR algorithm to convert the speech segments into corresponding text sequences. Through the joint decoding of acoustic and language models, combined with contextual semantic information, the ASR algorithm can output highly accurate recognition results. The audio processor can then encapsulate the recognition results into target text data and write the target text data into a text queue to complete the semantic response generation task. By calling the collaborative work of the VAD and ASR algorithms, a "trigger-on-demand, efficient recognition" speech interaction mechanism is realized, effectively reducing computational resource consumption and end-to-end response latency.

[0112] The text queue, acting as an asynchronous buffer channel in the coordination layer, can temporarily store text data to be processed, allowing the text processor to read it sequentially. This text queue supports FIFO (First In, First Out) scheduling and can be configured with maximum capacity and blocking / discarding policies to prevent memory overflow due to backend processing delays. By writing target text data into the text queue, the audio processor and subsequent function processors can be decoupled, enabling speech acquisition and recognition to run independently and at high speed without waiting for semantic generation to complete, thereby improving the overall system throughput and real-time response performance.

[0113] The text processor can continuously monitor and retrieve target text data sequentially from the text queue. After retrieving the target text data, the text processor performs format validation, session context association (such as binding user ID, session identifier, and timestamp), publishes a "text ready" event, and writes the target text data into the semantic understanding queue.

[0114] Optionally, when the text processor writes the target text data to the semantic understanding queue, it can publish a lightweight event notification (such as a condition variable or message bus event) to wake up the semantic understanding processor, which may be in a dormant state.

[0115] The semantic understanding processor can continuously monitor the semantic understanding queue and automatically pull and process a new task once it is detected.

[0116] After retrieving target text data from the semantic understanding queue, the semantic understanding processor can invoke the LLM model to perform semantic understanding operations on the target text data, including intent recognition and context disambiguation. It then combines the current dialogue state with a pre-built knowledge base to generate response text that conforms to the role setting and application logic. This response text can serve as the core content of the digital human's interactive response, triggering subsequent multimodal data generation tasks.

[0117] After receiving a multimodal data generation task, the multimodal data generation unit can generate response audio and lip-shape feature data based on the response text to execute the multimodal data generation task. Since the multimodal data generation unit can include a speech synthesis processor and a lip-shape driver processor, and the multimodal data generation task can include an audio generation subtask and a lip-shape driver generation subtask, the speech synthesis processor can call a text-to-speech algorithm to synthesize speech based on the response text and generate response audio to execute the audio generation subtask; similarly, the lip-shape driver processor can call a lip-shape feature engine based on the response text to generate lip-shape feature data to execute the lip-shape driver generation subtask.

[0118] The semantic understanding processor can write the response text to the TTS task queue and the lip-sync task queue, and publish a "TTS ready event".

[0119] The speech synthesis processor can monitor the TTS task queue. When a new reply text is detected, it can call the TTS algorithm to convert the reply text into the corresponding reply audio.

[0120] The lip-syncing processor can synchronously monitor the lip-syncing task queue. After receiving the same response text (or being triggered by the same "TTS Ready" event), it can call the lip-syncing feature engine to generate lip-syncing feature data (such as a 50-dimensional facial deformation weight sequence or a 3D keypoint trajectory) that is strictly aligned with the response audio based on the phoneme sequence, pronunciation duration, and stress position of the response text.

[0121] It should be noted that if the input data included in the interaction request is text data rather than speech data, then the semantic response generation task does not include a speech recognition subtask, but includes a response text generation subtask. In other words, the semantic response generation task is the same as the response text generation subtask. The processor layer 103 can write the text data into a text queue, retrieve the text data from the text queue through the text processor, and trigger the semantic understanding processor to retrieve the text data. The semantic understanding processor can perform semantic understanding on the text data to generate response text, thereby executing the response text generation subtask. Then, the speech synthesis processor converts the response text into the corresponding response audio, and the lip-sync processor generates lip-sync feature data that is strictly aligned with the response audio based on the response text.

[0122] By separating speech synthesis and lip-syncing into two independent but collaborative subtasks, with the speech synthesis processor executing the speech synthesis task and the lip-syncing processor executing the lip-syncing subtask, and each configured with its own dedicated task queue, parallel execution of computationally intensive subtasks is achieved. Even if TTS speech synthesis is time-consuming, the lip-syncing feature engine can still run in advance or synchronously, avoiding the accumulation of end-to-end latency caused by serial dependencies. At the same time, temporary load fluctuations of any functional processor will not block the other path, thus ensuring the real-time performance and naturalness of interactive responses in high-concurrency scenarios.

[0123] In an optional embodiment, the processor layer 103 can define a unified abstract interface to standardize and encapsulate the various functional processors, shielding them from underlying algorithm differences. Each functional processor receives a structured task description, can output standardized results, and supports asynchronous callbacks and state machine management.

[0124] Optionally, processor layer 103 may independently allocate and provide a set of isolated functional processor instances for each digital human instance, including but not limited to: audio processor instances, text processor instances, speech synthesis processor instances, semantic understanding processor instances, and lip-syncing processor instances.

[0125] When creating a digital human instance, the instance manager can send an instance initialization request to the processor layer 103. In response to the instance initialization request, the processor layer 103 can dynamically create the required functional processor instances for the digital human instance and perform initialization operations. For example, initialization operations may include, but are not limited to: allocating independent memory space, computing resources, and context environment for each functional processor instance; registering corresponding task queues for each functional processor so that it can receive tasks from upstream functional processors; and starting an internal status monitoring and heartbeat reporting mechanism for the process manager to perform health checks.

[0126] It should be noted that, in the embodiments of this application, each functional processor in the processor layer 103 allocates a corresponding functional processor instance to each digital human instance during runtime, and the corresponding subtasks of the digital human instance are executed by its dedicated functional processor instance. In describing the digital human interaction system of this application, for ease of expression and to reflect the generality of the architecture, the logical functional modules are collectively referred to as "functional processors," while the actual execution is completed by the corresponding functional processor instances.

[0127] In this embodiment, the digital human interaction system adopts a pipeline architecture combining multi-level task queues and event-driven processing. The end-to-end digital human interaction process is decomposed into multiple atomic subtasks, which are executed asynchronously and in parallel by corresponding functional processors. The multi-level task queue architecture includes text queues, semantic understanding queues, TTS task queues, lip-sync queues, etc., with each queue corresponding to a functional processor at a different processing stage. Loosely coupled collaboration between functional processors can be achieved through a "write queue + issue event" mechanism. Upstream processors do not need to be aware of downstream implementation details; they only need to write the results to the designated queue and issue a ready event, while downstream processors retrieve and process the results as needed.

[0128] Meanwhile, each queue can be independently configured with capacity, priority, and backpressure strategy, enabling computationally intensive tasks (such as semantic understanding tasks) and real-time sensitive tasks (such as speech synthesis tasks) to execute in parallel without blocking each other. In high-concurrency scenarios, even if a processing stage experiences a momentary load spike or latency fluctuation, its impact is limited to the local queue and will not cause the entire pipeline to stall, thereby significantly improving the system's throughput, real-time response, and operational stability.

[0129] The speech synthesis processor can call the IPC manager to write the response audio to the first input channel monitored by the data interaction layer 104. The lip-sync driver processor can synchronously call the IPC manager to write the lip-sync feature data to the second input channel monitored by the data interaction layer.

[0130] The data interaction layer 104 can continuously monitor the first input channel and the second input channel, and match the response audio and lip-shape feature data from the same digital human interaction session based on the session identifier. After confirming that both have arrived, the data interaction layer can fill or truncate the lip-shape feature data according to a preset storage granularity, place it before the response audio, embed relevant data in the header of the data packet, and encapsulate it to obtain a binary data packet.

[0131] The data interaction layer 104 may include an HSF API (High-Speed ​​Service Framework Application Programming Interface), an internal cluster call service interface, and an HTTP service interface. The HSF API can be used for high-performance, low-latency remote procedure calls between microservices within the same region and cluster; the internal cluster call service interface is used to achieve high-concurrency, high-availability inter-node communication within the cluster based on a distributed task scheduling framework, supporting event-driven task distribution and result callbacks; the HTTP service interface provides a standardized network interface to external clients, suitable for cross-platform access scenarios such as web browsers and mobile terminals.

[0132] Channel layer 105 supports both RTQ and RTC channels. Data reception, data transmission, and connection establishment or termination with clients can be performed through either channel. RTQ channels depend on the RTQ SDK (Software Development Kit), and RTC channels depend on the RTC SDK.

[0133] The channel layer 105 can provide a unified channel interface, which shields the differences in the underlying protocols and allows the data interaction layer 104 to call the unified channel interface.

[0134] When sending binary data packets, the data interaction layer 104 can determine the target service interface (such as an HTTP service interface) based on the client type (such as a live viewer terminal, a mobile app, etc.) and send a channel selection request to the channel manager in the coordination layer 102. In response to the channel selection request, the channel manager can dynamically determine the target channel type of the session channel from RTQ and RTC channels based on the client type, network quality, and task type, and return the target channel type and corresponding channel identifier to the data interaction layer 104.

[0135] Data interaction layer 104 can invoke the unified channel interface provided by channel layer 105 according to the target channel type and the corresponding channel identifier. Channel layer 105 can receive binary data packets sent by data interaction layer 104 through the unified channel interface, and send binary data packets to the client through the session channel based on the target channel type.

[0136] In an optional embodiment, the digital human interaction system 100 may further include an engineering infrastructure layer 107. The engineering infrastructure layer 107 may be configured with an end-to-end monitoring system covering the entire digital human interaction chain, used to collect, analyze and provide feedback on the operating status of the digital human interaction system in real time, ensuring service quality and user interaction experience.

[0137] Optionally, the engineering infrastructure layer 107 may include a log recording unit, an SLS (Structured Logging Service) unit, an LLM monitoring unit, a fluency monitoring unit, and a task success rate monitoring unit.

[0138] The log recording unit can embed key nodes in the processor layer 103, data interaction layer 104 and channel layer 105 to record raw logs, including task identifiers, processing time, exception stacks and input / output snapshots.

[0139] SLS units can automatically parse, extract fields, and build indexes from raw logs, and support efficient retrieval and aggregation analysis by session identifier, client identifier, or time window.

[0140] The LLM monitoring unit can statistically analyze the success rate, illusion rate, and semantic relevance score of responses generated by the LLM model, and identify scenarios such as model degradation or prompt word failure.

[0141] The fluency monitoring unit can continuously monitor real-time experience metrics such as end-to-end interaction latency, voice interruption rate, lip-sync error (in milliseconds), and frame rate stability.

[0142] The task success rate monitoring unit can assess the proportion of user goals achieved in multiple rounds of interaction based on the results of dialogue intent recognition, and quantify the effectiveness of the service.

[0143] To address key user experience bottlenecks, the engineering infrastructure layer 107 can also set up specific monitoring strategies, including but not limited to: LLM response latency (i.e., the time from user input to the return of the first character); audio-visual synchronization deviation (i.e., the absolute difference between the client audio playback time and the lip-sync animation driving time); and voice interruption rate (i.e., the proportion of voice segments lost due to network jitter, TTS anomalies, or queue congestion).

[0144] Through the aforementioned multi-dimensional and fine-grained monitoring system, the engineering infrastructure layer 107 can achieve rapid fault location, closed-loop optimization of service quality, and enhanced system stability. It has constructed an intelligent operation and maintenance hub with proactive perception, intelligent diagnosis, and driven optimization, significantly improving the reliability and maintainability of the digital human system.

[0145] The digital human interaction system in this embodiment achieves a highly natural, low-latency, and high-concurrency anthropomorphic interactive experience through a layered decoupled architecture and a multimodal collaborative mechanism: The processor layer can allocate a dedicated functional processor group to each digital human instance and decompose speech synthesis and lip-syncing into parallel subtasks, avoiding end-to-end latency accumulation caused by serial dependencies; the data interaction layer can encapsulate the response audio and lip-syncing feature data into the same binary data packet, where the lip-syncing features are located before the audio and aligned according to a preset storage granularity, with a timestamp embedded within the packet; the client can synchronously drive audio playback and digital human lip-syncing animation based on this timestamp, achieving millisecond-level lip-syncing with an error of <50ms, significantly improving immersion and naturalness; simultaneously, through the channel manager in the coordination layer, combined with client type, network quality, and task real-time requirements, a data transmission path meeting quality requirements can be dynamically selected between the RTC and RTQ channels; the data interaction layer can adapt to different receivers through the HSF API, cluster internal service calls, or HTTP service interfaces, accommodating the same IDC (Internet Data Center). The system boasts microsecond-level response and broad compatibility with public network terminals; the engineering infrastructure layer constructs an end-to-end monitoring system capable of supporting rapid fault location, closed-loop optimization of service quality, and elastic resource scheduling; each layer can be decoupled through standardized contracts such as IPC manager and unified channel interface, supporting independent upgrades and rapid integration with third-party SDKs / APIs (Application Programming Interfaces); while ensuring highly realistic interaction, the overall system has advantages such as high throughput, strong stability, and easy scalability, and can be widely used in large-scale real-time scenarios such as virtual customer service, live streaming e-commerce, and online education.

[0146] Below, based on any of the above embodiments, combined with Figure 4 This provides a method for digital human interaction.

[0147] Figure 4 This is a flowchart illustrating a digital human interaction method provided in an embodiment of this application. Please refer to [link / reference]. Figure 4 The method may include: S401. Receive an interaction request sent by the client. The interaction request is sent during the playback of the narration audio and video of the digital human instance and is used to request real-time interaction with the digital human instance.

[0148] S402. In response to the interaction request, perform the interaction task to generate the response audio and lip feature data corresponding to the interaction request.

[0149] S403. Encapsulate the response audio and lip shape feature data into a binary data packet, in which the lip shape feature data is located before the response audio and aligned according to a preset storage granularity.

[0150] S404. Send binary data packets to the client through a pre-established session channel so that the client can drive the digital human instance to interact with the user in real time during the explanation of the digital human instance, based on the response audio and lip feature data.

[0151] In some optional embodiments, before receiving an interaction request sent by the client, the method further includes: creating a digital human instance in response to a connection request sent by the client; the connection request is sent by the client after detecting an activation operation on the digital human instance; the digital human instance runs in an independent process; establishing a session channel between the digital human instance and the client; initializing at least one functional processor instance corresponding to the digital human instance; and preloading the explanatory audio and the digital human video for the digital human instance.

[0152] In some optional embodiments, the at least one functional processor instance includes an audio processor instance, a text processor instance, a speech synthesis processor instance, a semantic understanding processor instance, and a lip-sync processor instance; if the interaction request includes speech data; in response to the interaction request, generating response audio and lip-sync feature data includes: using the audio processor instance to perform speech activity detection and speech recognition processing on the speech data to obtain target text data, and writing the target text data into a text queue; using the text processor instance to retrieve the target text data from the text queue and send the target text data to the semantic understanding processor; using the semantic understanding processor instance to perform semantic understanding on the target text data to generate response text; using the speech synthesis processor instance to call a text-to-speech algorithm to perform speech synthesis based on the response text to generate the response audio; and using the lip-sync processor instance to call a lip-sync feature engine to generate the lip-sync feature data based on the response text.

[0153] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0154] Below, based on any of the above embodiments, combined with Figure 5 The above-mentioned digital human interaction method will be further explained.

[0155] Figure 5 This is a schematic diagram illustrating a digital human interaction method provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 5In response to a user's activation of a digital human instance, the client can send a connection request to the instance scheduler. The instance scheduler, in turn, responds to the connection request by sending an instance creation request to the coordination layer. The coordination layer, in response to the instance creation request, creates the digital human instance and establishes a session channel between the client and the digital human instance through the channel layer. After establishing the session channel, the channel layer can send a connection success callback to the coordination layer.

[0156] The coordination layer can serve as the core of the digital human interaction system. It allocates a dedicated set of functional processor instances to digital human instances through the processor layer (not shown in the figure), including audio processor instances, text processor instances, semantic understanding processor instances, speech synthesis processor instances, and lip-sync processor instances, and initializes the functional processor instances corresponding to the digital human instances through the processor layer.

[0157] When a user initiates an interaction, the client can generate an interaction request based on the user's input voice or text data and send the request to the channel layer. The interaction request carries the input data (i.e., voice or text data). After receiving the interaction request, the channel layer can send an input data reception event to the coordination layer.

[0158] If the input data is speech data, the coordination layer submits the input data to the audio processor instance corresponding to the digital human instance. The audio processor instance then calls the VAD algorithm in the algorithm layer to perform VAD speech detection and obtain the detection result. If speech is detected, the ASR algorithm in the algorithm layer can be called to perform speech recognition and obtain the target text data. The audio processor instance can then write the target text data to a text queue.

[0159] If the input data is text data, the coordination layer can directly write the text data into the text queue.

[0160] A text processor instance can retrieve target text data from a text queue and send a queue change notification to a semantic understanding processor instance, triggering the semantic understanding processor instance to perform semantic understanding on the target text data and generate a response text.

[0161] The speech synthesis processor instance can call the TTS algorithm in the algorithm layer to perform TTS speech synthesis on the response text, obtain the corresponding response audio, and send the response audio to the data interaction layer (not shown in the figure). The lip-sync processor instance can call the lip-sync feature engine to generate lip-sync feature data based on the response text and send the lip-sync feature data to the data interaction layer.

[0162] The data interaction layer can encapsulate the response audio and lip-sync feature data into a binary data packet, which is then sent to the client through the channel layer. Upon receiving the binary data packet, the client can parse it to obtain the response audio and lip-sync feature data, render the lip-sync animation of the digital human instance based on the lip-sync feature data, and synchronously play the response audio for interactive purposes during the explanation.

[0163] In this embodiment, the digital human interaction system can respond to a connection request sent by the client when it detects an activation operation on the digital human instance. It creates a digital human instance for the client through the scheduling layer and establishes a session channel between the digital human instance and the client through the channel layer. This provides a pre-creation mechanism for the digital human instance and a pre-connection mechanism for the channel, reducing the first-frame response time and effectively meeting the needs of real-time interaction scenarios. Because the response audio and lip-sync feature data can be encapsulated into the same binary data packet through the data interaction layer and sent to the client, inconsistent timing of client reception of the response audio and lip-sync feature data due to heterogeneous packet transmission is avoided, significantly reducing network round trips and improving interaction smoothness. Furthermore, in this binary data packet, the lip-sync feature data precedes the response audio, allowing the client to prioritize parsing the lip-sync feature data and rendering the lip-sync animation. Since the response audio and lip-sync feature data are in binary format and aligned according to a preset storage granularity, the client avoids the overhead of text parsing and deserialization during data parsing, and does not require additional copying, reducing the data parsing time and thus reducing the latency of the digital human responding to the user. In summary, the digital human interaction method provided in this embodiment comprehensively improves the real-time interaction effect of the digital human.

[0164] It should be noted that the execution subject of each step in the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps S401 to S404 can be device A; or the execution subject of steps S401 to S403 can be device A, and the execution subject of step S404 can be device B; and so on.

[0165] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0166] Figure 6 This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Please refer to... Figure 6 The electronic device 60 may include a memory 61 and a processor 62.

[0167] Memory 61 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0168] Processor 62, coupled to memory 61, is configured to execute a computer program in memory 61 for: receiving an interaction request sent by a client, the interaction request being sent during the playback of narration audio and digital human video by the digital human instance, for requesting real-time interaction with the digital human instance; executing an interaction task in response to the interaction request to generate response audio and lip-sync feature data corresponding to the interaction request; encapsulating the response audio and the lip-sync feature data into a binary data packet, wherein the lip-sync feature data is located before the response audio in the binary data packet and aligned according to a preset storage granularity; and sending the binary data packet to the client through a pre-established session channel, so that the client drives the digital human instance to interact with the user in real-time during the narration process based on the response audio and lip-sync feature data while the digital human instance is playing the narration audio and digital human video.

[0169] Optionally, before receiving the interaction request sent by the client, the processor 62 is further configured to: create the digital human instance in response to a connection request sent by the client; the connection request is sent by the client after detecting an activation operation on the digital human instance; the digital human instance runs in an independent process; establish a session channel between the digital human instance and the client; initialize at least one functional processor instance corresponding to the digital human instance; and preload the narration audio and the digital human video for the digital human instance.

[0170] Optionally, the at least one functional processor instance includes an audio processor instance, a text processor instance, a speech synthesis processor instance, a semantic understanding processor instance, and a lip-sync processor instance; if the input data carried by the interaction request is speech data; when the processor 62 generates response audio and lip-sync feature data in response to the interaction request, it is specifically configured to: perform speech activity detection and speech recognition processing on the speech data through the audio processor instance to obtain target text data, and write the target text data into a text queue; retrieve the target text data from the text queue through the text processor instance, and trigger the semantic understanding processor instance to retrieve the target text data; perform semantic understanding on the target text data through the semantic understanding processor instance to generate response text; call a text-to-speech algorithm through the speech synthesis processor instance to perform speech synthesis based on the response text to generate the response audio; and call a lip-sync feature engine through the lip-sync processor instance to generate the lip-sync feature data based on the response text.

[0171] Furthermore, such as Figure 6 As shown, the electronic device also includes other components such as a communication component 63, a display 64, and a power supply component 65.

[0172] Figure 6 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 6 The components shown. Additionally... Figure 6 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the cloud server's product form. The cloud server in this embodiment can be a terminal device such as a desktop computer, laptop, smartphone, or IoT device, or a server-side device such as a conventional server or server array. If the cloud server in this embodiment is implemented as a terminal device such as a desktop computer, laptop, or smartphone, it may include... Figure 6 The components within the dashed box; if the cloud server in this embodiment is implemented as a conventional server, cloud server, or server array, etc., then it may not be included. Figure 6 The component within the dashed box.

[0173] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0174] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0175] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0176] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0177] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0178] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0179] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0180] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A digital human interaction system, characterized in that, include: The scheduling layer is used to create a digital human instance for the client in response to the connection request sent by the client, and load the narration audio and digital human video corresponding to the digital human instance. The digital human instance corresponds to a unique session identifier. The coordination layer is used to route the interaction requests sent by the client during the playback of the explanatory audio and the digital human video by the digital human instance, based on the session identifier; The processor layer is used to perform an interaction task in response to the interaction request, so as to generate response audio and lip feature data corresponding to the interaction request; The data interaction layer is used to encapsulate the response audio and the lip shape feature data into a binary data packet; In the binary data packet, the lip shape feature data is located before the response audio and is aligned according to a preset storage granularity; The channel layer is used to establish a session channel between the digital human instance and the client, and to send the narration audio, digital human video and binary data packets to the client through the session channel, so that the client can drive the digital human instance to play the narration audio and the digital human video, and in the process, drive the digital human instance to interact with the user in real time during the narration based on the response audio and lip feature data.

2. The system according to claim 1, characterized in that, The binary data packet employs a compact encoding format; the binary data packet includes a fixed header and a dynamic payload, wherein... The fixed head includes at least one of the following: lip feature data length, response audio length, timestamp, digital human instance information, and data encoding metadata; The dynamic payload portion includes lip shape feature data and response audio, and the dynamic payload portion is aligned according to the preset storage granularity.

3. The system according to claim 1 or 2, characterized in that, The connection request is sent by the client when it detects an activation operation on the digital human instance.

4. The system according to claim 1 or 2, characterized in that, The processor layer includes a task processing unit, a semantic response unit, and a multimodal data generation unit; the interaction task includes a semantic response generation task and a multimodal data generation task, wherein, The task processing unit is used to generate the semantic response generation task and the multimodal data generation task based on the interaction request, and to distribute the semantic response generation task and the multimodal data generation task to the corresponding processing units; the semantic response generation task and the multimodal data generation task have higher priority than the explanation task; The semantic response unit is used to generate response text based on the input data carried in the interaction request, so as to perform the semantic response generation task; The multimodal data generation unit is used to generate the response audio and the lip shape feature data based on the response text in order to perform the multimodal data generation task.

5. The system according to claim 4, characterized in that, If the input data is speech data, the semantic response generation task includes a speech recognition subtask and a response text generation subtask, and the semantic response unit includes an audio processor, a text processor, and a semantic understanding processor; wherein... The audio processor is used to perform speech activity detection and speech recognition processing on the speech data, generate target text data, and write the target text data into a text queue to execute the speech recognition subtask. The text processor is used to retrieve target text data from the text queue and trigger the semantic understanding processor to retrieve the target text data. The semantic understanding processor is used to perform semantic understanding on the target text data and generate response text to execute the response text generation subtask.

6. The system according to claim 4, characterized in that, The multimodal data generation task includes an audio generation subtask and a lip-syncing generation subtask, and the multimodal data generation unit includes a speech synthesis processor and a lip-syncing processor; wherein, The speech synthesis processor is used to invoke a text-to-speech algorithm to synthesize speech based on the reply text and generate the reply audio in order to perform the audio generation subtask. The lip shape driver processor is used to, based on the response text, invoke the lip shape feature engine to generate the lip shape feature data in order to execute the lip shape driver generation subtask.

7. The system according to any one of claims 1-2 or 5-6, characterized in that, The scheduling layer includes an instance scheduler and an instance manager, wherein, The instance scheduler is used to, in response to the connection request sent by the client, determine the resource configuration of the digital human instance and send an instance creation instruction to the instance manager; The instance manager is used to create the digital human instance in a separate process according to the instance creation instruction, and to load the explanatory audio and the digital human video.

8. The system according to any one of claims 1-2 or 5-6, characterized in that, The coordination layer includes a task manager, a message manager, and a channel manager, wherein... The task manager is used to monitor the status of the interactive task; The message manager is used to monitor events generated by at least one functional processor in the processor layer, and trigger the next functional processor to execute a corresponding task based on the events. The channel manager is responsible for maintaining multiple channel types, determining the target channel type among the multiple channel types based on client type, network quality, and task type, and providing the target channel type to the channel layer so that the channel layer can create the session channel according to the target channel type.

9. The system according to claim 8, characterized in that, The channel layer is also used to provide a unified channel interface to the data interaction layer, receive binary data packets sent by the data interaction layer through the unified channel interface, and send the binary data packets to the client through the session channel.

10. A digital human interaction method, characterized in that, The method, applied to a digital human interaction system, includes: The system receives an interaction request sent by the client, which is sent during the playback of explanatory audio and video by the digital human instance, and is used to request real-time interaction with the digital human instance. In response to the interaction request, an interaction task is executed to generate response audio and lip-shape feature data corresponding to the interaction request; The response audio and the lip shape feature data are encapsulated into a binary data packet, in which the lip shape feature data is located before the response audio and is aligned according to a preset storage granularity; The binary data packet is sent to the client through a pre-established session channel, so that the client can drive the digital human instance to interact with the user in real time during the explanation of the digital human instance, based on the response audio and lip feature data.

11. The method according to claim 10, characterized in that, Before receiving the interaction request sent by the client, the method further includes: In response to a connection request sent by the client, the digital human instance is created; the connection request is sent by the client after detecting an activation operation on the digital human instance; the digital human instance runs in an independent process; Establish a session channel between the digital human instance and the client; Initialize at least one functional processor instance corresponding to the digital human instance; The explanatory audio and the digital human video are preloaded for the digital human instance.

12. The method according to claim 11, characterized in that, The at least one functional processor instance includes an audio processor instance, a text processor instance, a speech synthesis processor instance, a semantic understanding processor instance, and a lip-syncing processor instance; If the input data carried in the interaction request is voice data; In response to the interaction request, generate reply audio and lip feature data, including: Using the audio processor instance, voice activity detection and voice recognition processing are performed on the voice data to obtain target text data, and the target text data is written into a text queue. The text processor instance retrieves target text data from the text queue and triggers the semantic understanding processor instance to retrieve the target text data. The semantic understanding processor instance performs semantic understanding on the target text data to generate a response text. Using the speech synthesis processor instance, a text-to-speech algorithm is invoked to synthesize speech based on the reply text, generating the reply audio. The lip shape driver processor instance invokes the lip shape feature engine to generate the lip shape feature data based on the response text.

13. An electronic device, characterized in that, include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program in the memory to implement the steps of the method according to any one of claims 10-12.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 10-12.

15. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, causes the processor to perform the steps of the method according to any one of claims 10-12.

Images