A real-time interactive digital human system supporting high concurrency and its implementation method

Abstract

The present invention discloses a real-time interactive digital human system supporting high concurrency and its implementation method, which relates to the field of digital human interaction technology. The system includes: a model instance pool module, which is used to load the weights of deep learning models into a shared memory area through memory mapping technology; a multi-threaded scheduling module, which uses a lock-free queue and a dynamic priority algorithm to schedule user requests; an asynchronous pipeline processing module, which is composed of decoupled microservices, and the modules are connected in series through asynchronous message queues; a client SDK, which is used to dynamically switch rendering modes according to terminal hardware performance and network conditions; an elastic scaling module, which is used to monitor resource loads in real time and automatically adjust the number of service instances; an audio and video synchronization calibration module, which is used to ensure that the synchronization error of audio and video frames is lower than a preset threshold through a timestamp alignment algorithm. The present application solution can systematically solve the core challenges in high-concurrency digital human interaction scenarios and provide innovative support for large-scale real-time applications.

Description

Technical Field

[0001] The present invention relates to the field of digital human interaction technology, and in particular to a real-time interactive digital human system supporting high concurrency and an implementation method thereof. Background Art

[0002] With the rapid development of artificial intelligence and computer graphics, virtual humans have become a key research area in human-computer interaction. The core goal is to build highly realistic digital human avatars through technologies such as speech synthesis, natural language processing, and real-time animation rendering to support real-time interactions in scenarios such as intelligent customer service, online education, and virtual live broadcasts. However, large-scale user concurrency poses significant challenges to system architecture: on the one hand, users expect digital humans to provide low-latency, real-time response capabilities; on the other hand, traditional architectures face significant bottlenecks in resource management, multi-module collaboration, and scalability.

[0003] Existing technologies mainly achieve interaction through centralized cloud processing and streaming media transmission, but are limited by problems such as model loading redundancy, inefficient asynchronous scheduling, and accumulated network delays, making it difficult to meet performance requirements in high-concurrency scenarios. Specifically, existing solutions usually load a complete AI model independently for each user session, resulting in a linear increase in memory and computing resource consumption with the number of concurrent users; modules such as speech recognition, semantic understanding, speech synthesis, and animation rendering use independent asynchronous processing and lack a global timestamp synchronization mechanism, which can easily lead to audio and video asynchrony and delay accumulation; existing systems mostly rely on centralized cloud processing, making it difficult to reduce cross-regional delays through edge computing, and lack elastic scaling mechanisms. Therefore, based on the above-mentioned difficulties, the present invention proposes a real-time interactive digital human system that supports high concurrency and its implementation method. Summary of the Invention

[0004] Technical Purpose

[0005] To address the above-mentioned issues, the present invention aims to provide a high-concurrency, real-time interactive digital human system and its implementation method. This system aims to address the high response delays, low resource utilization, and insufficient scalability inherent in digital human real-time interactive systems in high-concurrency scenarios, while achieving low-latency, high-throughput, and robust interactive capabilities through innovative architectural design. Its core objectives include reducing resource waste caused by repeated model loading, optimizing the end-to-end latency of multi-module collaborative processing, enhancing the system's elastic expansion and cross-regional deployment capabilities, and adapting to the rendering requirements of diverse terminal devices to support the efficient operation of large-scale real-time interactive scenarios such as online education and cloud customer service.

[0006] Technical Solution

[0007] In order to achieve the above-mentioned purpose, the present invention provides a real-time interactive digital human system supporting high concurrency and its implementation method, which constructs a lightweight model instance pool through memory mapping technology to realize multi-threaded sharing of the same model weight, thereby reducing memory usage and initialization overhead; adopts dynamic priority algorithm and lock-free queue scheduling task, combined with the streaming processing mechanism of asynchronous pipeline microservices, shortens end-to-end delay; integrates Kubernetes elastic scaling and edge node deployment strategy, dynamically adjusts service instances and optimizes cross-regional request routing; the client adapts to different terminal performance and network conditions through adaptive rendering strategy and adaptive bitrate algorithm; and also introduces audio and video layered synchronization calibration mechanism to ensure interactive coherence. It systematically solves the resource redundancy, delay accumulation and scalability bottlenecks of the existing technology, and realizes efficient real-time interaction under high concurrency.

[0008] In a first aspect, the present invention provides a real-time interactive digital human system supporting high concurrency, comprising:

[0009] The model instance pool module is used to load deep learning model weights into a shared memory area through memory mapping technology, allowing multiple parallel inference threads to share the same model weights;

[0010] A multi-threaded scheduling module that uses a lock-free queue and a dynamic priority algorithm to schedule user requests. The dynamic priority algorithm calculates the priority based on the task waiting time and urgency;

[0011] The asynchronous pipeline processing module consists of multiple decoupled microservices, including speech recognition, semantic understanding, speech synthesis, animation rendering, and streaming media push modules. The modules are connected in series through asynchronous message queues to achieve end-to-end streaming processing;

[0012] The client SDK supports adaptive rendering strategies, dynamically switching between local rendering mode and remote rendering mode based on terminal hardware performance and network conditions;

[0013] The elastic scaling module, based on containerized deployment and Kubernetes orchestration, monitors resource load in real time and automatically adjusts the number of service instances;

[0014] The audio and video synchronization calibration module is used to ensure that the synchronization error between audio and video frames is lower than a preset threshold through a timestamp alignment algorithm.

[0015] Furthermore, the model instance pool module supports hot updating of model weights in shared memory without interrupting the current session. The update process includes the following steps: loading new model weights into a spare memory area through double buffering technology; switching to the new weight area during model inference intervals and releasing the memory occupied by the old weights; the hot update process ensures weight consistency through version number verification.

[0016] Furthermore, the dynamic priority algorithm of the multi-thread scheduling module satisfies the following formula:

[0017] (1);

[0018] Where, is the scheduling priority; and is an adjustable parameter; is the waiting time of task i in the queue; is the urgency of session i;

[0019] The lock-free queue uses atomic operations based on a ring buffer to implement task distribution.

[0020] This method significantly reduces memory usage and initialization delay, solves the bottleneck of resource consumption growing linearly with the number of concurrency in traditional architectures, and improves system throughput and high concurrency support capabilities.

[0021] Furthermore, the asynchronous pipeline processing module adopts a cross-model collaborative reasoning mechanism, combines the output of the large language model with the preset knowledge base rules to generate dialogue responses, and selects the optimal results through the confidence threshold.

[0022] Furthermore, the adaptive rendering strategy of the client SDK includes: in local rendering mode, the client receives parameterized animation instructions issued by the server, and generates digital human images in real time in combination with local personalized configuration; in remote rendering mode, the client receives the complete audio and video stream synthesized by the server, and dynamically adjusts the bit rate through an adaptive bitrate algorithm; the adaptive rendering strategy also integrates a network jitter prediction model, predicts the network status based on historical delay data, and switches the rendering mode in advance.

[0023] Furthermore, the elastic scaling module distributes user requests through a weighted round-robin algorithm based on the geographic location and real-time load of the edge nodes. When the GPU utilization exceeds the threshold, it prioritizes scaling edge node instances deployed in the same geographic area, and retains the minimum instance pool when scaling down to avoid cold start delays.

[0024] This method optimizes resource utilization and network transmission efficiency, reduces cross-regional interaction delays, and enhances the robustness of the system in high-load fluctuations and node failure scenarios, providing scalable underlying architecture support for large-scale real-time interactions.

[0025] Furthermore, the audio and video synchronization calibration module adopts a layered correction strategy. The first layer synchronizes the audio and video timestamps through a global clock, and the second layer dynamically adjusts the frame interval based on audio features and video lip shape key points. When the synchronization error exceeds the threshold, the interpolation frame supplementation or frame drop mechanism is triggered.

[0026] Furthermore, when the WebRTC protocol is unavailable, the streaming media push module automatically downgrades to a low-latency transmission protocol based on QUIC; the switching process ensures data integrity through forward error correction and redundant coding.

[0027] Furthermore, it also includes a model weight optimization module, which is used to dynamically adjust the local parameters of the shared model weight by analyzing user behavior data in real time during the inference process. The updated model weight is:

[0028] (2);

[0029] Where, is the updated model weight; is the shared weight; is the learning rate decay factor; is the weight correction function; is the Sigmoid function; is the L2 norm of the feature vector F.

[0030] This method significantly improves the semantic matching accuracy of digital human responses and user interaction satisfaction without increasing additional resource overhead, while maintaining the global sharing characteristics of model weights and avoiding the drawbacks of redundant model loading in traditional personalization solutions.

[0031] Furthermore, it also includes an audio and video synchronization enhancement module, which is used to train a lightweight temporal convolutional network using historical audio and video frame sequences to predict the audio features and mouth shape key points of future frames. The features of the next k frames are predicted using the following formula:

[0032] (3);

[0033] Where, is the predicted audio and video features of the next k frames; It is a temporal convolutional network; is the input historical frame sequence;

[0034] The dynamic synchronization correction is defined as:

[0035] (4);

[0036] Where, is the dynamic synchronization correction; is the modified strength coefficient; is the index variable; For the Predictive characteristics of the moment; For the the actual characteristics of the moment; For the current moment L2 norm of the eigenvector.

[0037] This method significantly improves the audio and video synchronization accuracy and the system's anti-interference ability through predictive frame buffering and dynamic fusion strategies, while ensuring the stability of end-to-end delay and overcoming the response lag defect of traditional a posteriori calibration methods in high-concurrency scenarios.

[0038] In a second aspect, the present invention further provides a method for implementing a real-time interactive digital human system that supports high concurrency. The method is based on the system described in the first aspect, comprising:

[0039] On the server side, the deep learning model weights are loaded into a shared memory area through memory mapping technology for parallel calls by multiple threads. Each thread independently maintains the session state and shares the same model weights.

[0040] A dynamic priority algorithm based on a lock-free queue is used to schedule user requests, and tasks are assigned to idle model instances through atomic operations.

[0041] The speech recognition, semantic understanding, speech synthesis, and animation rendering modules are decoupled into independent microservices. Streaming processing is achieved through asynchronous message queues. Each module triggers downstream processing based on segmented output to shorten end-to-end latency.

[0042] The rendering mode is dynamically switched based on the terminal hardware performance and network conditions. High-performance terminals generate digital human images through local parameterized instructions, while low-performance terminals receive the complete audio and video stream synthesized in the cloud and adjust the transmission bit rate based on the adaptive bit rate algorithm.

[0043] Based on containerized orchestration and edge node deployment strategies, server resource load is monitored in real time, service instances are dynamically scaled up and down, and user requests are assigned to the nearest edge node through geographic routing.

[0044] The timestamp alignment algorithm ensures that the synchronization error between audio and video frames is lower than the preset threshold.

[0045] In a third aspect, the present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a management platform, the method for implementing the aforementioned real-time interactive digital human system supporting high concurrency is implemented.

[0046] This invention achieves multi-threaded parallel reasoning and end-to-end streaming processing through shared memory mapping of the model instance pool, a dynamic priority scheduling algorithm, and an asynchronous pipeline microservice module. It combines elastic scaling mechanisms with edge node deployment to optimize resource allocation, and ensures cross-terminal interaction fluency through client-side adaptive rendering strategies and spatiotemporal prediction synchronization algorithms. This solution systematically addresses core issues in traditional technologies, such as redundant model loading, audio and video synchronization errors, low resource utilization, and insufficient scalability. It significantly reduces interaction latency, improves system throughput and robustness, and supports personalized semantic adaptation and high-precision synchronization in dynamic network environments, providing an efficient, stable, and economical technical implementation path for large-scale real-time interaction scenarios.

[0047] Beneficial effects

[0048] By implementing the above-mentioned real-time interactive digital human system supporting high concurrency and its implementation method provided by the present invention, the following technical effects are achieved:

[0049] (1) This application uses memory mapping technology to load model weights into a shared memory area, combining lock-free queues with a dynamic priority scheduling algorithm to achieve efficient reuse of the same model weights across multiple parallel inference instances. This approach significantly reduces memory usage and initialization latency, resolving the bottleneck of traditional architectures where resource consumption increases linearly with the number of concurrent connections, and improving system throughput and high concurrency support.

[0050] (2) Through Kubernetes container orchestration and dynamic routing of edge nodes, automatic scaling of service instances and intelligent distribution of cross-region requests are achieved. This approach optimizes resource utilization and network transmission efficiency, reduces cross-region interaction latency, and enhances the system's robustness in high-load fluctuations and node failure scenarios, providing a scalable underlying architecture for large-scale real-time interaction.

[0051] (3) By analyzing user interaction characteristics in real time and dynamically adjusting the local parameters of the shared model weights, session-level personalized reasoning is achieved. This method significantly improves the semantic matching accuracy of digital human responses and user interaction satisfaction without adding additional resource overhead, while maintaining the global sharing characteristics of model weights and avoiding the drawbacks of redundant model loading in traditional personalization solutions.

[0052] (4) By using a lightweight temporal convolutional network to predict future audio and video features and combining it with a feedforward error compensation mechanism, we can effectively reduce audio and video synchronization issues caused by network jitter or computational fluctuations. This method significantly improves the accuracy of audio and video synchronization and the system's anti-interference capability through predictive frame buffering and dynamic fusion strategies, while ensuring the stability of end-to-end latency and overcoming the response lag of traditional a posteriori calibration methods in high-concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to make the above-mentioned real-time interactive digital human system supporting high concurrency and its implementation method of the present invention more obvious and easy to understand, the following is a brief introduction to the drawings required for use in the specific implementation of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 A schematic diagram showing the system architecture of this application;

[0055] Figure 2 A diagram showing the backend microservices and asynchronous pipeline architecture;

[0056] Figure 3 A diagram showing elastic scaling and edge node scheduling;

[0057] Figure 4 Represents the interaction process sequence diagram;

[0058] Figure 5 A schematic diagram showing the logic of selecting a rendering strategy in the client SDK. DETAILED DESCRIPTION

[0059] Example 1:

[0060] Provides a real-time interactive digital human system that supports high concurrency and its implementation method. The system architecture is as follows Figure 1 As shown, it specifically includes: a model instance pool module, which is used to load deep learning model weights into a shared memory area through memory mapping technology, so that multiple parallel inference threads can share the same model weights; a multi-thread scheduling module, which uses a lock-free queue and a dynamic priority algorithm to schedule user requests. The dynamic priority algorithm calculates the priority based on the task waiting time and urgency; an asynchronous pipeline processing module, which is composed of multiple decoupled microservices, including speech recognition, semantic understanding, speech synthesis, animation rendering, and streaming media push modules. The modules are connected in series through asynchronous message queues to achieve end-to-end streaming processing; a client SDK, which supports adaptive rendering strategies and dynamically switches between local rendering mode and remote rendering mode according to terminal hardware performance and network conditions; an elastic scaling module, which monitors resource load in real time and automatically adjusts the number of service instances based on containerized deployment and Kubernetes orchestration; and an audio and video synchronization calibration module, which uses a timestamp alignment algorithm to ensure that the synchronization error between audio and video frames is below a preset threshold. The details are as follows.

[0061] 1. Model loading logic and multi-threaded scheduling algorithm

[0062] To achieve real-time interaction between digital humans in a highly concurrent environment, the system uses a lightweight inference model instance pool and multi-threaded shared scheduling technology on the server side to efficiently share AI model weights and support parallel processing of multiple user requests. Specifically, the following features are implemented:

[0063] 1. During system startup, the model instance pool module deployed in the backend pre-loads lightweight AI models for key functions of the digital human. The loading process uses memory mapping technology, that is, mapping the model weight file to a shared memory area in read-only mode. This method ensures that all subsequently created model instances reference the same memory area, avoiding repeated loading of model weights, thereby reducing memory consumption and initialization delays. During the loading process, the system also performs integrity checks to ensure the correctness of the data. After loading is complete, each model instance logically generates a reference handle, which is used for subsequent inference calculations, and each instance only maintains its own dynamic state, thereby achieving state isolation.

[0064] 2. After a user request arrives, the system encapsulates the request data into a task and enters it into the global asynchronous task queue. To ensure low-latency distribution in a high-concurrency environment, the scheduler uses a lock-free queue to queue tasks. For task i, its scheduling priority is calculated according to the following formula:

[0065] (1);

[0066] Where, is the scheduling priority; and It is an adjustable parameter used to balance waiting time and task urgency; is the waiting time of task i in the queue; is the urgency or historical completion rate of session i, such as the ratio of unresponded tasks in consecutive interactions;

[0067] The scheduler is based on The value selects the idle model instance for task processing to ensure that each thread is in an independent session state Call the shared model function , and its reasoning process is expressed as:

[0068] (5);

[0069] Where, Output the results for reasoning; is a shared model function; It is an independent session state; Enter data for the user.

[0070] To ensure multi-threaded safety, all model instances share this read-only memory area through memory mapping after weights are loaded. Each thread maintains only independent session state to ensure data isolation. Furthermore, the multi-threaded scheduling module uses a lock-free queue to queue tasks, determines task scheduling priority based on a scheduling priority formula, and selects idle model instances to assign threads to tasks, thus achieving multi-threaded, highly concurrent reasoning with state isolation.

[0071] Through the above-mentioned model loading and multi-threaded shared scheduling design, the system can achieve efficient reuse of the same AI model weights under high concurrency conditions, so that memory consumption does not increase linearly with the number of concurrency, greatly improving the system's throughput and response speed.

[0072] 2. Asynchronous Pipeline and Microservice Module Design

[0073] Backend microservices and asynchronous pipeline architecture Figure 2 As shown, the backend system adopts a microservices architecture. Core functional modules are decoupled through standard interfaces and asynchronous message queues, forming a streamlined parallel processing flow. Each module can operate independently, but also form an end-to-end data pipeline through a unified session scheduling and task distribution mechanism, achieving low-latency, high-concurrency, real-time interaction with digital humans. To this end, this module mainly includes the following modules and their collaborative working mechanisms:

[0074] 1. The WebSocket access module is responsible for establishing and maintaining a persistent connection between the client and the server. This module generates a unique session ID upon session initialization and ensures connection stability through heartbeat detection. It also encapsulates user-uploaded voice or text data in a predefined format and forwards it to the backend message queue for asynchronous processing by subsequent modules.

[0075] 2. ASR recognition module, used to implement automatic speech recognition, transcribes user-uploaded voice data into text in real time. This module uses a streaming algorithm and combines short-term energy and zero-crossing rate to detect endpoints. The basic formula is:

[0076] (6);

[0077] Where, is the short-time energy of the nth frame, which is used to measure the strength of the speech signal in that frame; is the window length; is the sampling point index within the window; is the speech signal amplitude of the mth sampling point in the nth frame;

[0078] (7);

[0079] Where, is the zero-crossing rate of the nth frame, reflecting the frequency of the signal passing through the zero point in the frame; It is a sign function that returns the sign of the sampling point.

[0080] The ASR module gradually outputs the recognition results in a streaming manner, and instantly pushes part of the transcribed text to the semantic understanding module through an asynchronous message queue.

[0081] 3. The semantic understanding and dialogue generation module performs natural language understanding on the ASR output text based on a large language model or a pre-built knowledge base, generating the digital human's response text. To reduce latency, this module supports streaming output, that is, the model generates answers sentence by sentence. The generation process is described by a recursive function:

[0082] (8);

[0083] Where, is the generation step, which represents the time interval or text span from the current moment to the next generation moment; It is a generation function, representing the core logic of the semantic understanding and dialogue generation model; is the text sequence generated at the current time t; is the conversation context information; To input text.

[0084] The generated reply is segmented and immediately passed to the TTS speech synthesis module through the message queue.

[0085] 4、 The TTS speech synthesis module is used to convert the answer text into the speech of the digital human using the neural network TTS model. The speech synthesis adopts a streaming synthesis strategy. The process is expressed as follows:

[0086] (9);

[0087] Where, is the generated speech signal; Standardized naming prefixes for generated functions; is the input text sequence; are TTS model parameters.

[0088] The TTS speech synthesis module generates and outputs audio data during the synthesis process to shorten the overall response delay.

[0089] 5. Animation driver and rendering module, used to generate digital human facial expressions and mouth movements based on the audio output of the TTS speech synthesis module and its related features using the speaker generation model. This module uses a time-series convolutional neural network or LSTM architecture. The output video frame timestamps are calibrated with the audio playback time to ensure audio and video synchronization. The calibration formula is:

[0090] (10);

[0091] Where, is the calibration output; The audio playback time; is the video frame timestamp; is the absolute value symbol. When it is less than the preset threshold, it is considered synchronized, otherwise buffer correction is triggered.

[0092] 6. The WebRTC streaming module encodes and encapsulates the audio and video data output by the TTS speech synthesis module and the animation driver and rendering module, and pushes it to the client in real time via the RTP protocol using WebRTC. This module supports dynamic adjustment of encoding parameters to adapt to varying network conditions, achieving end-to-end latency within milliseconds.

[0093] 7. The Task Distribution and Coordination Module is responsible for scheduling and distributing tasks across modules and coordinating intermediate results. This module utilizes unified timestamps and synchronization signals to ensure consistent data output order across all stages within a session, preventing audio and video asynchrony caused by varying processing speeds. It also implements non-blocking transmission through asynchronous message queues, improving overall throughput.

[0094] 8. The model state synchronization and user data isolation module is used to transmit necessary historical states in continuous sessions to ensure the consistency of digital human dialogue and actions; at the same time, each user's data is logically completely isolated to ensure that the data is only used within the normal range, protect user privacy and prevent data confusion between different sessions.

[0095] The system adopts a pipeline parallel design, and non-blocking transmission is achieved between modules through asynchronous message queues, so that the processing of each stage can be overlapped. The overall response time is close to the sum of the average processing time of each module, rather than a strict accumulation, thus achieving an end-to-end response time of less than 1.5 seconds. Specifically, it includes:

[0096] Step 1: After the user's voice data is uploaded through the WebSocket access module, it is immediately processed by the ASR recognition module. The ASR recognition module uses a streaming recognition algorithm to output partial text in real time.

[0097] Step 2: The recognition result is passed to the semantic understanding and dialogue generation module through the message queue. This module uses the streaming large language model to generate the answer text based on the context and passes it to the TTS speech synthesis module in sentence segments.

[0098] Step 3: The TTS speech synthesis module starts streaming speech synthesis immediately after receiving the text, outputs the audio stream while synthesizing, and notifies the animation driver and rendering module to generate the corresponding video frames;

[0099] Step 4: The model state synchronization module corrects the timestamps of the audio and video frames to ensure strict alignment, and then transmits them to the client in real time via the WebRTC streaming module.

[0100] Step 5. During the entire process, the task distribution and coordination module uses unified timestamps and asynchronous queue management to ensure that data within the same session is delivered in sequence and in an orderly manner. Each module can process tasks of different sessions simultaneously, achieving highly parallel pipeline processing.

[0101] Through this asynchronous pipeline and module collaboration mechanism, each microservice module can fully utilize server resources in high-concurrency scenarios to achieve real-time interactive effects with low latency and high throughput.

[0102] 3. Cross-platform deployment and verification

[0103] The backend of this system uses containerization technology and Kubernetes orchestration to achieve unified deployment across platforms and regions. When the system starts the container, the model instance pool module uses preheating technology to load the model weight file into the shared memory area through memory mapping. During the loading process, the system performs a file integrity self-check to ensure that the model data is accurate. By using memory mapping and sharing technology, each microservice module can achieve consistent behavior on operating systems such as Linux, Windows, and macOS without the need to repeatedly load model weights, thereby reducing memory consumption and improving startup efficiency. The modules communicate with each other through standard interfaces to ensure interoperability and cross-platform consistency between services.

[0104] Furthermore, the system supports multi-regional distributed deployment, with backend service clusters deployed simultaneously across multiple data centers and edge nodes globally. Intelligent routing mechanisms automatically distribute user requests to the nearest node, reducing network latency and improving user experience. Rigorously validated in the cloud, at edge nodes, and across diverse terminal environments, the system maintains stable operation in high-concurrency scenarios, maintaining millisecond-level response times and smooth adaptive video streaming during real-world interactions.

[0105] The client SDK has also been fully cross-platform tested and verified on PC web pages, Android, iOS, and AR / VR devices. It implements data interaction and media transmission through standard WebSocket and WebRTC protocols, ensuring that low-latency, real-time interaction performance requirements can be met in various terminals and network environments.

[0106] 4. Exception Handling and Fault Tolerance Mechanism

[0107] To ensure the stable operation of the system in high concurrency and network fluctuation environments, an exception handling and fault tolerance mechanism has been added. Each module has set up strict exception monitoring, error capture, retry and automatic expansion and contraction strategies during operation, including:

[0108] 1. Each module has a clear processing timeout. For example, if the ASR recognition module does not return a recognition result within 500ms, it will automatically trigger a retry or call a backup algorithm. Similarly, if the TTS speech synthesis module detects a long pause during the speech synthesis process, it will immediately switch to the backup model or return a degraded voice prompt. The retry mechanism uses an exponential backoff algorithm, whose formula is:

[0109] (11);

[0110] Where, Output for exponential backoff; is the basic retry delay; The current number of retries;

[0111] Through this mechanism, the system can automatically remedy when some modules encounter temporary failures, ensuring that critical processes are not blocked for a long time.

[0112] 2. To ensure the stability of WebSocket and WebRTC channels in persistent connections, the system implements periodic heartbeat packet detection on both sides. If multiple consecutive heartbeat packets go unanswered, the disconnection reconnection mechanism is automatically triggered, notifying the session scheduling module to reestablish the connection, thus ensuring that all sessions remain uninterrupted. This mechanism effectively mitigates network fluctuations and brief connection interruptions, ensuring continuous and real-time data transmission.

[0113] 3. Each module implements a try-catch exception catching mechanism to comprehensively capture runtime exceptions. Captured exception information is recorded in detail and reported through a unified monitoring platform. The scheduling module takes appropriate action based on the exception type, such as redistributing tasks, switching to a backup instance, or initiating downgrade processing. This mechanism prevents local errors from spreading throughout the system and provides a detailed basis for subsequent troubleshooting.

[0114] 4. When the system detects excessive load or frequent anomalies in a module, it automatically activates fault tolerance and degradation strategies. For example, in high-concurrency scenarios, core functionality is prioritized, and the animation rendering module can simplify some detailed processing. When a node fails, the system reassigns unfinished tasks to other healthy nodes through the task queue. This way, even if some functions are temporarily degraded, the entire system can maintain the consistency and stability of the core interaction process.

[0115] 5. The system has a built-in elastic scaling module, elastic expansion and contraction and edge node scheduling Figure 3 As shown, the system monitors the CPU, GPU, and memory usage of each container instance in real time. When resource utilization reaches a preset threshold, the system automatically triggers capacity expansion, quickly launching new instances through the container orchestration platform and registering them with the service registry for task allocation by the scheduling module. When the load decreases, idle instances are automatically released to ensure efficient resource utilization and reduce operating costs. This capacity expansion not only ensures stable system response in high-concurrency environments but also improves the overall scalability and flexibility of the system.

[0116] The interaction process is as follows Figure 4 As shown in the figure, this system uses end-to-end streaming processing to achieve real-time interaction between users and digital humans. The overall system process combines the cloud-edge-end collaborative architecture and asynchronous pipeline processing mechanism to ensure that each processing link can run overlappingly in a high-concurrency environment, with a total response time of less than 1.5 seconds. The specific process is as follows:

[0117] First, the user inputs voice through the microphone on the terminal device, and the data is uploaded to the access layer in real time via a persistent WebSocket connection. The access layer intelligently routes the user request to a pool of idle model instances in the backend service cluster. The model instance pool uses a multi-threaded shared scheduling mechanism to efficiently share the weights of the same deep learning model. This ensures that the model weights are loaded only once, and each parallel thread calls the shared model for inference processing in an independent session state.

[0118] The ASR recognition module then uses a segmented recognition algorithm to transcribe and output partial text in real time. The conversation management module sequentially passes the recognized text to the semantic understanding and dialogue generation module. This module uses a streaming large language model, combined with the conversation context and input text, to generate response text sentence by sentence using a recursive function. Subsequent processing begins after the first sentence is generated, achieving streaming output.

[0119] The generated response text is then passed to the TTS speech synthesis module, which uses a streaming synthesis strategy to generate speech data segment by segment based on a neural network TTS model. During the synthesis process, the audio features output by the TTS speech synthesis module synchronously drive the animation driver and rendering module. This module uses a time-series convolutional neural network or LSTM architecture to generate video frames of the digital human's facial expressions and lip movements based on the audio features. To ensure strict synchronization of audio and video data, the system uses a timestamp calibration algorithm.

[0120] Finally, the integrated audio and video data is encoded via the WebRTC streaming module and pushed to the client in real time using the RTP protocol. The client uses an adaptive algorithm to dynamically adjust the video bitrate based on network conditions to ensure smooth playback. The entire processing pipeline utilizes an asynchronous parallel mechanism, with each module immediately initiating subsequent processing after completing a partial output. Unified session management and task scheduling ensure the correct and sequential delivery of data from each session, achieving closed-loop real-time interaction.

[0121] 5. Practical Application Scenarios and Industry Implementation Capabilities

[0122] This system has a wide range of practical application scenarios and industry landing value, and can provide technical support for large-scale user interaction needs in various fields, including but not limited to the following examples:

[0123] In the field of online customer service, this system can be deployed as a cloud customer service agent to provide a large number of customers with 24 / 7 uninterrupted intelligent interactive services. Through the high concurrency support of this system, the customer service center of an e-commerce platform or operator can provide real-time voice Q&A and problem handling for hundreds or even thousands of users at the same time. The digital human customer service image can communicate with customers through fluent voice and expressions, answer common questions, guide business processing, or accept complaints and suggestions. With the help of edge node deployment, user requests from different regions will be routed to the nearest server node for processing, reducing network latency and improving the interactive experience. In the cloud customer service scenario, this system can also reduce server resource usage under large-scale deployment through the model instance pool mechanism, saving operating costs for enterprises while providing a consistent and high-quality service experience.

[0124] In the field of distance education and training, this system can be used as a virtual teacher or intelligent teaching assistant in large-scale online classes. Hundreds of students can each interact with the digital human teacher in real time, asking and answering questions, and the system's high-concurrency architecture ensures that each student receives a timely response and that the interactions are independent and undisturbed. For student devices with better terminal performance, the client SDK can choose local rendering mode to achieve high-resolution teacher image presentation; for mobile or low-performance terminals, cloud rendering is used to ensure smoothness. The digital human teacher can combine voice recognition and knowledge base to answer students' questions in real time, or actively explain teaching content according to the course progress. By deploying edge computing nodes close to campuses or educational institutions, this system can also reduce network latency for distance education. In online education scenarios, this system provides a scalable method for teacher-student interaction, supporting a large number of students online simultaneously while improving teaching quality, and facilitating the large-scale dissemination of high-quality educational resources.

[0125] In summary, this system can adapt to the large-scale user interaction needs of multiple fields and has excellent scenario-based implementation capabilities. Through cloud deployment and edge collaboration, this system can be flexibly applied to the various scenarios mentioned above and customized and expanded according to actual business needs, thereby significantly improving the level of service automation and intelligence in various industries.

[0126] 6. Client Adaptation Solution

[0127] The client SDK rendering strategy selection logic is as follows Figure 5 As shown, in order to fully meet the computing power, display capabilities of different terminal devices and the real-time interaction requirements in various network environments, this system provides a set of flexible and efficient client SDK rendering modes to ensure that the digital human interaction experience achieves the best effect on all platforms. Specifically, it includes the following aspects:

[0128] The system supports two rendering modes:

[0129] Local rendering mode: On high-performance terminals, the server only delivers processed commands and control parameters, including audio streams, lip motion parameters, expression curves, and animation compensation information. The client SDK utilizes the local GPU for real-time graphics rendering to generate the digital human video. This mode not only fully utilizes the rendering capabilities of the terminal hardware but also supports personalized customization. For example, users can locally adjust the digital human's skin and clothing, achieving high-quality, personalized digital human display effects.

[0130] Remote Rendering Mode: For lower-performance devices or scenarios requiring uniform output, the server completes the entire digital human audio and video synthesis process and pushes the resulting complete video stream to the client in real time via the WebRTC protocol with low latency. This mode ensures a consistent, high-quality viewing experience regardless of the device's computing power and is suitable for low-power devices or browser-based applications.

[0131] In addition, the client SDK features adaptive network bandwidth regulation. The system uses an adaptive bitrate algorithm to monitor current network latency, packet loss, and bandwidth conditions in real time, and dynamically adjusts video resolution, frame rate, and bitrate to ensure high-definition video when bandwidth is sufficient. In poor network conditions, the bitrate and resolution are automatically reduced to avoid lag and maintain smooth interaction. This adaptive algorithm is based on historical network statistics and real-time feedback, and uses the following algorithm model:

[0132] (12);

[0133] Where, is the adjusted bit rate; is the basic bit rate; is the currently measured bandwidth; The target bandwidth threshold.

[0134] Based on the above design, the client adaptation solution can not only fully utilize the computing power of high-performance terminals to achieve personalized, high-quality rendering, but also provide stable cloud rendering output for low-performance terminals. Combined with the adaptive network adjustment strategy, the system can maintain a smooth and realistic real-time interactive experience of digital humans under various terminal and network conditions.

[0135] In summary, this system not only technically achieves low latency, low resource consumption, and high scalability for digital human systems in high-concurrency environments, but also demonstrates its cross-platform deployment, stability, and cost-effectiveness in practice. This system architecture provides a mature, economical, and efficient real-time interactive solution for scenarios such as online education, cloud customer service, and live broadcasts of large-scale events, with significant market application prospects and technological innovation value.

[0136] Example 2:

[0137] Building on the previous examples, a dynamic model weight optimization mechanism based on user behavior feedback has been added. By analyzing user behavior data in real time during inference, the local parameters of the shared model weights are dynamically adjusted, thereby improving the personalization and accuracy of digital human responses without adding additional model loading overhead. The core of this mechanism is to encode user behavior characteristics as weight correction factors, integrating them with the basic model weights to achieve session-level adaptive reasoning.

[0138] During the conversation, user interaction data is collected in real time, including response delay tolerance, semantic keyword matching, and sentiment tendency score, and the feature vector F is generated through normalization processing;

[0139] Correction function based on weights , the updated model weights are:

[0140] (2);

[0141] Where, is the updated model weight; is the shared weight; is the pre-trained correction matrix; is the learning rate decay factor; is the weight correction function; is the Sigmoid function, which is used to control the correction amplitude; is the L2 norm of the feature vector F;

[0142] In the model instance pool, each session thread is calculated independently based on user characteristics , and perform reasoning based on this. After the session ends, the weight correction value is automatically recycled and the shared weight remains unchanged.

[0143] Verification shows that while achieving an average error similar to that of the above-mentioned embodiment, the dynamic weight optimization mechanism improves the semantic matching accuracy by 15%-20%, the weight correction only increases the single inference time by 3ms, and the memory usage increase is less than 1%. The results show that this mechanism can significantly improve the semantic accuracy and personalized adaptation capabilities of digital human responses by dynamically adjusting the shared model weights. The semantic matching accuracy is significantly optimized during user interaction, and user satisfaction scores show a systematic improvement. At the same time, because the weight correction process only acts on local parameters at the session level, the global model weights remain shared, and the growth of system memory usage and computing overhead is effectively controlled at an extremely low level, avoiding the resource waste caused by multiple model instantiations in traditional personalization solutions.

[0144] Example 3:

[0145] Building on the previous examples, an audio and video synchronization enhancement algorithm based on spatiotemporal prediction is added. This algorithm uses historical audio and video frame sequences to train a lightweight temporal convolutional network, predicting the audio features and lip shape key points of future frames. This generates buffered frames in advance to reduce synchronization errors. By fusing the predicted results with the actual output, feedforward error compensation is implemented.

[0146] Audio fundamental frequency , energy spectrum And the coordinates of the key points of the mouth shape in the video Extract and construct spatiotemporal sequences ;

[0147] Use TCN network to predict the features of future k frames:

[0148] (3);

[0149] Where, is the predicted audio and video features of the next k frames; It is a temporal convolutional network; is the input historical frame sequence;

[0150] Define the dynamic synchronization correction amount:

[0151] (4);

[0152] Where, is the dynamic synchronization correction; It is the correction strength coefficient used to control the replacement ratio of the prediction frame; is the index variable; For the Predictive characteristics of the moment; For the the actual characteristics of the moment; For the current moment L2 norm of the eigenvector.

[0153] when When , the predicted frame is enabled to replace the actual frame; otherwise, the weight Fuse predictions with actual frames;

[0154] Where, The threshold determines whether to enable the prediction frame, usually 0.1; is the fusion weight, and the actual frame ratio is , the predicted frame ratio is .

[0155] Suppose, during real-time interaction, a digital human system processes a 300ms speech input segment and generates corresponding video frames. Sudden network jitter causes the arrival of audio frames 6-8 to be delayed. Using a spatiotemporal prediction algorithm, buffered frames are generated in advance to compensate for synchronization errors.

[0156] The input data is the audio fundamental frequency, energy spectrum, and lip key point coordinates of the historical frames (t=1 to t=10), and the prediction target is the audio and video features of the next three frames;

[0157] Based on the historical data from t=1 to t=10, the audio fundamental frequency, energy spectrum, and mouth key point coordinates from t=11 to t=13 are predicted. The actual t=11 frame is delayed and the system starts predicting the frame. Replacement;

[0158] When the t=11 frame actually arrives, calculate the prediction error:

[0159]

[0160] Therefore, the predicted frame is fully used Render.

[0161] The effect of the audio and video synchronization enhancement algorithm is shown in Table 1.

[0162] Table 1. Summary of the effects of audio and video synchronization enhancement algorithms

[0163]

[0164] The experimental table shows that at a 10% packet loss rate, synchronization error decreased from 55ms to 15ms, a 72.7% reduction; the occurrence of audio and video desynchronization decreased from 18% to 4%, a 77.8% reduction; and end-to-end latency increased by only 0.2%-0.3%. The TCN model took 0.5ms to predict a single frame, and memory usage increased by 50KB. The results show that the algorithm, by predicting future audio and video features and implementing feed-forward error compensation, can significantly reduce audio and video desynchronization caused by network jitter or computational delay. In tests simulating complex network environments, the system's tolerance to sudden delays and packet loss was significantly enhanced, audio and video synchronization errors were reduced to negligible levels, and the overall end-to-end latency did not incur any additional burden due to the prediction computation. Furthermore, the lightweight temporal convolutional network design ensures efficient prediction, with computational overhead having minimal impact on the system's real-time performance.

[0165] Example 4:

[0166] Based on the above embodiments, this embodiment deploys and tests a high-concurrency digital human real-time interaction system.

[0167] The server-side architecture is deployed in a cloud cluster using a containerized microservices architecture, orchestrated and managed by Kubernetes. First, upon cluster startup, the various deep learning models for the digital human are preloaded, and the model weights are loaded into a shared memory area using memory mapping technology. This ensures that all model instances in the cluster reference the weight data at the same memory address, preventing duplicate model file loading on each service node. Subsequently, a multi-threaded scheduling service is initiated, establishing a global asynchronous task queue to receive user session requests. The scheduler utilizes lock-free queues and atomic operations to ensure efficient concurrent access and dynamically assigns tasks to idle model instance threads based on a predefined priority algorithm. Several compute nodes are deployed on the edge as extended server instances. When users connect locally, intelligent routing dispatches requests to the nearest edge node, reducing regional network latency. After the cloud-edge collaborative deployment is complete, the client connects to the server using the distributed SDK and begins interacting with the digital human. The entire system maintains WebSocket persistent connections and session routing through an access-layer gateway, ensuring that user requests are quickly delivered to idle backend nodes for processing. In high-concurrency scenarios, Kubernetes automatically scales out additional container instances to accommodate additional sessions. Through the above deployment process, end-to-end collaboration is achieved from cloud clusters, edge nodes to client terminals, providing low-latency real-time digital human interaction services for high-concurrency users.

[0168] The client SDK adopts a modular design, which includes a network communication module, a rendering engine module and an adaptive control module. The network communication module sends the user's voice or text input through the WebSocket protocol, and receives audio and video streams or rendering instructions from the server through the WebRTC protocol; the rendering engine module is responsible for generating digital human images locally or remotely according to different modes. Specifically, this system supports two modes: local rendering and remote rendering: when it is detected that the client device has sufficient computing power, the system selects the local rendering mode, and the server only sends down the processed audio data, mouth shape and expression parameters and other control information. The client SDK calls the local rendering engine to synthesize the digital human image in real time and play the voice synchronously. When the client is a low-performance device or is in a scenario that requires unified output, the system switches to remote rendering mode, and the server completes the complete digital human audio and video synthesis, and pushes the encoded audio and video stream to the client with low latency through WebRTC. The client only needs to decode and play it. The client SDK's adaptive control module monitors the terminal's network status in real time and dynamically adjusts rendering strategies. In local rendering mode, poor network conditions reduce texture resolution or frame rate. In remote rendering mode, an adaptive bitrate algorithm automatically adjusts the video bitrate and resolution, ensuring smooth interaction regardless of network conditions. Through the SDK's internal logic and strategy switching, all end users can achieve optimal digital human interaction.

[0169] Under the above deployment architecture, this embodiment tested system performance. The test environment consisted of five backend GPU servers and two geographically distributed edge nodes. By simulating a high-concurrency user voice conversation scenario, the system operated stably under a load of approximately 1,000 users interacting online simultaneously. Test results showed that the average end-to-end response time per user was approximately 1.2 seconds. This means that the delay from the start of a user's speech to the start of playback of the digital human video and voice feedback was approximately 1.2 seconds. 95% of user sessions had a response delay of less than 1.5 seconds, meeting the requirements of real-time interaction. With 500 concurrent users, the CPU utilization of a single server was approximately 65%, the GPU utilization was approximately 75%, and the memory usage was approximately 60% of the total capacity. Compared to traditional architectures that load a complete model for each session, resulting in a linear increase in memory and GPU usage with concurrency, this system significantly slows resource usage growth by sharing a pool of model instances. For example, with 100 concurrent users, total memory usage only increased by approximately 20% compared to a single user, demonstrating the effectiveness of model weight reuse. At the same time, by leveraging the elastic scalability of container orchestration, when the number of concurrent users surged to 1,000, the system promptly expanded capacity and added container instances. The overall CPU / GPU utilization remained within a safe range, and there was no server overload or crash. This verified the availability and stability of this system architecture in actual high-concurrency scenarios.

[0170] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media containing computer-usable program code.

[0171] The present invention can provide computer program instructions to a management platform of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the management platform of the computer or other programmable data processing device produce a device for implementing the system.

[0172] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions of the system.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions of the described system.

Claims

1. A real-time interactive digital human system supporting high concurrency, characterized by: include: The model instance pool module is used to load deep learning model weights into a shared memory area through memory mapping technology, and supports hot updates of model weights in shared memory using double buffering and version number checking without interrupting the current session; Multi-threaded scheduling module, which uses lock-free queues and dynamic priority algorithms to schedule user requests; Asynchronous pipeline processing module, each microservice module is connected in series through an asynchronous message queue to achieve stream processing. The asynchronous pipeline processing module can trigger the stream processing of the downstream microservice module after the output of the upstream microservice module; The client SDK is used to dynamically switch between local rendering mode and remote rendering mode based on the terminal hardware performance and network conditions, adjust the transmission bit rate based on the adaptive bit rate algorithm in remote rendering mode, and receive the interactive data generated by the asynchronous pipeline processing module; The elastic scaling module is used to monitor server resource load in real time through the Kubernetes container orchestration mechanism and automatically adjust the number of service instances; The audio and video synchronization calibration module is used to ensure that the synchronization error between audio and video frames is lower than a preset threshold through a timestamp alignment algorithm; The model weight optimization module is used to dynamically adjust the local parameters of the shared model weight by analyzing user behavior data in real time during the inference process; The audio and video synchronization enhancement module is used to train a lightweight temporal convolutional network using historical audio and video frame sequences, predict the audio features and mouth shape key points of future frames, generate buffer frames in advance, and achieve feedforward error compensation by fusing the prediction results with the actual output.

2. The system according to claim 1, wherein: The process of the model instance pool module performing hot update of model weights includes: loading the new model weights into the spare memory area, switching to the memory area where the new model weights are located during the idle interval of model inference, and releasing the memory occupied by the old model weights.

3. The system according to claim 1, wherein: The dynamic priority algorithm of the multi-thread scheduling module satisfies the following formula: (1)； Where, is the scheduling priority; and is an adjustable parameter; is the waiting time of task i in the queue; is the urgency of session i; The lock-free queue uses atomic operations based on a ring buffer to implement task distribution.

4. The system according to claim 1, wherein: The asynchronous pipeline processing module adopts a cross-model collaborative reasoning mechanism, combines the output of the large language model with the preset knowledge base rules to generate dialogue responses, and selects the optimal results through the confidence threshold.

5. The system according to claim 1, wherein: The elastic scaling module distributes user requests based on the geographic location and real-time load of edge nodes through a weighted polling algorithm. When GPU utilization exceeds a threshold, it prioritizes scaling edge node instances deployed in the same geographic area, and retains the smallest instance pool when scaling down.

6. The system according to claim 1, wherein: The audio and video synchronization calibration module adopts a layered correction strategy, in which the first layer synchronizes audio and video timestamps through a global clock, the second layer dynamically adjusts the frame interval based on audio features and video lip shape key points, and triggers interpolation or frame drop mechanisms when the synchronization error exceeds a threshold.

7. The system according to any one of claims 1 to 6, characterized in that: The model weight optimization module collects user interaction data in real time during the session, including response delay tolerance, semantic keyword matching, and sentiment tendency score, generates feature vectors through normalization processing, and updates the model weights according to the weight correction function: (2)； Where, is the updated model weight; is the shared weight; is the learning rate decay factor; is the weight correction function; is the Sigmoid function; is the L2 norm of the feature vector F.

8. The system according to any one of claims 1 to 6, characterized in that: The audio and video synchronization enhancement module extracts the audio fundamental frequency, energy spectrum and the coordinates of the key points of the video mouth shape, constructs a spatiotemporal sequence, and predicts the audio and video features of the next k frames: (3)； Where, is the predicted audio and video features of the next k frames; It is a temporal convolutional network; is the input historical frame sequence; The dynamic synchronization correction is defined as: (4)； Where, is the dynamic synchronization correction; is the modified strength coefficient; is the index variable; For the Predictive characteristics of the moment; For the the actual characteristics of the moment; For the current moment L2 norm of the eigenvector; when When , the predicted frame is enabled to replace the actual frame; otherwise, the weight Fusion of predicted frame and actual frame; Where, The threshold determines whether to enable the prediction frame, usually 0.1; is the fusion weight, and the actual frame ratio is , the predicted frame ratio is .

9. A method for implementing a high-concurrency real-time interactive digital human system, characterized by: The method is implemented based on the system according to any one of claims 1 to 8: The method comprises: The deep learning model weights are loaded into a shared memory area through memory mapping technology. Each thread independently maintains the session state and shares the same model weights. A dynamic priority algorithm based on a lock-free queue is used to schedule user requests, and tasks are assigned to idle model instances through atomic operations. Decouple different modules into independent microservices and implement stream processing through asynchronous message queues; Dynamically switch rendering modes based on terminal hardware performance and network conditions, and adjust transmission bitrate based on adaptive bitrate algorithm; Based on containerized orchestration and edge node deployment strategies, server resource load is monitored in real time, the number of service instances is dynamically scaled, and user requests are assigned to the nearest edge node through geographic routing. The timestamp alignment algorithm ensures that the synchronization error between audio and video frames is lower than the preset threshold.

10. A computer-readable storage medium storing a computer program, wherein: The computer program implements the method according to claim 9 when executed.

Images