A highly compatible pluggable digital human real-time interaction system and method

By using mirror deployment, a global plugin registry, and a three-thread concurrent architecture, the component coupling problem in existing digital human real-time interaction systems is solved, enabling component decoupling and dynamic replacement, and improving the system's scalability and availability.

CN123114576APending Publication Date: 2026-11-17WUXI GUOKE DIGITAL INTELLIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611282185.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-24
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing digital human real-time interaction systems suffer from poor scalability and maintainability due to their monolithic architecture design, which results in tight coupling between functional modules. This makes it difficult to replace or add components without modifying the core system code.

Method used

It adopts an architecture consisting of an image deployment layer, an infrastructure layer, a service and session management layer, a global plugin registry, and a digital human real-time interaction engine layer. It achieves component decoupling and dynamic replacement through a Docker container engine, an environment self-check script, a global plugin registry, and a three-thread concurrent architecture, supporting one-click image deployment and self-check and self-start.

Benefits of technology

It enables independent development, registration, and replacement of components without modifying business logic code, improving system scalability and availability. It supports multi-engine parallel backup and ensures rapid deployment and maintenance on any cloud platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114576A_ABST
    Figure CN123114576A_ABST
Patent Text Reader

Abstract

This invention discloses a highly compatible, pluggable, real-time interactive system and method for digital humans, comprising: an image deployment and infrastructure layer, which constructs virtualized containers based on standard container images and achieves one-click self-starting through an environment self-check script; a global pluggable registry, containing four sub-registries: TTS engine, digital human model, output transmission protocol, and audio feature extraction, which achieves automatic component registration and runtime replacement through a predefined mapping table combined with dynamic module import; a service and session management layer, which manages session lifecycle and LRU cache resources; and a real-time interactive engine layer for digital humans, which adopts a three-thread concurrent architecture to realize pipelined processing of audio synthesis, feature extraction, lip-sync inference, and frame synthesis, and achieves millisecond-level interruption response through message queue clearing and state switching. This invention achieves cross-platform one-click deployment, hot-swappable components, and a smooth real-time interactive experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence interaction technology, and in particular to a highly compatible pluggable digital human real-time interaction system and method. Background Technology

[0002] Real-time digital human interaction technology is a technical solution that comprehensively utilizes multimodal artificial intelligence technologies such as speech synthesis, lip-syncing, and video generation to construct digital avatars with human-like appearance and interactive capabilities in virtual space. In recent years, with the rapid development of deep learning technology and the widespread application of large language models, digital humans have been widely used in online education, virtual customer service, live-streaming e-commerce, social entertainment, and other fields, becoming an important form of human-computer interaction.

[0003] Existing real-time digital human interaction systems typically employ a monolithic architecture design, with tightly coupled functional modules. Specifically, core modules such as speech synthesis engines, digital human driving models, audio feature extraction components, and audio / video output transmission protocols are often hard-coded into the system code, resulting in strong dependencies between them. When a component needs to be replaced (such as replacing one speech synthesis engine with another) or a new component needs to be added, developers have to modify the core system code, recompile, and redeploy the entire application, leading to poor system scalability and maintainability. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a highly compatible pluggable digital human real-time interaction system and method.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Firstly, this application provides a highly compatible, pluggable, real-time interactive digital human system, including:

[0007] The image deployment and infrastructure layer provides a containerized runtime environment, including an environment self-check script, a Docker container engine, and a model data volume. The Docker container engine builds and starts virtualized containers based on standard container images, which are read-only template files of the system's required runtime environment. The environment self-check script automatically performs environment self-checks when the virtualized containers start. The model data volume stores preprocessed digital human data.

[0008] The service and session management layer is used to receive and process external interaction requests and manage the lifecycle and resources of digital human interaction sessions.

[0009] The global plugin registry includes a TTS engine registry for maintaining class mapping relationships among various speech synthesis engines, a digital human model registry for maintaining class mapping relationships among various digital human driving models, an output transmission protocol registry for maintaining class mapping relationships among various audio and video output protocols, an audio feature extraction registry for maintaining class mapping relationships among various audio feature extraction components, and a registry management unit. The registry management unit is used to register and discover components by combining a predefined component mapping table with module import-triggered registration.

[0010] The digital human real-time interaction engine layer includes a TTS engine instance, an audio feature extractor, a lip-sync inference engine, and a frame synthesis output pipeline. The TTS engine instance receives text message queues and outputs audio frames. The audio feature extractor converts the input audio stream into feature vectors and pushes them into a feature queue. The lip-sync inference engine consumes the feature vectors in batches and, combined with the audio frames generated by the TTS engine, drives the digital human model to generate lip-sync inference frames. The frame synthesis output pipeline receives the inference frames, fuses them with background frames to generate the final video stream, and pushes the video stream to different user access terminals through the output transmission protocol module according to the type of client request.

[0011] The user access terminal is used to receive and display the audio and video streams output by the system.

[0012] Furthermore, the Docker container engine mounts the host machine's GPU device through container runtime parameters for use by the digital human inference engine within the container, and optimizes network communication performance by reusing parameters of the host machine's network protocol stack; the standard container image is constructed by integrating virtual environment management tools, language runtime dependency libraries, digital human real-time interactive program code, and preset model files on a container base image that supports GPU-accelerated algorithm inference; the environment self-check script executes environment self-checks including at least: checking whether the graphics processor GPU driver is normal, checking whether the CUDA version of the unified computing device architecture is compatible with the deep learning framework within the container, verifying the integrity of the preset model files, and injecting the application programming interface (API) key.

[0013] Furthermore, the service and session management layer includes:

[0014] A unified API gateway is used to receive client requests and assign unique session identifiers;

[0015] The session manager is used to create independent digital human interaction session instances and asynchronously dispatch interaction tasks to the real-time interaction engine layer according to the instantiation link in the global plugin registry.

[0016] The Task Manager provides asynchronous task scheduling capabilities for the Session Manager, executing session-related time-consuming model loading operations through a thread pool.

[0017] The Digital Human Cache Manager is used to cache pre-processed digital human data in memory or video memory based on the LRU policy. When the maximum cache limit is reached, the protected list and active list are excluded, the least used digital human is evicted, and resource release operations are performed.

[0018] Furthermore, the registry management unit is used to register and discover components by combining a predefined component mapping table with module import-triggered registration. Specifically, this includes: predefining a mapping table from the names of four types of components to module paths in the initialization program at the system entry point; traversing the mapping table when the system starts and calling the dynamic module import mechanism to load the corresponding module files; when a module is imported, the class with the registration decorator is executed during the class definition phase, and the class object is written into the corresponding registry dictionary in key-value pairs to complete the registration; it supports dynamic injection of new plugins at runtime, loading new component modules through the dynamic module import mechanism, or replacing components by directly calling the class constructor.

[0019] Furthermore, the digital human real-time interaction engine layer adopts a three-thread concurrent architecture, including:

[0020] The main rendering thread unit is used to drive the ASR stepping and implement dynamic flow control through the frame queue backlog. Specifically, in each loop, the ASR is first driven to process one frame of audio data, and then the backlog of the output layer buffer corresponding to the output transmission protocol is immediately queried. When the backlog reaches or exceeds a preset threshold, the main thread is dynamically put to sleep according to a preset formula based on the backlog. The larger the backlog, the longer the sleep time, thus naturally reducing the call frequency of the ASR stepping. When the backlog falls back below the threshold, the normal operating rate is restored.

[0021] The inference thread unit is used to consume feature vectors in batches and call the lip-sync inference engine to perform lip-sync inference.

[0022] The frame synthesis output thread unit is used to synthesize the inference results into the original background frame and push audio and video data through a unified output module.

[0023] The second aspect: Provides a highly compatible, pluggable, real-time interactive method for digital humans, including the following steps:

[0024] Image deployment steps: Build a standard container image based on a container base image that supports GPU-accelerated algorithm inference. During the image building phase, package the real-time interactive program code for the digital human, all dependent libraries, virtual environment, and preset model files all at once.

[0025] After the standard container image is pushed to the image repository, the virtualization container is pulled and started on the target running platform, and the GPU device is mounted through the container running parameters. When the container starts, the built-in environment self-check script is automatically triggered to perform multi-stage environment self-checks. After the self-checks pass, the digital human main process is automatically started, and the service behavior is configured through the command line starting parameters.

[0026] Component registration steps: When the main service starts, it iterates through the system's predefined mapping table of component names to module paths, and imports each plugin module sequentially through the dynamic module import mechanism. During the import process, the automatic registration logic of the registration marker is triggered, and the classes of various interactive components are registered in key-value pairs to the TTS engine registry, digital human model registry, output transmission protocol registry, and audio feature extraction registry under the global plugin registry. The default digital human model is warmed up by injecting preprocessed data into the digital human cache manager, and then the service is started and connected to the unified API gateway, listening on the specified port.

[0027] Interactive session steps: The unified API gateway receives connection requests initiated by clients, completes signaling interaction and connection establishment; the session manager assigns a unique session identifier to each connection and creates an independent digital human interactive session instance; it receives text or audio input from users, determines whether to call the large language model to generate reply text based on the request type, and then sequentially synthesizes audio through the TTS engine instance, extracts feature vectors through the audio feature extractor, performs lip-sync inference through the lip-sync inference engine, and synthesizes the inference frame with the background frame through the frame synthesis output pipeline. Finally, the audio and video streams are pushed to the client in real time through the output transmission protocol.

[0028] Furthermore, this also includes interrupting the recovery process:

[0029] When an interruption request is received from the client, the message queue of the TTS engine instance is cleared to prevent subsequent text from entering the synthesis process, and the TTS engine state is switched from running state to paused state, so that TTS stops outputting new audio frames to the pipeline.

[0030] At the same time, the input audio queue of the audio feature extractor is cleared, so that the audio feature extractor continuously outputs silent frames. When the inference thread consumes a silent frame, it outputs a background frame instead of a lip-sync inference frame. After the audio frames already in transit in the pipeline are naturally cleared, the digital human remains in the silent broadcast state. When a new interaction request enters the TTS message queue, the state is automatically restored to the running state, triggering the normal synthesis and output of the new audio frame.

[0031] Furthermore, in the interactive session step, the digital human inference pipeline adopts a three-thread concurrent architecture, including:

[0032] The main rendering thread is used to drive the ASR (Automatic Speech Recognition) step and to control the flow of upstream input by limiting the frame queue by a preset threshold.

[0033] The inference thread is used to consume the feature vectors in batches and call the lip-sync inference engine to perform lip-sync inference and generate inference frames.

[0034] The frame synthesis output thread is used to synthesize the inference frame into a preset background frame to obtain the target audio and video data, and transmit it to the output layer corresponding to the target output transmission protocol.

[0035] The rendering main thread, inference thread, and frame synthesis output thread collaborate in a pipeline through a frame queue of preset capacity.

[0036] Furthermore, the flow control of the main rendering thread specifically includes: in each loop, the main rendering thread first drives ASR to process one frame of audio data, and then immediately queries the buffer backlog of the output layer corresponding to the output transmission protocol; when the backlog reaches or exceeds a preset threshold, the main thread is dynamically put to sleep according to a preset sleep formula based on the backlog, the larger the backlog, the longer the sleep time, thereby naturally reducing the call frequency of ASR stepping; when the backlog falls back below the threshold, the main thread resumes normal operation speed.

[0037] Furthermore, in the interruption recovery step, the time for the audio frames already in transit in the pipeline to be naturally emptied is twice the inference batch size multiplied by the duration of each frame, where the duration of each frame is 20ms.

[0038] The beneficial effects of this invention are as follows:

[0039] (1) Plug-in architecture achieves true component decoupling: Through the decorator mechanism of the global registry, the TTS engine, digital human model and output protocol can be developed, registered and replaced independently without modifying any business logic code.

[0040] (2) One-click image deployment achieves self-check and self-start: Through Docker image packaging, CUDA dependencies, Python environment, all third-party libraries and model files are packaged at once during the image building stage; when the container starts, the environment self-check script automatically completes the initialization verification such as GPU driver detection, CUDA version compatibility adaptation and API key injection. After the self-check is passed, the service process is automatically started, and it can be pulled and self-started on any cloud platform.

[0041] (3) High availability with multiple engines in parallel and with mutual backup: Multiple TTS engines and digital human models can be registered in the same system for easy switching. When one engine fails, it can be quickly switched to a backup engine to improve system availability. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a block diagram of the highly compatible pluggable digital human real-time interactive system of the present invention.

[0044] Figure 2 This is a flowchart of a highly compatible pluggable digital human real-time interaction method according to the present invention. Detailed Implementation

[0045] Example 1

[0046] This embodiment provides a highly compatible pluggable digital human real-time interaction system, referring to... Figure 1 As shown, the system includes a mirror deployment and infrastructure layer, a service and session management layer, a global plugin registry, a digital human real-time interaction engine layer, and a user access terminal. The following provides a detailed description of each functional module.

[0047] I. Mirror Deployment and Infrastructure Layer

[0048] The image deployment and infrastructure layer is used to provide a containerized runtime environment, which includes the Docker container engine, environment self-check scripts, and model data volumes.

[0049] 1. Docker container engine

[0050] The Docker container engine is used to build and launch virtualized containers based on standard container images. A standard container image is a read-only template file containing the runtime environment required by the system. It is built by integrating virtual environment management tools, language runtime dependency libraries, real-time interactive digital human program code, and preset model files onto a container base image that supports GPU-accelerated algorithm inference. As an optional implementation, the container base image includes, but is not limited to, nvidia / cuda:11.6.1-cudnn8-devel-ubuntu20.04 or its equivalent alternatives.

[0051] During the image building phase, the following installations and configurations are completed in one go: conda environment creation and initialization, PyTorch and its CUDA dependencies installation, all Python dependency library installation, and copying of the digital human interaction program code and default model files. After the build is complete, the standard container image is pushed to the image repository for storage.

[0052] On the target operating platform, the standard container image is pulled and the virtualized container is started using the `docker run` command. The host machine's GPU device is mounted using the `--gpusall` parameter for use by the digital human inference engine within the container, and the host machine's network protocol stack is reused using the `--network=host` parameter to optimize network communication performance.

[0053] 2. Environment self-test script

[0054] The environment self-check script is used to automatically perform a multi-stage environment self-check when the virtualization container starts. When the container starts, the entry point automatically triggers the built-in environment self-check script to perform initialization verification. The multi-stage environment self-check includes at least: checking whether the graphics processing unit (GPU) driver is functioning correctly, checking whether the CUDA version of the unified computing device architecture is compatible with the deep learning framework within the container, verifying the integrity of the preset model files, and injecting the application programming interface (API) key. After the self-check passes, the digital human main process is automatically launched and connected to the unified API gateway.

[0055] As an optional implementation, the system also provides an auxiliary startup script for non-containerized deployment scenarios. This script performs the following in sequence: conda environment activation, injection of API key (such as DASHSCOPE_API_KEY, TENCENT_APPID, etc.) environment variables, and background startup of the digital human main process via nohup, specifying various startup parameters (including --transport, --model, --avatar_id, --tts, --max_session, etc.).

[0056] 3. Model Data Volume

[0057] The model data volume is used to store digital human preprocessing data, including video frame sequences, face detection coordinates, mask data, etc. This data volume exists independently of the container and is retained even after the container is destroyed, supporting multiple containers sharing the same data volume.

[0058] II. Global Plugin Registry

[0059] The global plugin registry serves as the core of the system, comprising the TTS engine registry, digital human model registry, output transmission protocol registry, and audio feature extraction registry. Each sub-registration maintains the class mapping relationship of corresponding category components using a key-value pair structure. All components inherit from their corresponding abstract base classes (including but not limited to BaseTTS, BaseAvatar, BaseOutput, and BaseASR) to ensure interface consistency.

[0060] The global plugin registry is generated as follows: The system predefines a mapping table of four types of component names to module paths in the entry point's initialization procedure. This is maintained in tabular form; for example, the digital human model mapping table contains:

[0061] The TTS engine mapping table contains entries such as `musetalk→avatars.musetalk_avatar` and `wav2lip→avatars.wav2lip_avatar`; the output transport protocol mapping table contains entries such as `webrtc→streamout.webrtc` and `rtmp→streamout.rtmp`; and the audio feature extraction mapping table contains entries such as `whisper→asr.whisper` and `hubert→asr.hubert`. When developers add new components, they need to add new entries to the corresponding mapping tables.

[0062] When the system starts, it iterates through the predefined mapping table and calls the dynamic module import mechanism to load the corresponding module file for each entry. This mechanism locates the file based on the module's full path and executes the module's top-level code. When a module is imported, classes with registration declarations (such as `@register(category, name)`) are executed during the class definition phase, writing the class object as key-value pairs to the corresponding registry dictionary (such as `_REGISTRY["tts"]["edgetts"]=EdgeTTS`), thus completing the registration. The registry supports multi-category isolation, with components in each category uniquely identified by name.

[0063] When the system starts, it searches for the corresponding class in the registry based on startup parameters (such as --ttsedgetts, --modelwav2lip), calls the class's constructor with configuration parameters to create an instance, and completes the initialization of the specific component. During subsequent service operation, the session manager can also dynamically call the creation interface of the registry snap-in to create component instances for new sessions.

[0064] In addition to importing and registering components at startup, the system also supports dynamically injecting new components at runtime. When a new component needs to be added, it can be registered by loading the new component module through the dynamic module import mechanism, or by directly calling the class's constructor to replace the component, supporting dynamic switching of components during service operation.

[0065] III. Service and Conversation Management Layer

[0066] The service and session management layer is used to receive and process external interaction requests and manage the lifecycle and resources of digital human interaction sessions. It includes a unified API gateway, session manager, task manager, and digital human cache manager.

[0067] 1. Unified API Gateway

[0068] The unified API gateway is used to receive client requests and assign unique session identifiers. For WebRTC connection requests, the unified API gateway handles SDP signaling interaction and ICE connection establishment.

[0069] 2. Session Manager

[0070] The session manager is designed using a singleton pattern and is responsible for maintaining all active digital human interaction sessions. Each session is uniquely identified by a UUID and internally holds an independent digital human instance and its associated TTS engine instance and ASR instance. The session manager provides interfaces for creating, searching, and destroying sessions, and supports asynchronous calls to time-consuming model loading operations via the task manager's thread pool.

[0071] Specifically, the session manager searches for and instantiates the corresponding target interactive component from the global plugin registry based on the startup parameters, builds a component instantiation chain for each digital human interactive session instance, and then asynchronously dispatches the interactive task to the digital human real-time interactive engine layer.

[0072] 3. Task Manager

[0073] The task manager is used to provide asynchronous task scheduling capabilities for the session manager, and executes session-related time-consuming model loading operations through a thread pool.

[0074] 4. Digital Human Cache Manager

[0075] The Digital Human Cache Manager employs an LRU eviction policy to manage pre-processed digital human data (including video frame sequences, face detection coordinates, mask data, etc.) cached in memory or video memory. A maximum cache size can be configured (default is 10). When this limit is reached, the longest-inactive (non-protected, non-default roles) digital humans are prioritized for eviction, and deep cleanup ensures complete resource release.

[0076] Deep cleaning specifically includes the following steps: removing the entries of the eliminated digital humans from the cache dictionary; iterating through each value in the preprocessed data dictionary of the digital humans, and calling the clear method to clear all element references in the list if the value is a list type; deleting the reference of each value one by one; calling the clear method of the dictionary to clear the dictionary itself; deleting dictionary object references; and finally triggering the memory reclamation mechanism to force a complete garbage collection to ensure that all memory occupied by the eliminated digital humans (including NumPy array objects, list containers, etc.) is released in a timely manner.

[0077] IV. Digital Human Real-Time Interaction Engine Layer

[0078] The digital human real-time interaction engine layer is used to execute specific business logic for multimodal data processing, lip-sync inference, and video frame synthesis. It includes a TTS engine instance, an audio feature extractor, a lip-sync inference engine, a frame synthesis output pipeline, and a three-thread concurrent architecture.

[0079] 1. TTS Engine Examples

[0080] A TTS engine instance is a pluggable component created by the session manager based on startup parameters, which retrieves the corresponding class from the global plugin registry's TTS engine registry and creates it via a constructor call. Its unified interface receives text message queues, processes them internally using the `process_tts` thread, and outputs PCM audio frames with a 16kHz sampling rate and a 20ms frame length (320 samples / frame). It supports more than 10 TTS engines, including but not limited to EdgeTTS, CosyVoice, GPT-SoVITS, Tencent Cloud TTS, Doubao TTS, AzureTTS, and FishSpeech.

[0081] Interruption response is achieved through a combination of message queue clearing and state switching: When an interruption request is received, the TTS message queue is cleared to prevent new text from entering the synthesis process, and the internal state of the TTS is switched from RUNNING to PAUSE; the TTS thread checks the state before each synthesis of a new audio frame, and if it is PAUSE, it stops outputting new audio frames to the pipeline, thereby blocking the continuous synthesis of new audio.

[0082] 2. Audio Feature Extractor

[0083] The audio feature extractor is used to convert the input audio stream into feature vectors and push them into the feature queue. It supports multiple feature extractors, including Whisper, HuBERT, Mel spectrum, etc., and the feature output dimensions of each model are independent (Whisper: 1024-dimensional, HuBERT: 1024-dimensional, Mel: 80-dimensional).

[0084] 3. Lip-sync inference engine

[0085] The lip-sync inference engine is used to batch consume the feature vectors and, in conjunction with the audio frames generated by the TTS engine instance, drives the digital human model to generate lip-sync inference frames. It supports switching between multiple digital human models, including Wav2Lip, MuseTalk, Ultralight, Wav2LipLS, and Dinet, with each model connected to a unified inference pipeline via a registry mechanism. The default inference batch size (batch_size) is 16, but it is configurable.

[0086] 4. Frame synthesis output pipeline

[0087] The frame synthesis output pipeline is used to receive the inference frame, fuse it with the background frame to generate the final video stream, and push the video stream to different user access terminals through the output transmission protocol module according to the type of client request.

[0088] The frame synthesis specifically includes the following steps:

[0089] a) Obtain information of the current frame from the three sets of coordinate sequences of the digital human preprocessing data: The frame synthesis output thread retrieves the inference frame and the corresponding audio frame data from the frame queue. Based on the cycle index idx and the current role identifier avatarid carried by the audio frame, it retrieves the corresponding data from the preloaded coordinate sequence coord_list_cycle (stores the coordinates of the face bounding box of each frame), the frame sequence frame_list_cycle (stores the original complete video frame image), and the face sequence face_list_cycle (stores the corresponding cropped facial region image).

[0090] b) Inference Frame Scaling and Edge Blending: The inference frame (i.e., the lip region image output by the digital human-driven model) is scaled to match the size of the face bounding box recorded in the current frame in `coord_list_cycle`. Then, the scaled inference frame undergoes three-edge processing: linear gradient blending is performed pixel-by-pixel on the top, left, and right edges (with a gradient width of 20 pixels, weighted according to distance from the edge from far to near), eliminating the stitching boundary between the inference frame and the original background frame. Simultaneously, an analog chroma key matting mask is constructed based on the green color difference channel (by subtracting the G and B channel pixel values ​​of the face region in the original background frame; pixels with a difference greater than 60 are identified as green background regions, and the original background pixels of these regions are retained), ensuring that non-lip facial regions in the original frame are not covered by the inference frame.

[0091] c) Composite Frame Overlay and Output: The scaled and edge-blended lip-inference frame is overlaid onto the face region of the original background frame to obtain the final composite video frame. This frame is then pushed to the target output transmission protocol through a unified output module.

[0092] The three sets of coordinate sequences used in the above process—coord_list_cycle, frame_list_cycle, and face_list_cycle—are extracted and stored from the source video during the digital human preprocessing stage using face detection and feature point localization algorithms. Frame_list_cycle represents all the original frames of the source video, face_list_cycle represents the facial region image cropped from the original frames based on the face bounding box (used as input for model inference), and coord_list_cycle represents the coordinate position of the face bounding box in the original frame.

[0093] 5. Three-thread concurrent architecture

[0094] The digital human inference pipeline adopts a three-thread concurrent architecture, including a rendering main thread, an inference thread, and a frame synthesis output thread. The threads cooperate in pipeline parallelism through a frame queue of preset capacity.

[0095] The main rendering thread drives the speech recognition ASR stepping and controls upstream input flow by limiting the frame queue to a preset threshold. Specifically, in each loop, the main rendering thread first calls `asr.run_step()` to drive ASR to process one frame of audio data, and then immediately queries the buffer backlog of the output layer corresponding to the output transmission protocol (i.e., the number of video frames generated but not yet sent over the network). When the backlog reaches or exceeds the preset threshold (default is 5 frames), the main thread is dynamically put to sleep according to the formula `sleep(0.04 × number of backlog frames × 0.8)`. The larger the backlog, the longer the sleep time, thus naturally reducing the call frequency of ASR stepping and reducing the queuing speed of new audio frames. When the backlog falls below the threshold, the main thread resumes normal operation. Through the above mechanism, the main rendering thread achieves adaptive response to downstream inference and output pressure, completing pipeline flow limiting from the output layer to the input layer without the need for explicit locking or backpressure signal protocols.

[0096] The inference thread is used to consume the feature vectors in batches and call the lip-sync inference engine to perform lip-sync inference and generate inference frames.

[0097] The frame synthesis output thread is used to synthesize the inference frame into a preset background frame to obtain the target audio and video data, and transmit it to the output layer corresponding to the target output transmission protocol.

[0098] V. User Access Terminal

[0099] The user access terminal is used to receive and display the audio and video streams output by the system, including a browser, a streaming server, and a virtual camera or local playback device. The output transmission protocol module supports at least one or more of the following protocols: WebRTC, RTMP, VirtualCam, and RTCPush, each corresponding to a different user access terminal: WebRTC corresponds to a browser, RTMP corresponds to a streaming server, and VirtualCam or RTCPush corresponds to a virtual camera or local playback device.

[0100] Implementation 2

[0101] The following is in conjunction with the appendix Figure 2 This invention provides a detailed description of the specific implementation of the highly compatible pluggable digital human real-time interaction method. The invention aims to achieve highly compatible real-time digital human interaction through containerized one-click deployment, dynamic pluggable registration, an efficient three-threaded inference pipeline, and a low-latency interruption mechanism.

[0102] Phase 1: Deployment Phase (Image Building and Auto-Start). This phase aims to achieve containerized deployment of the system, shielding it from underlying environment differences and ensuring rapid service startup on any cloud platform that supports NVIDIA GPUs. It includes the following steps:

[0103] Step S101: Build a standardized Docker image

[0104] The system administrator first builds a standard container image of the system in the local development environment based on the official GPU-accelerated base image provided by NVIDIA (such as nvidia / cuda:11.6.1-cudnn8-devel-ubuntu20.04). During the image building phase (i.e., the Dockerfile building process), the following core components are packaged in one go using layered instructions:

[0105] Environment initialization: Install the Miniconda virtual environment management tool, create and activate the specified Python runtime environment (Python 3.10 is preferred).

[0106] Dependency installation: Install PyTorch and its corresponding CUDA dependencies, as well as all third-party Python libraries required by the project, via pip.

[0107] Code and Model Solidification: Copy the source code of the real-time interactive digital human program and the preset default digital human model files (such as Wav2Lip, MuseTalk, etc.) to the specified working directory within the image.

[0108] The completed image is pushed to a private or public image repository.

[0109] Step S102: Cloud container startup and hardware mounting

[0110] On the host machine of the target cloud platform (such as Alibaba Cloud, AWS, or local IDC), execute the `docker pull` command to obtain the latest image, and then use the `docker run --gpusall --network=host` command to start the virtualized container. The `--gpusall` parameter is responsible for fully mounting the host machine's GPU device into the container, providing hardware computing power support for subsequent deep learning inference; the `--network=host` parameter reuses the host machine's network protocol stack to optimize the network transmission performance of audio and video streams.

[0111] Step S103: Environment self-check and process auto-start

[0112] After the container starts, the entrypoint script automatically triggers the built-in environment self-check script. This script performs multi-dimensional initialization checks:

[0113] GPU driver detection: Verify whether the container can correctly identify the host machine's GPU device and check whether the driver version meets the minimum requirements.

[0114] CUDA compatibility adaptation: Detects the compatibility between the CUDA runtime version in the container and the host machine driver version to ensure that the deep learning framework can normally call GPU acceleration.

[0115] Model file integrity check: Checks whether the preset model file is corrupted or not fully downloaded.

[0116] API key injection: Read and inject API keys (such as LLM large model interface keys) from environment variables or mounted configuration files.

[0117] If all validations pass, the script will automatically start the digital human main process; if validation fails, it will log errors and exit safely to prevent the service from running with defects. System startup parameters (such as --transportwebrtc, --max_session10) can be passed via command line or encapsulated scripts for flexible configuration.

[0118] Phase Two: Registration Phase (Plugin Registration and Initialization)

[0119] This phase utilizes a global plugin registry mechanism to achieve dynamic discovery and decoupling of core system components, allowing developers to extend new functionalities without modifying the underlying code. The steps are as follows:

[0120] Step S201: Traverse the predefined mapping table

[0121] After the main service process starts, the initialization program reads the predefined component mapping table. This mapping table maintains the correspondence between the names of four types of core components and their module paths in the form of a dictionary (e.g., {"tts":{"edgetts":"tts.edge"}}).

[0122] Step S202: Dynamic Module Import and Automatic Registration

[0123] The system iterates through the mapping table above, and for each entry, it calls the dynamic module import mechanism (such as importlib.import_module() in Python) to load the corresponding plugin module file. When a module is imported, its component class with the @register(category, name) decorator is executed during the class definition phase. This decorator writes the class object as key-value pairs into the corresponding sub-registry dictionary under the global plugin registry (e.g., _REGISTRY["tts"]["edgetts"]=EdgeTTS), thus completing the automatic registration of the component. This mechanism supports dynamically injecting new plugin implementations at runtime; simply importing a new module triggers the registration logic.

[0124] Step S203: Warm up the default model and start the service

[0125] After registration, the system locates and instantiates the target component from the registry based on the startup parameters. Then, the system performs a warm-up operation on the default digital human model: preprocessed data (including video frame sequences, face detection coordinates, etc.) is injected into the digital human cache manager to avoid cold start delays when the first user connects. Finally, the unified API gateway is started, listening on a specified port (e.g., 8010) and waiting for external connection requests.

[0126] Phase 3: Interaction Phase (Real-time Audio and Video Processing Pipeline). This phase describes the complete data stream processing process from user access to seeing the real-time video.

[0127] Step S301: Signaling Interaction and Session Creation

[0128] Users access the front-end page through a browser and initiate a WebRTC connection request. The unified API gateway takes over the SDP (Session Description Protocol) signaling interaction and ICE (Interactive Connection Creation) connection establishment process. After the signaling handshake is completed, the session manager assigns a unique UUID to the connection and creates an independent digital human interaction session instance.

[0129] Step S302: Multimodal input processing

[0130] Users enter text or upload audio files on the interface. The system branches its processing based on the request type: if it is a non-interruptible request (such as ordinary chat), the system may call the LLM (Large Language Model) to generate a reply text; if it is an echo test, the system will directly return the user's audio file.

[0131] Step S303: TTS audio synthesis

[0132] The generated text or raw audio enters the TTS engine instance. The TTS engine receives the text message queue, processes it through the internal process_tts thread, and outputs standardized audio frames (16kHz sampling rate, 20ms frame length, 320 samples / frame, float32PCM format).

[0133] Step S304: Audio Feature Extraction

[0134] The audio feature extractor receives audio frames output by TTS, converts them into high-dimensional feature vectors (such as 1024-dimensional features of the Whisper model or 80-dimensional features of the Mel spectrum), and pushes the feature vectors into the feature queue for consumption by downstream inference threads.

[0135] Step S305: Lip-sync reasoning

[0136] The lip-sync inference engine consumes feature vectors in batches from the feature queue. Combining the timestamp information of the audio frames, the inference engine drives the selected digital human model (such as Wav2Lip) to perform inference calculations and generate a lip-sync grayscale image (inference frame) that is highly synchronized with the target speech.

[0137] Step S306: Frame Composition and Network Streaming

[0138] The frame synthesis output pipeline retrieves the original background frame and face coordinates from the preprocessed data. The inference frame is scaled to the size of the face region and edge-blending is performed to eliminate stitching artifacts. Finally, the processed lip region is overlaid onto the original background frame to generate a complete synthesized video frame (fixed output 25fps). This video frame and its corresponding audio stream are encapsulated and pushed to the user's browser in real time via the WebRTC protocol for display.

[0139] Phase Four: Interruption Phase (State Switching and Traffic Draining). This phase uses software mechanisms to achieve millisecond-level conversation interruption, improving the naturalness of human-computer interaction.

[0140] Step S401: Receive interrupt request

[0141] If the user initiates new input (interruption request) while the user is speaking or the digital human is broadcasting, the front end will send an interrupt signal to the server.

[0142] Step S402: Pause the TTS engine and clear the queue

[0143] Upon receiving the request, the server executes the interruption logic: First, it clears the message queue in the TTS engine instance to prevent subsequent text from entering the synthesis process; second, it forcibly switches the TTS internal state machine from RUNNING to PAUSE. When the TTS thread detects this state in the next synthesis loop, it immediately stops outputting new audio frames to the pipeline.

[0144] Step S403: ASR queue clearing and silent filling

[0145] Simultaneously, the system clears the input audio queue associated with the Audio Feature Extractor (ASR). After clearing, the ASR's get_audio_frame() interface will continuously return preset silence frames. When the inference thread consumes a fully silent frame, the logic determines that the current state is silent, and no longer generates dynamic lip-sync inference frames, but instead continuously outputs static background frames.

[0146] Step S404: Natural purging and restoration of pipeline condition

[0147] Because TTS stops synthesizing new audio and ASR continues to return silence, the small number of audio frames already in transit in the pipeline will be naturally emptied within a very short time (approximately equal to the inference batch size × 40ms, or about several hundred milliseconds under default configuration) after inference and output. At this time, the digital human's screen remains in a silent state. When a new user request enters the TTS message queue, the TTS state machine automatically returns to RUNNING, and the system resumes normal audio synthesis and video streaming, achieving seamless interruption and continuation of the previous round of dialogue.

[0148] Implementation of key technical parameters

[0149] Registry Architecture: The global plugin registry is divided into four categories: TTS engine, digital human model, output transmission protocol, and audio feature extraction. Theoretically, there is no limit to the number of registrations in a single category, but it is recommended not to exceed 50 in practice to prevent memory overflow.

[0150] TTS engine concurrency: It uses a single instance and a single thread (process_tts) to process composition tasks, and multi-user concurrency is achieved by creating multiple independent TTS instances through the session manager to achieve resource isolation.

[0151] LRU cache eviction policy: The digital human cache manager sets the maximum cache size to 10. When the cache reaches the limit, it prioritizes retaining digital humans used by the "default role" and "currently active session", evictions the least used inactive digital humans, and calls the Python garbage collection mechanism to forcibly release video memory and system memory.

[0152] Output protocol support: The system supports multiple output protocols such as WebRTC (Web-RTC Low Latency Interactive), RTMP (RTMP Live Streaming), and VirtualCam (Virtual Camera), and dynamically matches the corresponding output module according to the client request.

[0153] The beneficial effects are as follows:

[0154] (1) Plug-in architecture achieves true component decoupling: Through the decorator mechanism of the global registry, the TTS engine, digital human model, and output protocol can be developed, registered, and replaced independently without modifying any business logic code. Compared with the prior art, the registry of this invention abstracts four component dimensions, covering the entire link of the digital human system, rather than just replacing a certain layer.

[0155] (2) One-click image deployment achieves self-check and self-start: Through Docker image packaging, CUDA dependencies, Python environment, all third-party libraries and model files are packaged at once during the image building stage; when the container starts, the environment self-check script automatically completes the initialization verification such as GPU driver detection, CUDA version compatibility adaptation and API key injection. After the self-check passes, the service process is automatically started, which can achieve "pull and start automatically" on any cloud platform. Compared with purely offline tools such as FaceFusion, this solution supports complete real-time interactive capabilities.

[0156] (3) High availability with multiple engines in parallel and with mutual backup: Multiple TTS engines and digital human models can be registered in the same system and switched at runtime through the startup parameters (--tts--model). When one engine fails, it can quickly switch to the backup engine to improve system availability.

[0157] (4) Millisecond-level interruption response: Through a dual-queue clearing mechanism (TTS message queue + ASR feature queue) + PAUSE / RUNNING state machine, new audio frame synthesis is prevented, and ASR outputs a silent frame. The residual frames already in transit in the pipeline are naturally cleared within approximately batch_size × 40ms (approximately 640ms under default configuration). After that, the digital human switches to a silent broadcasting state. Compared with the traditional pause / termination scheme, which requires waiting for the current complete sentence to finish playing before responding, the response time is shortened by more than an order of magnitude.

[0158] (5) Intelligent cache management enhances concurrency capabilities: The LRU eviction policy, combined with a deep cleanup mechanism, efficiently manages multiple digital human figures under limited video memory / memory conditions. The protection list mechanism ensures that the default role is not accidentally uninstalled due to a full cache.

[0159] (6) The output protocol can be flexibly configured: Through the BaseOutput abstract base class, the same set of business code can seamlessly switch between multiple push streaming methods such as WebRTC, RTMP, and VirtualCam to adapt to the needs of different deployment scenarios.

[0160] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A highly compatible pluggable digital human real-time interactive system, characterized in that, include: The image deployment and infrastructure layer provides a containerized runtime environment, which includes environment self-check scripts, the Docker container engine, and model data volumes. The Docker container engine builds and starts virtualized containers based on standard container images, which are read-only template files of the system's required runtime environment. The environment self-check script is used to automatically perform an environment self-check when the virtualization container starts; the model data volume is used to store digital human preprocessing data. The service and session management layer is used to receive and process external interaction requests and manage the lifecycle and resources of digital human interaction sessions. The global plugin registry includes a TTS engine registry for maintaining class mapping relationships of various speech synthesis engines, a digital human model registry for maintaining class mapping relationships of various digital human driving models, an output transmission protocol registry for maintaining class mapping relationships of various audio and video output protocols, an audio feature extraction registry for maintaining class mapping relationships of various audio feature extraction components, and a registry management unit. The registry management unit is used to register and discover components by combining a predefined component mapping table with module import to trigger registration. The digital human real-time interaction engine layer includes a TTS engine instance, an audio feature extractor, a lip-sync inference engine, and a frame synthesis output pipeline. The TTS engine instance receives text message queues and outputs audio frames. The audio feature extractor converts the input audio stream into feature vectors and pushes them into a feature queue. The lip-sync inference engine consumes the feature vectors in batches and, combined with the audio frames generated by the TTS engine, drives the digital human model to generate lip-sync inference frames. The frame synthesis output pipeline receives the inference frames, fuses them with background frames to generate the final video stream, and pushes the video stream to different user access terminals through the output transmission protocol module according to the type of client request. The user access terminal is used to receive and display the audio and video streams output by the system.

2. The highly compatible pluggable digital human real-time interactive system according to claim 1, characterized in that, The Docker container engine mounts the host machine's GPU device through container runtime parameters for use by the digital human inference engine within the container, and optimizes network communication performance by reusing parameters of the host machine's network protocol stack. The standard container image is built by integrating virtual environment management tools, language runtime dependency libraries, digital human real-time interactive program code, and preset model files on a container base image that supports GPU-accelerated algorithm inference. The environment self-check script executes environment self-checks that include at least: checking whether the graphics processor GPU driver is normal, checking whether the CUDA version of the computing unified device architecture is compatible with the deep learning framework within the container, verifying the integrity of the preset model files, and injecting the application programming interface (API) key.

3. The highly compatible pluggable digital human real-time interactive system according to claim 1, characterized in that, The service and session management layer includes: A unified API gateway is used to receive client requests and assign unique session identifiers; The session manager is used to create independent digital human interaction session instances and asynchronously dispatch interaction tasks to the real-time interaction engine layer according to the instantiation link in the global plugin registry. The Task Manager provides asynchronous task scheduling capabilities for the Session Manager, executing session-related time-consuming model loading operations via a thread pool. The Digital Human Cache Manager is used to cache pre-processed digital human data in memory or video memory based on the LRU policy. When the maximum cache limit is reached, the protected list and active list are excluded, the least used digital human is evicted, and resource release operations are performed.

4. The compatible plug-in digital human real-time interactive system according to claim 1, characterized in that, The registry management unit is used to register and discover components by combining a predefined component mapping table with module import-triggered registration. Specifically, this includes: predefining a mapping table from the names of four types of components to module paths in the initialization program at the system entry point; traversing the mapping table when the system starts and calling the dynamic module import mechanism to load the corresponding module files; when a module is imported, the class with the registration decorator is executed during the class definition phase, and the class object is written into the corresponding registry dictionary in key-value pairs to complete the registration; it supports dynamic injection of new plugins at runtime, loading new component modules through the dynamic module import mechanism, or replacing components by directly calling the class constructor.

5. The compatible plug-in digital human real-time interactive system according to claim 2, characterized in that, The digital human real-time interaction engine layer adopts a three-thread concurrent architecture, including: The main rendering thread unit is used to drive the ASR stepping and implement dynamic flow control through the frame queue backlog. Specifically, in each loop, the ASR is first driven to process one frame of audio data, and then the backlog of the output layer buffer corresponding to the output transmission protocol is immediately queried. When the backlog reaches or exceeds a preset threshold, the main thread is dynamically put to sleep according to a preset formula based on the backlog. The larger the backlog, the longer the sleep time, thus naturally reducing the call frequency of the ASR stepping. When the backlog falls back below the threshold, the normal operating rate is restored. The inference thread unit is used to consume feature vectors in batches and call the lip-sync inference engine to perform lip-sync inference. The frame synthesis output thread unit is used to synthesize the inference results into the original background frame and push audio and video data through a unified output module.

6. A highly compatible plug-in digital human real-time interaction method, employing the compatible plug-in digital human real-time interaction system as described in any one of claims 1-5, characterized in that, Includes the following steps: Image deployment steps: Build a standard container image based on a container base image that supports GPU-accelerated algorithm inference. During the image building phase, package the real-time interactive program code for the digital human, all dependent libraries, virtual environment, and preset model files all at once. After the standard container image is pushed to the image repository, the virtualization container is pulled and started on the target running platform, and the GPU device is mounted through the container running parameters. When the container starts, the built-in environment self-check script is automatically triggered to perform multi-stage environment self-checks. After the self-checks pass, the digital human main process is automatically started, and the service behavior is configured through the command line starting parameters. Component registration steps: When the main service starts, it traverses the system's predefined mapping table of component names to module paths, and imports each plugin module in sequence through the dynamic module import mechanism. During the import process, the automatic registration logic of the registration marker is triggered, and the classes of various interactive components are registered in key-value pairs to the TTS engine registry, digital human model registry, output transmission protocol registry and audio feature extraction registry under the global plugin registry. The default digital human model is warmed up by injecting preprocessed data into the digital human cache manager, then the service is started and connected to the unified API gateway, listening on the specified port; Interactive session steps: The unified API gateway receives connection requests initiated by clients, completes signaling interaction and connection establishment; the session manager assigns a unique session identifier to each connection and creates an independent digital human interactive session instance; it receives text or audio input from users, determines whether to call the large language model to generate reply text based on the request type, and then sequentially synthesizes audio through the TTS engine instance, extracts feature vectors through the audio feature extractor, performs lip-sync inference through the lip-sync inference engine, and synthesizes the inference frame with the background frame through the frame synthesis output pipeline. Finally, the audio and video streams are pushed to the client in real time through the output transmission protocol.

7. The highly compatible pluggable digital human real-time interaction method according to claim 6, characterized in that, This also includes interrupting the recovery process: When an interruption request is received from the client, the message queue of the TTS engine instance is cleared to prevent subsequent text from entering the synthesis process, and the TTS engine state is switched from running state to paused state, so that TTS stops outputting new audio frames to the pipeline. At the same time, the input audio queue of the audio feature extractor is cleared, so that the audio feature extractor continuously outputs silent frames. When the inference thread consumes a silent frame, it outputs a background frame. After the audio frames that are already in transit in the pipeline are naturally cleared, the digital human remains in the silent broadcasting state. When a new interaction request enters the TTS message queue, the state is automatically restored to the running state, triggering the normal synthesis and output of the new audio frame.

8. The highly compatible pluggable digital human real-time interaction method according to claim 7, characterized in that, In the interactive session step, the digital human inference pipeline adopts a three-thread concurrent architecture, including: The main rendering thread is used to drive the ASR (Automatic Speech Recognition) step and to control the flow of upstream input by limiting the frame queue by a preset threshold. The inference thread is used to consume the feature vectors in batches and call the lip-sync inference engine to perform lip-sync inference and generate inference frames. The frame synthesis output thread is used to synthesize the inference frame into a preset background frame to obtain the target audio and video data, and transmit it to the output layer corresponding to the target output transmission protocol. The rendering main thread, inference thread, and frame synthesis output thread collaborate in a pipeline through a frame queue of preset capacity.

9. The highly compatible pluggable digital human real-time interaction method according to claim 8, characterized in that, The flow control of the main rendering thread specifically includes: in each loop, the main rendering thread first drives ASR to process one frame of audio data, and then immediately queries the buffer backlog of the output layer corresponding to the output transmission protocol; when the backlog reaches or exceeds a preset threshold, the main thread is dynamically put to sleep according to a preset sleep formula based on the backlog. The larger the backlog, the longer the sleep time, thereby naturally reducing the call frequency of ASR stepping; when the backlog falls below the threshold, the main thread resumes normal operation speed.

10. The highly compatible pluggable digital human real-time interaction method according to claim 1, characterized in that, In the interruption recovery step, the time for the audio frames already in transit in the pipeline to be naturally cleared is the inference batch size multiplied by the duration of each frame, where the duration of each frame is 40ms.