Single-connection streaming-based virtual character multi-modal interaction method and system
By optimizing the virtual character interaction system through single-connection streaming and multi-factor weighted routing strategies, the problems of high latency and resource waste were solved, and low-latency, low-consumption multimodal interaction was achieved, thereby improving user experience and system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU SHIYUXINGJING INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-29
AI Technical Summary
Existing virtual character interaction systems suffer from high latency, resource waste, and improper asynchronous task scheduling, resulting in poor immersion, network jitter, and excessive server resource consumption.
It adopts a single-connection streaming architecture, transmits multimodal data through the SSE channel, and combines multi-factor weighted routing strategy and on-demand connection management to achieve synchronous playback of text and audio, and optimizes asynchronous task scheduling.
It enables low-latency multimodal interaction, reduces network and resource consumption, improves system response speed and resource utilization, and ensures content security and performance.
Smart Images
Figure CN122116875A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, specifically to a real-time interactive system for virtual characters based on Large Language Model (LLM) and streaming technology, which enables text generation, text-to-speech (TTS), and multi-agent collaboration. Background Technology
[0002] With the widespread application of Large Language Models (LLM), application scenarios such as virtual digital humans and AI companions place extremely high demands on the real-time performance of interactions. Current virtual character interaction systems typically adopt the following architecture: Serial processing architecture: User input -> ASR recognition -> LLM inference (waiting for complete text) -> TTS synthesis (waiting for complete audio) -> Playback.
[0003] Multi-connection parallel architecture: The client requests text via HTTP and audio via WebSocket connection.
[0004] High First Frame Response Time (TTFT): The traditional REST API model requires users to wait for the LLM to generate all tokens before triggering TTS, resulting in a "silent period" of 5-15 seconds, which seriously undermines the immersive experience.
[0005] Multimodal data asynchrony (Lip-sync Latency): Text streams and audio streams are transmitted through different network protocols (such as HTTP + WebSocket). Affected by network jitter, the client often experiences the phenomenon of "the text has been output but the sound has not arrived" or "audio and video misalignment".
[0006] Waste of connection resources: Traditional TTS services typically pre-establish a fixed number of WebSocket connection pools (long connections), which consume a large number of server handles and memory resources during the user idle time; while during peak concurrency periods, the fixed connection pool is easily exhausted, leading to queuing and blocking.
[0007] Asynchronous task scheduling is not only problematic in scenarios involving AI drawing / video generation, where the traditional "batch submission + polling" mode is prone to triggering rate limiting and circuit breaking (HTTP 503) of upstream services, and cannot dynamically adjust the distribution rate according to the upstream load. Summary of the Invention
[0008] The present invention aims to solve the above problems and provide a method and system for multimodal interaction of virtual characters based on single-connection streaming transmission.
[0009] This invention constructs a method and system for multimodal interaction of virtual characters based on single-connection streaming transmission.
[0010] In a single SSE channel, define multiple custom event types (event: text, event: audio, event: control).
[0011] The server listens to the token generation stream of the LLM; Set thresholds for "punctuation detection" and "minimum semantic length". When the accumulated tokens meet the requirements of a complete syntactic unit, the TTS engine is immediately triggered to perform segmented synthesis. The text tokens generated by LLM (as data: text) and the audio segments generated asynchronously by TTS (as Base64 encoded data: audio) are pushed alternately in the same TCP connection in the order of generation.
[0012] The client receives events through a single EventSource listener, which routes them to the text rendering queue and the audio playback buffer respectively, based on the event type, to achieve "word-sound synchronization".
[0013] Multi-dimensional weighted intelligent routing strategy To address the issue that a single model cannot simultaneously achieve compliance and performance, a routing decision model based on multi-factor weights is designed: Routing factor: $R = \alpha \cdot P_{score} + \beta \cdot H_{char} + \gamma \cdot L_{load}$ $P_{score}$ (Compliance Sensitivity): Detects the political / violent sensitivity of user input through a lightweight front-end model (such as Qwen-Turbo); if $P_{score} > Threshold$, forces routing to a higher compliance model (such as Doubao).
[0014] $H_{char}$ (Role Consistency): Hash splitting based on $CharacterID \mod N$ ensures session state cache hit rate for the same role.
[0015] $L_{load}$ (real-time load): dynamically degrades based on the current queuing depth of each model service.
[0016] "On-demand" management of TTS connection pools On-Demand: The initial number of connections is 0 or a very small value, and a new connection is created only when a request arrives and there are no idle connections.
[0017] Async Generator: Utilizes the generator features of Python / Node.js to convert the WebSocket binary stream of TTS into an SSE text stream in real time, avoiding memory buffering.
[0018] Idle GC: Mark the last_used_timestamp for each connection object. The daemon thread scans every cycle. If CurrentTime - last_used > IdleTimeout, it actively disconnects the connection and releases the handle.
[0019] For asynchronous generation tasks (such as AI-generated images), the polling mechanism is abandoned, and a "callback-driven state machine" is adopted: State machine design: Pending -> Processing -> Completed / Failed -> Rollback (if 503). Chain triggering: When a callback is received from an upstream service, the pool_status (number of idle nodes) is parsed.
[0020] Logical decision: A new task submission is popped from the database Pending queue only when pool_status.idle > 0. If an HTTP 503 response is received, the task status is automatically rolled back to Pending and an exponential backoff strategy is executed to prevent queue overflow.
[0021] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: Extremely low interaction latency: Through SSE single-connection transmission and sentence pre-synthesis, the first word response time (TTFT) is reduced to less than 500ms, and the first sentence speech playback latency is reduced to less than 2 seconds (85% lower than existing technologies).
[0022] Reduced network overhead: The client only needs to maintain one HTTP long connection to achieve multimodal interaction. Compared with the "HTTP polling + WebSocket" solution, the power consumption and data consumption of mobile devices are reduced by about 30%.
[0023] Maximizing resource utilization: The "breathing" mechanism of the TTS connection pool reduces server memory usage by 80% during off-peak periods while ensuring high availability during peak periods.
[0024] Content security and performance balance: The multi-model routing strategy ensures 100% political compliance (for specific sensitive words) while reducing API call costs by about 40% through traffic splitting. Attached Figure Description
[0025] Figure 1: System overall architecture diagram (showing the layered structure of access layer, routing layer, agent layer and output layer).
[0026] Figure 2: Timing diagram of text / audio time-interleaved transmission based on SSE (core diagram, showing the parallel relationship between text event and audio event on the time axis).
[0027] Figure 3: Multi-model intelligent routing decision-making flowchart (showing the judgment logic of Hash calculation and sensitive word detection).
[0028] Figure 4: TTS connection pool lifecycle state transition diagram (showing Create, Busy, Idle, Destroy states). Detailed Implementation
[0029] Example 1: Multimodal Streaming Interaction Process Connection established: The client initiates a GET request to / api / chat, with the request header including Accept: text / event-stream. The server returns 200 OK, keeping the TCP connection open.
[0030] 2. Parallel processing: The server receives text input and calls IntentionAgent (intent recognition) and RetrievalModule (vector library memory retrieval) in parallel.
[0031] Assemble the Prompt and request the LLM streaming interface.
[0032] 3. Streaming distribution loop: Text event: For each token generated by LLM, the server immediately encapsulates it as id: N, event: text, data: "word" and pushes it.
[0033] Audio trigger: A period is detected in the server buffer. The current sentence is then extracted and sent to the TTS engine (reusing an idle WebSocket connection).
[0034] Audio event: The PCM audio stream returned by TTS is transcoded to Base64 in real time, encapsulated as id: M, event: audio, data: "Base64..." and pushed.
[0035] End: LLM generation complete and TTS sent, push event: complete.
[0036] Example 2: Sensitive Content Routing Logic 1. User input: "Do you know about [sensitive keywords]?" 2. Pre-detection: SafetyAgent identifies sensitive word hits or classifies intent as "political".
[0037] 3. Route calculation: Sensitivity_Flag = True Decision tree logic: IF Sensitivity_Flag THEN Route = "Doubao-Pro" (HighCompliance Model).
[0038] 4. Execution: The system ignores the Gemini model originally bound to the role, forcibly calls the Doubao model to generate a compliant response, and marks Audit_Log: Rerouted in the log.
[0039] Example 3: Task Scheduling Rollback Mechanism 1. The system attempted to submit AI drawing task ID: 101.
[0040] 2. The upstream service returned an HTTP 503 Service Unavailable error (due to GPU overload).
[0041] 3. Exception handling: The scheduler caught a 503 error.
[0042] 4. Status rollback: The status of task ID:101 in the database is changed from Processing to Pending.
[0043] 5. Passive Trigger: The system pauses submissions. Submissions for ID:101 are triggered again only when a callback indicating task completion is received from upstream (carrying the idle_server > 0 signal).
Claims
1. A method for multimodal interaction of virtual characters based on single-connection streaming, characterized in that, The method includes the following steps: Establish a single long-lived connection channel based on the server-side push event SSE protocol; Listen to the text token stream generated by the Large Language Model (LLM); When the accumulated text tokens meet the preset sentence segmentation threshold, the speech synthesis TTS engine is triggered to perform segmented audio synthesis. The text token generated by the LLM is encapsulated into a text event frame, and the audio data generated by the TTS engine is encapsulated into an audio event frame; The text event frames and audio event frames are interleaved and pushed to the client in the order of their generation through the single long connection channel.
2. The method according to claim 1, characterized in that, The sentence segmentation thresholds include punctuation detection and / or a minimum semantic length threshold.
3. The method according to claim 1, characterized in that, Before establishing a single long-lived connection channel based on the SSE protocol, the following is also included: Obtain user input requests and extract compliance features, role features, and system load features from the requests; The routing decision value is obtained by weighting the compliance characteristics, role characteristics, and system load characteristics. Based on the routing decision value, the user input request is directed to a specific LLM model in the preset model pool.
4. The method according to claim 1, characterized in that, The triggering of the TTS engine to perform segmented audio synthesis includes: Retrieve idle connection objects from the TTS connection pool; If no idle connection object exists, a new WebSocket connection will be created dynamically according to the current task requirements; The segmented text is sent to the TTS engine through the connection object, and the returned binary audio stream is received in real time.
5. The method according to claim 4, characterized in that, The TTS connection pool management includes: Record the last used timestamp for each connection object; The last used timestamp is periodically scanned. If the difference between the current time and the last used timestamp exceeds a preset idle threshold, the connection object is actively closed and destroyed.
6. The method according to claim 1, characterized in that, The encapsulation process of the audio event frame includes: Obtain the binary audio stream data returned by the TTS engine; Convert the binary audio stream data into Base64 encoded format; Fill the Base64 encoded data into the data field of the SSE protocol and mark the event field as audio.
7. The method according to claim 1, characterized in that, The method also includes task scheduling based on upstream service load feedback: Receive a callback signal returned by the upstream asynchronous task processing node, the callback signal carrying node idle status information; When the idle status information indicates that there are available nodes, a task is extracted from the task queue and submitted. If a rate-limiting error code is received from the upstream after submitting the task, the task status is rolled back to pending and an exponential backoff retry is performed.
8. A virtual character multimodal interaction system based on single-connection streaming transmission, characterized in that, include: The channel establishment module is used to establish a single long-lived connection channel based on the SSE protocol; The monitoring and distribution module is used to monitor the text token stream generated by the LLM and the audio stream generated by the TTS engine. The encapsulation and push module is used to encapsulate text data and audio data into heterogeneous event frames, and push them to the client via the single long connection channel.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 7.