Stream processing method, system for multi-modal interaction and application thereof

Abstract

The application discloses a kind of for multimodal interaction stream processing method, system and its application, the method includes the following steps: receiving input data from user client, input data at least contains audio data;Audio data is subjected to automatic speech recognition ASR processing, and the user text of current round is generated;Trigger word detection is carried out to user text, whether hit preset visual trigger condition is judged;If visual trigger condition is hit, then execute visual injection sub-process;If visual trigger condition is not hit, then current round is determined as pure text round;Based on user text, visual input determination result and the context information obtained from fixed rolling memory window, construct multimodal prompt context.The application can realize the comprehensive promotion of system in low delay, high consistency, high concurrent stability and response accuracy, significantly improve the core engineering technical index of fad product and humanoid robot in real-time multimodal interaction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a streaming processing method, system and application for multimodal interaction. Background Technology

[0002] With the development of artificial intelligence technology, multimodal interactive AI (artificial intelligence) terminals based on "voice + vision" are shifting from the traditional "question-and-answer dialogue" of voice assistants to an integrated approach of "low-latency real-time conversation + embodied perception + executable agent." The technological focus is no longer on the quality of single-transaction text generation, but rather on the temporal and control structures of the end-to-end interactive closed loop. International technological trends show an integration of streaming input and output of audio, images, and text through real-time APIs to build voice-first multimodal intelligent agents, emphasizing agent capabilities such as "computer usage," and incorporating screen understanding and operation into the model's executable space.

[0003] In China, especially in the "trendy toy" companion device sector, the current common approach is a three-stage link: Automatic Speech Recognition (ASR), Multimodal Large Models (MLLMs), and Text-to-Speech (TTS), with streaming modifications. Emphasis is placed on engineering metrics such as first packet / first audio latency, reconnection after disconnection, concurrent rate limiting, and content security to maintain a usable experience within cost constraints. However, current technology faces numerous systemic bottlenecks in actual productization: 1. Uncontrollable jitter in end-to-end interaction latency: Any delay in the tail of ASR, visual upload, LLM (Large Language Model) first token (character), or TTS first sound packet will be amplified into the "stuttering" perceived by the user. The lack of first packet / first sound packet segmentation indicators will make it impossible to make accurate attribution and targeted optimization. 2. Streaming speech output lacks a stable "playable unit" segmentation mechanism: If sentence-level / pause-level segmentation and aggregation and early synthesis prompts are not performed, TTS must wait for the long text to converge, resulting in a structural wait where "the text has been generated in streaming but the audio is delayed." 3. Insufficient back pressure management in concurrent rounds can trigger avalanche queuing: In scenarios with multiple users or a single user speaking rapidly and continuously, the lack of inflight limits, queuing strategies, and fail-fast semantics (a design principle that immediately stops and exposes problems when an error occurs, preventing the error from spreading further) can easily lead to request accumulation, timeout chains, and resource contention, ultimately resulting in round disorder or response loss. 4. The “latest state selection” of multimodal assets has a competitive consistency defect: When audio / images enter the system in the form of file writing to disk or asynchronous upload, if there is a lack of atomic writing, write completion judgment and secondary stability check, the backend will read the incomplete file or mistakenly treat the old asset as the new asset, thus causing round-level mismatch. 5. The interruption mechanism of full-duplex voice interaction is difficult to reliably implement at the product level: The system needs to be highly sensitive to external human voices for VAD / interruption, and also needs to suppress the echo triggering of its own speaker playback. Without echo cancellation, playback cooling window and gate state machine, two failure modes will occur: "self-interruption" or "excessive sluggishness". 6. Short window memory lacks discipline in its injection strategy: If fixed windows do not comply with rules such as "instruction priority, memory degradation, empty input suppression, and trigger round isolation", memory is prone to take over, repetitive output and context drift, causing the interaction to degenerate from "responding to the current intention" to "rereading the history summary". 7. Lack of verifiable protocol boundaries for cross-round dialogue states: In the WebSocket streaming event model (a network protocol that allows clients and servers to maintain a long connection and conduct bidirectional real-time communication), without clear round identifiers, stage events, and error semantics, disconnection and reconnection or network jitter can cause the client to be unable to determine which round the current audio / text belongs to, resulting in a hidden fault that is "seemingly successful but with misaligned content".

[0004] 8. Insufficient handling of failures and degradation links caused by dependence on commercial APIs: Upstream 4xx / 5xx (error status codes returned by application servers), rate limiting, model cold start and sharding timeout are common occurrences. If there is a lack of unified timeout budget, retry strategy, circuit breaker and safety fallback, the system will directly stop or output unbroadcastable content under boundary conditions. 9. Lack of fine-grained measurement and observability will hinder scalable iteration: Without latency breakdown by round, token / audio duration measurement, error classification and sampling logs, the team cannot establish a quantitative closed loop of cost-experience, and therefore cannot do quota, rate limiting and capacity planning when deploying in the cloud. 10. The lack of multi-tenant isolation and privacy compliance modules will limit commercialization: In trendy toy products targeting children or family scenarios, voice and images are highly sensitive data. If user directory isolation, minimum retention, key authentication and permission boundaries are lacking, even if the system is functional, it will be difficult to enter the sustainable operation stage.

[0005] Therefore, while ensuring low-latency response, existing technologies struggle to balance overall system consistency and reliability.

[0006] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to solve the technical problem that existing multimodal streaming interaction methods are difficult to balance the overall consistency and reliability of the system while ensuring low latency response. The invention proposes a streaming processing method, system and application for multimodal interaction.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A streaming processing method for multimodal interaction includes the following steps: S1, receiving input data from a user client, wherein the input data includes at least audio data; S2, performing Automatic Speech Recognition (ASR) processing on the audio data to generate user text for the current round; S3, performing trigger word detection on the user text to determine whether a preset visual trigger condition is met; if the visual trigger condition is met, then executing a visual injection sub-process: retrieving the latest timestamped image file from the storage directory associated with the user identifier, performing a stability check on the image file, and determining it as the visual input for the current round after the check passes; the stability check includes: sampling the file attributes and readable status of the candidate image file at least twice within a short time interval, and determining it as stable only when the consecutive sampling results are consistent and the file generation time exceeds a safe age threshold; if the visual trigger condition is not met, then the current round... S4. Based on the user text, visual input determination results, and context information obtained from the fixed scrolling memory window, construct a multimodal prompt context; wherein, if the current round is a visually triggered round, suppress or reduce the injection intensity of historical information in the fixed scrolling memory window; S5. Input the multimodal prompt context into the multimodal large language model MLLM and receive the streaming text increment returned by it; perform real-time aggregation and sentence boundary analysis on the streaming text increment to determine and extract playable sentence segments; the playable sentence segments must simultaneously meet the following conditions: contain a sentence segment end signal, have no unclosed grammatical structure at the end, have a length exceeding the minimum perception threshold, and form a complete reading unit semantically; S6. Send the playable sentence segments into the text-to-speech (TTS) synthesis module to generate streaming audio data; and return the streaming text increment and / or streaming audio data to the user client.

[0009] In some embodiments, in step S4, the fixed scrolling memory window is used to store user input and system response pairs from the most recent multiple rounds of dialogue in chronological order; when a new round is written, if the window is full, the earliest round of dialogue is removed before writing.

[0010] In some embodiments, the method further includes a concurrency control step: for a received user request, first attempt to obtain session-level processing eligibility corresponding to the user session identifier; if successful, request a system-level active processing quota; only when the multimodal large language model MLLM needs to be called, request a model call-level concurrency quota; if obtaining eligibility or quota fails or times out at any stage, immediately terminate the current request processing and return a failure response.

[0011] The present invention also provides an electronic device, comprising: a processor; and a memory for storing a computer program executed by the processor; wherein, when the computer program is executed by the processor, it implements the method described in any of the above embodiments.

[0012] This invention also provides a multimodal interaction system, comprising: an input receiving module configured to receive input data from a user client; a speech recognition module configured to recognize audio data and generate user text; a trigger routing module configured to parse the user text and determine whether visual injection is triggered; a visual processing module configured to retrieve and verify an image file from storage when visual injection is triggered; a memory management module configured to maintain and provide a fixed capacity of historical dialogue context; a context construction module configured to fuse user text, visual input information, and historical context to generate multimodal prompts; an inference generation module configured to receive multimodal prompts and return streaming text increments; an output processing module configured to aggregate text increments, determine playable segments, and perform purification; a speech synthesis module configured to synthesize playable segments into streaming audio data; and a response return module configured to return the text and audio streams to the client.

[0013] In some embodiments, the visual processing module is further configured to: filter candidate image files in reverse chronological order; sample the file attributes of the target candidate file at least twice within a short time interval, the file attributes including file size and modification time; compare the two sampling results, and if they are consistent and the file generation time exceeds a safe age threshold, determine that the image file is stable and usable.

[0014] In some embodiments, the memory management module uses a fixed-length first-in-first-out queue to store dialogue rounds, and actively suppresses the injection intensity of historical context or retains only the most recent round context during visually triggered rounds.

[0015] In some embodiments, the system further includes a concurrency control module configured to manage session-level mutexes, system-level active request semaphores, and model call-level concurrent semaphores, and to trigger a fast-fail mechanism when acquisition fails.

[0016] The present invention also provides a humanoid robot comprising the multimodal interaction system described in any of the above claims. The system is configured as an upper-level cognitive interaction module of the humanoid robot, used to process audio and visual inputs collected through the microphone and camera mounted on the robot, and to generate interactive voice and control commands.

[0017] The present invention also provides an AI interactive toy, comprising the multimodal interaction system described in any of the above claims. The system is configured as the core interactive control module of the AI ​​interactive toy, used to process audio and visual inputs collected by the microphone and camera mounted on the toy body, and to generate interactive voice and feedback commands with companionship attributes.

[0018] The beneficial effects of this invention compared to the prior art include: The streaming processing method and system for multimodal interaction provided by this invention effectively avoids problems such as reading incomplete or incorrect versions of files due to asynchronous writing, network transmission delays, or cache refresh by introducing a stability verification mechanism for image files. This ensures the reliability of injected visual content and the accuracy of round matching, thereby improving the quality of multimodal fusion inference and the overall consistency of the system. By performing real-time sentence boundary analysis on the incremental streaming text based on multiple dimensions such as sentence end signals, grammatical structure integrity, minimum length, and semantic integrity, it can accurately determine and release playable sentences in advance. This allows speech synthesis to begin without waiting for the entire response text to be generated, significantly reducing the user-perceived initial audio delay from depending on the full text generation time to depending only on the generation time of the first complete sentence segment, thus significantly improving the real-time performance and fluency of the interaction. By employing a fixed-capacity scrolling window to manage historical dialogue context, and implementing a dynamic injection strategy based on trigger conditions and input validity, the system maintains contextual consistency throughout the dialogue while effectively preventing outdated or lengthy historical information from overwhelming or interfering with the current user's core commands. In particular, accurate understanding of visual content ensures that the system response always prioritizes the current user's intent. These various technical methods work synergistically to achieve a comprehensive improvement in the system's low latency, high consistency, high concurrency stability, and response accuracy, significantly enhancing the core engineering and technical indicators of trendy toys and humanoid robots in real-time multimodal interaction. This invention achieves a comprehensive performance improvement in the real-time performance, reliability, continuity, and stability of multimodal streaming interaction systems, thereby significantly improving the user experience.

[0019] Furthermore, by constructing a layered concurrency control and backpressure management mechanism at the session, system, and model call levels, and immediately executing fast failure when any stage fails, infinite queuing of requests and resource deadlock are avoided. This effectively suppresses tail latency amplification and system avalanche in high-concurrency scenarios, ensuring the stability and availability of the system under high load.

[0020] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0021] Figure 1 This is a flowchart of the upstream client link processing in an embodiment of the present invention; Figure 2 This is a flowchart of the downstream backend and server processing in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the fixed scrolling memory window mechanism in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0023] This invention relates to the field of software technology for "trendy" companion terminals and extendable to humanoid robot interaction systems, specifically belonging to the technology of multimodal dialogue agents and real-time human-computer interaction orchestration based on large-scale pre-trained models. Its core object is an end-to-end interaction link with voice as the primary channel and vision as an on-demand channel. It structures user semantics through voice acquisition and automatic speech recognition, then performs multimodal fusion reasoning with the latest image from the camera in the same round, and finally generates a stream of voice output that can be played instantly through text-to-speech synthesis, thereby achieving companion-style interaction with context awareness and continuous response capabilities. To meet the constraints of low latency, strong consistency, and controllable cost for trendy toy products, the embodiments of this invention further focus on key aspects such as multimodal round orchestration, streaming output control, concurrent backpressure management, and fixed window memory management. This enables the system to form a deployable, scalable, and portable interactive kernel on top of the large-scale model capabilities of commercial APIs (an interface that allows different software systems to communicate and call functions according to agreed rules). It also supports future reuse on humanoid robot platforms with visual perception and actuators as a higher-level cognitive interaction module for dialogue and visual collaboration.

[0024] This invention provides a streaming processing method for multimodal interaction, comprising the following steps: S1. Receive input data from the user client, the input data including at least audio data.

[0025] S2. Perform Automatic Speech Recognition (ASR) processing on the audio data to generate the user text for the current round.

[0026] S3. Trigger word detection is performed on the user text to determine if it meets the preset visual trigger conditions. If the visual trigger conditions are met, the visual injection sub-process is executed: the latest timestamped image file is retrieved from the storage directory associated with the user identifier, stability verification is performed on the image file, and if the verification passes, it is determined as the visual input for the current round (it is stored in a subfolder automatically created by the backend based on the device IP address, and the model automatically extracts it as the visual input for the current round). The stability verification includes: sampling the file attributes and readability of the candidate image file at least twice within a short time interval, and determining it as stable only when the continuous sampling results are consistent and the file generation time exceeds the safe age threshold; if the visual trigger conditions are not met, the current round is determined as a plain text round. Specifically, dedicated visual trigger words are designed in the process to detect user text and determine if it meets the preset visual trigger conditions. If the visual trigger conditions are met, the visual injection sub-process is executed, the front-end camera takes a picture and automatically stores it in the ID subfolder created by the backend for different devices. The process extracts the latest audio and image data from the backend folder and uploads them to the model.

[0027] S4. Based on the user text, visual input judgment results, and context information obtained from the fixed scroll memory window, construct a multimodal prompt context; wherein, if the current round is a visually triggered round, suppress or reduce the injection intensity of historical information in the fixed scroll memory window.

[0028] S5. Input the multimodal prompt context into the multimodal large language model MLLM and receive the streaming text increment returned by it; perform real-time aggregation and sentence boundary analysis on the streaming text increment to determine and extract the reportable sentence segments; the reportable sentence segments must simultaneously meet the following conditions: contain the sentence segment end signal, have no unclosed grammatical structure at the end, have a length exceeding the minimum perceptual threshold, and form a complete reading unit semantically.

[0029] S6. Send the playable sentence segment into the text-to-speech (TTS) synthesis module to generate streaming audio data; and return the streaming text increment and / or streaming audio data to the user client.

[0030] In step S3, the visual injection sub-process also includes: if the latest candidate image file fails the stability check, it will automatically revert to the previous image file that passed the stability check, or abandon the visual injection of the current round.

[0031] In step S4, a fixed scrolling memory window is used to store user input and system response pairs from the most recent multiple rounds of dialogue in chronological order. When a new round is written, if the window is full, the earliest round of dialogue is removed before writing. It also includes: if the user input for the current round is detected to be empty or below the valid instruction length threshold, injecting historical information from the fixed scrolling memory window is skipped.

[0032] The streaming processing method for multimodal interaction also includes a concurrency control step: for a received user request, first attempt to obtain session-level processing eligibility corresponding to the user session identifier; if successful, request a system-level active processing quota; only when it is necessary to call the multimodal large language model MLLM, request a model call-level concurrency quota; if obtaining eligibility or quota fails or times out at any stage, immediately terminate the current request processing and return a failure response.

[0033] In step S6, before sending the playable sentence segments into the text-to-speech (TTS) synthesis module, the playable sentence segments after incremental or aggregated streaming text are also subjected to output purification processing. The output purification processing includes: stripping metadata, cleaning abnormal control characters, removing prompt word fragments, filtering debugging information, and repairing incomplete sentences.

[0034] This invention also provides an electronic device, including: a processor; and a memory for storing a computer program executed by the processor; wherein, when the computer program is executed by the processor, it implements the method described in any of the above embodiments.

[0035] This invention also provides a multimodal interaction system, comprising: an input receiving module configured to receive input data from a user client; a speech recognition module configured to recognize audio data and generate user text; a trigger routing module configured to parse the user text and determine whether visual injection is triggered; a visual processing module configured to retrieve and verify an image file from storage when visual injection is triggered; a memory management module configured to maintain and provide a fixed capacity of historical dialogue context; a context construction module configured to fuse user text, visual input information, and historical context to generate multimodal prompts; an inference generation module configured to receive multimodal prompts and return streaming text increments; an output processing module configured to aggregate text increments, determine playable segments, and perform purification; a speech synthesis module configured to synthesize playable segments into streaming audio data; and a response return module configured to return the text and audio streams to the client.

[0036] The visual processing module is further configured to: filter candidate image files in reverse chronological order; sample the file attributes of the target candidate file at least twice within a short time interval, including file size and modification time; compare the two sampling results, and if they are consistent and the file generation time exceeds the safe age threshold, then the image file is determined to be stable and usable.

[0037] The memory management module uses a fixed-length first-in-first-out queue to store dialogue rounds, and actively suppresses the injection intensity of historical context or retains only the most recent round's context during visually triggered rounds.

[0038] The multimodal interaction system also includes a concurrency control module, configured to manage session-level mutexes, system-level active request semaphores, and model call-level concurrent semaphores, and trigger a fast-fail mechanism when acquisition fails.

[0039] This invention also provides a humanoid robot, comprising the multimodal interaction system described in any of the above embodiments. The system is configured as an upper-level cognitive interaction module of the humanoid robot, used to process audio and visual inputs collected through the microphone and camera mounted on the robot, and to generate interactive voice and control commands.

[0040] The present invention also provides an AI interactive toy, comprising the multimodal interaction system described in any of the above claims. The system is configured as the core interactive control module of the AI ​​interactive toy, used to process audio and visual inputs collected by the microphone and camera mounted on the toy body, and to generate interactive voice and feedback commands with companionship attributes.

[0041] The following describes specific embodiments of the present invention.

[0042] This embodiment proposes a streaming processing method and a multimodal interaction system (also referred to as "Maruko" in this embodiment) for multimodal interaction. Its key mechanisms for achieving "stability, low latency, and continuous dialogue" in trendy toy interaction are as follows: Figures 1 to 3 As shown, this involves a three-module collaboration: voice closed-loop as the main axis, triggered vision as an extension, and fixed window memory as continuous support. The upstream client is responsible for converting voice and optional visual acquisition from the real environment into controllable input and prioritizing the initial experience. The downstream backend and server use turn-based orchestration and streaming pipelines to solidify ASR-LLM-TTS into an observable output process. The memory module maintains dialogue coherence without increasing the risk of context loss of control through time-series windows and gating constraints, thus making high-frequency interactions in trendy toy scenarios usable and scalable.

[0043] like Figure 1As shown, the upstream client link (input receiving module) has a default closed loop of "voice input - voice output", and uses the trigger word "Wanzi Take a Look" to activate the conditional branch of "voice - vision - voice". The key lies in overlapping the parallel waiting items for execution to compress the delay of the first token (character) and the first audio packet. The link first performs VAD and segment preprocessing on the microphone audio, and then solidifies the current round of voice into a stable and readable input snapshot through asset stabilization to avoid race reading and writing. Subsequently, two tasks are started in parallel during the round trigger and routing phase: one is to call Qwen3 ASR Flash (real-time speech recognition model, i.e., the speech recognition module) to quickly transcribe and form consumable text and prioritize trigger detection, and the other is to extract memory snapshots from a fixed window for context assembly. If the trigger is established, the latest image is stably selected and compressed / scaled and then jointly forms a multimodal input with the text and is handed over to Doubao 1.6 Flash (large language model) for streaming generation; if the trigger is not established, the text and memory snapshots directly enter the same streaming generation path. The generated text increment is aggregated by sentence boundaries to form a segment that can be synthesized in advance and sent to Qwen3 TTS Flash (text-to-speech model, i.e., the speech synthesis module) for streaming synthesis. WebSocket (abbreviated as WS, a network protocol that allows the client and server to maintain a long connection and perform two-way real-time communication, i.e., the response return module) simultaneously returns the text delta (increment) and audio chunk (block), thus achieving real-time reporting while generating. The concurrent guardrail suppresses round accumulation and tail delay amplification through per-user single flight and global inflight gating, ensuring the stability of high-frequency short-round interactions of trendy toys.

[0044] After the input-side convergence of the upstream "voice closed loop / triggered vision injection" is completed, Figure 2The mechanism described above accesses the server side with the user turn through a WebSocket (a network protocol that enables long - term connections and two - way real - time communication between the client and the server) two - way event stream, and advances the interaction from "collection and triggering" to "session orchestration and stable feedback". The server module hosts the long - term connection with a FastAPI (a framework for developing Web APIs) WebSocket endpoint. First, it manages the connection and session, generating stable session_id (session identifier) and request_id (request identifier). It performs turn - level parsing and verification (such as text length, audio / image size, base64 (a coding method) legality, etc.) on the inbound JSON (structured information) payload. Subsequently, it synchronizes the turns of the same user through a per - user gate to avoid concurrent writing and context crosstalk. When the input is empty or the user declares auto_latest (a control flag indicating whether the system automatically selects the "latest generated and qualified assets" to participate in the current turn processing), the server selects the latest audio / image from the diagnostic input directory for the user_id (user identifier) and performs trigger - phrase constraint. Only when "丸子看看" is detected, is image auto - injection allowed, thus hard - isolating the "voice turn" and "visual turn" on the service side first.

[0045] When the turn enters the backend module, the core entry drives an end - to - end streaming pipeline with stream_turn (a processing process or state that receives, generates, and distributes in a streaming manner in units of "single conversation turns"): it parallelly obtains voice semantics and context elements, including secondary injection and budget trimming strategies with a fixed - window rolling memory, and then Doubao1.6Flash (an inference and generation module) performs streaming generation. The generated result is unpacked, purified, and sentence - boundary aggregated through a guardrail protocol in the post - processing stage, ensuring that only directly speakable content is output externally and suppressing pseudo - fallback such as "没听清" to overwrite the generated valid streaming answers. Subsequently, Qwen3 TTS Flash performs streaming synthesis by segments to form a PCM (Pulse - Code Modulation) audio chunk that can be generated and broadcast simultaneously. Finally, the server frames the text delta and audio chunk into stable WebSocket JSON (structured information) events respectively and returns them to the client according to the requested output preference, thus achieving a continuous low - latency closed - loop of "generation - synthesis - feedback" within the same connection.

[0046] Figure 3The goal of the memory mechanism module shown is to make the dialogue state "controllable and reproducible." Its principle does not rely on vector retrieval, but instead uses a fixed-window sliding queue to incorporate the upstream current-round text into the temporal context. Before entering downstream inference, it completes predictable context reorganization to simultaneously suppress latency amplification and historical hijacking. During the input access phase, the upstream text is cleaned and standardized, then a round number is generated and bound to the session ID, ensuring strict round pairs between user input and assistant output within the same session. Working memory writing begins with enqueuing, first performing token statistics and budget evaluation to reserve hard space for system constraints and multimodal input. If a threshold is exceeded, the oldest round is gradually eliminated through a "head-out" rolling pruning until convergence to the upper limit of the window (defaulting to the most recent 12 rounds), thus limiting context expansion to a predictable range. To reduce scanning costs caused by I / O (input / output) jitter and directory bloat, memory enumeration focuses only on text and metadata entries related to round pairs, and combines directory signatures and short TTL caching to reduce repeated traversals. The resulting fixed window snapshots are structured and encapsulated in the persistent path, and then atomically committed via `write_temp→rename` (a programming pattern that ensures data integrity and consistency by first writing to a temporary file and then atomically replacing it) to ensure crash consistency and update the history index to support auditing and replay. The injection path places system constraints with the highest priority, then injects window history from old to new, appends the current round's input, and concatenates it into a structured Prompt context. At the same time, gating constraints ensure that "user instructions take precedence over memory" and skip injection when the input is empty or too short. Memory is disabled in visually triggered rounds such as "Look at Maruko" to avoid cross-modal misbinding and hallucination transfer, thus making memory a stable background evidence rather than a decision source.

[0047] This embodiment uses rounds as the smallest unit of consistency, completing voice acquisition and stabilization within a WebSocket long connection. It utilizes Qwen3 ASR Flash transcription and trigger routing, optional "Wanzi Kankan" visual injection and multimodal encapsulation; Doubao1.6 Flash streaming generation, output cleansing and sentence boundary aggregation; and Qwen3 TTS Flash streaming synthesis with text_delta (text change) and audio_chunk (audio chunk) events for feedback, thus achieving low-latency voice closed loop and trigger-based visual interaction closed loop through parallel and streaming orchestration.

[0048] To illustrate the actual working process of the streaming multimodal interaction backend kernel described in this embodiment, the following example illustrates the system's processing chain from input to output using a specific interaction scenario. It should be noted that this embodiment is only used to illustrate the processing mechanism of the present invention and does not constitute a limitation on the scope of protection of the present invention.

[0049] In some embodiments, users interact with a trendy smart agent in real-time via voice through a terminal device. Assuming a persistent connection has been established for the current session, the system is in a state of continuously receiving audio streams and transmitting text and voice results. The user first issues a voice command: "Maruko, look what's on my desk." After the terminal device collects the audio data for this round, it sends it as the main input for this round to the server-side processing link, along with the current session identifier, request identifier, and control flag for this round. The system first performs round-level acceptance and status registration on the incoming request, confirming that the current session is not occupied by other concurrent rounds, and allocates processing rights for this round. Subsequently, the voice input enters an automatic transcription process, resulting in the text result "Maruko, look what's on my desk." The routing determination unit (trigger routing module) performs standardized trigger word detection on the text, identifying that it contains the visual trigger phrase "Maruko, look," thus classifying this round as a "conditional visual injection round" rather than a plain text round.

[0050] During the visual injection phase, the system does not directly read the most recently generated image, but instead enters the "latest and stable asset selection" process. This process first filters recent assets in the candidate image set in reverse order of generation time, and then performs stability verification on the target file most likely to belong to this round of interaction. Specifically, the system continuously samples the file size, modification time, and readable status of candidate images within short time intervals; if two consecutive sampling results are consistent, and the file age exceeds a preset safety time window, the image is considered to have left the asynchronous writing phase and can be used as visual input for this round. If the latest candidate fails the verification, it automatically reverts to the previous stable candidate, or terminates visual injection when there are no qualified candidates. In this embodiment, the system successfully obtains a stable image and encapsulates it together with the transcribed text as the multimodal input for this round.

[0051] Meanwhile, the short-term memory loading unit retrieves the text context directly related to the current session from the most recent rounds using a fixed scrolling window strategy. However, since this round is visually triggered, the system only retains the most recent necessary context summary and does not inject longer history to avoid the semantics of older rounds interfering with the current visual judgment. After this processing, the structured input for this round (context construction module) consists of four parts: the current user text, a stable image, the necessary shortest context, and system-level behavioral constraint information. This structured input is then fed into the upstream multimodal generation unit.

[0052] The upstream model begins streaming incremental text. For example, the model first generates "I see a laptop on the desktop," then continues to generate "there's also a keyboard and a small bottle next to it," and finally adds "it looks like an office or development scene where people are working." This content is not returned all at once, but arrives gradually in continuous incremental segments. The system-side segment aggregation unit (output processing module) accumulates the incremental text in real time and continuously detects whether a playable segment has been formed. When the first segment of text, "I see a laptop on the desktop," meets the playable conditions in terms of punctuation, length, and semantic completeness, the system immediately marks it as the first playable segment and sends it to the speech synthesis module without waiting for all subsequent segments to finish generating. Subsequently, the second and third sentences, after satisfying the segment closure conditions respectively, are sequentially sent to the speech synthesis module, achieving a streaming response of simultaneous generation, aggregation, and playback.

[0053] During the output shaping phase, the system performs playable boundary cleaning on the upstream returned content, removing internal control information, debugging traces, and potential meta-hints that should not be output to the user, and trimming incomplete segments at the end that may affect the fluency of reading aloud. If the upstream result is normal, user-visible text and audio output are generated; if an empty response, anomaly marker, or unplayable content is detected, a conservative fallback strategy is triggered, outputting a safe and concise alternative response. In this embodiment, the final text result returned by the system to the terminal may be: "I see a laptop on the desktop, with a keyboard and a small bottle next to it, like an office or development scene." Simultaneously, the voice result is streamed back in multiple sequential audio segments, allowing the terminal to start playing immediately after receiving the first audio segment, thereby shortening the perceived waiting time for the user.

[0054] In this embodiment, the typical outputs of each processing unit can be summarized as follows: the speech transcription unit outputs the text transcription result of this round; the trigger determination unit outputs the routing conclusion "visual injection = yes"; the asset parsing unit (visual processing module) outputs the injection permission result "a stable image has been selected"; the memory loading unit outputs the "most recent necessary context summary"; the multimodal generation unit outputs continuous text increments; the sentence aggregation unit outputs at least one sentence segment that can be pre-played; the speech synthesis unit outputs the corresponding streaming audio segment; and the output shaping unit (output processing module) outputs the final playable text facing the front end. Therefore, this embodiment does not perform unified post-processing after the answer is completed, but rather coordinates mechanisms such as trigger determination, stable asset selection, short-window memory constraints, sentence-level pre-release, and streaming audio synthesis within the round, enabling the system to achieve low latency, high consistency, and reproducible real-time response in multimodal interactive scenarios. Specifically: 1. Upstream client model API call project In the engineering implementation of the client link, the input is not assumed to be "naturally text-bearing", but rather enters the interaction loop in the form of voice bytes as the primary input. When only audio is submitted upstream and there is a lack of text, the link will trigger an automatic transcription logic on the service side, sending the audio bytes into Qwen3 ASR Flash to obtain the text result of the controllable length for this round, and using the request-level request_id / session_id (session identifier) to penetrate into the subsequent full-link tracking and concurrent governance.

[0055] Subsequently, the routing layer performs trigger word detection on the text: when "Wanzi take a look" is hit, the system determines it as a visual branch round. If the client already carries an image, it directly enters the multimodal round and strips the trigger word to avoid wasting invalid tokens. Otherwise, according to the "latest stable asset" strategy, the most recently modified image file is selected from the agreed diagnostic directory, read, and then re-encoded and compressed under size and volume constraints, and the image is encapsulated into an image_url (image uniform resource identifier) data block (data URL, data uniform resource identifier) that can be directly consumed by the model, ensuring that the visual injection is both usable and does not violate the latency target. On this basis, the text or multimodal input is uniformly organized into a usermessage (user message) for the generation model, enabling the main path "voice → voice" and the branch "voice → vision → voice" to share the same set of subsequent generation and feedback mechanisms, thus strictly converging the routing differences to the point of "whether to attach image blocks", reducing the uncontrollable complexity brought by engineering forks.

[0056] At the level of three-model collaboration, the link positions Doubao 1.6 Flash as the core of streaming generation, and splits the text feedback and voice feedback into parallel event streams. The model output first undergoes secondary purification through the protocol and style layer, stripping markers, bracket fragments, and action symbols that may pollute TTS, and performing lightweight normalization on the output to ensure the consistency of "readability" and "pronounceability". To avoid directly driving voice jitter with token-level fragments, the system introduces an incremental aggregator to packetize the delta with a fixed length and form a more stable TTS trigger granularity at the sentence boundary level. At the same time, the text delta can be continuously fed back to maintain the "visible response", while the voice side sends the sentence segment into the streaming synthesizer of Qwen3 TTS Flash. It adopts a "two-stage strategy" under queue pressure to accelerate the first packet and merge the throughput, and applies deterministic tail padding and optional last-word repair at the end stage, providing a recovery path for streaming boundary problems such as "the loss of the last one or two words" from an engineering perspective, and finally feeding back to the client in the form of audio_chunk / done (end marker) to complete the voice loop.

[0057] 2. Fixed Rolling Memory Window (Memory Management Module) Implementation and Gated Injection Engineering The memory module adopts the method of "diagnostic output-driven file-based working memory + fixed window injection" in engineering implementation: The system maintains an independent memory management object for each user_id, explicitly parameterizes the window size, cache TTL, eviction policy, and injection budget, and uses mutex / commit locks to constrain the reading and writing order within the same session to avoid cross-contamination of memory snapshots caused by concurrent rounds. Memory writing takes the "round pair" as the smallest unit, pairs and archives the user-side structured information (JSON) and the assistant-side text (TXT) under the same six-digit round identifier, and sorts them by timestamp to form a strict time series; to reduce the I / O cost of the hot path, the scanning stage only focuses on JSON / TXT and constructs a short-term cache with "directory signature (quantity + latest mtime (modified time))", so as to maintain a stable traversal overhead even in the case of continuous growth of large files such as audio. In the window maintenance stage, head eviction is performed on historical rounds exceeding the upper limit to ensure a constant upper bound, and both eviction and retention are at the granularity of round pairs to ensure replay consistency. In the injection stage, the "user input priority" is solidified as a strong gate: when the current round of input is empty or insufficient to form a valid instruction, memory splicing is directly skipped to avoid historical overwriting of the current intention; when "look at the ball" is detected or it is marked as a visual round, memory injection is forcibly closed, and only the current round of instruction after removing the trigger word and the latest visual evidence are retained to reduce the risk of cross-modal misbinding; in a regular voice round, the module first reserves a budget for system constraints and style control, then compresses and splices the historical rounds within the fixed window in the order from old to new, and assembles them with the current round of input into a structured prompt for downstream inference, thus achieving a consistent implementation closed-loop among the three engineering goals of "controllable context cost, predictable latency, and visual rounds not being interfered by memory".

[0058] 3. Downstream Back-end and Server Engineering The downstream project uses a WebSocket server to receive real-time messages from the upstream client and complete the thin-layer scheduling of "parsing - verification - isolation - forwarding", while encapsulating multimodal reasoning, streaming generation, and speech synthesis into reusable backend pipeline executors. On the server side, session identifiers and stable user_ids are generated during the handshake phase, and each round of requests is constrained within a unified message body protocol, strictly restricting the sizes of text, picture, audio, and tactile payloads. It supports automatically picking up the latest audio from the diagnostic input directory for audio / auto_latest, and at the same time, a hard gate is implemented for auto_latest_image that "only allows injection when the text contains the trigger phrase '丸子看看'" to avoid unnecessary expansion caused by unconditional visual context. After completing payload extraction, the server constructs the request context and the user turn object, passes them to the backend executor, and encapsulates the returned events frame by frame into JSON and pushes them back to the client. Among them, the two event streams of text_delta (a small increment of newly arrived text during the streaming output process) / done (a completion flag indicating that the text output of this round has ended) and audio_chunk / done are separated. The audio is transmitted in base64 after being fragmented into PCM and supports being saved as a WAV (Waveform Audio File Format, an audio file format) for playback and regression testing.

[0059] On the backend side, ASR, orchestrator, and TTS are organized in a pure pipeline form. When the input lacks text but provides audio bytes, ASR is called for transcription. Subsequently, explicit master-slave marking is performed on the current round of user input and character budgeting is implemented. Working memory is only retrieved and cropped as a secondary context according to a fixed window, and the memory is directly degraded in visual / tactile rounds to avoid overwriting the current perception. The system prompt is assembled in layers of "personality / control strategy / memory / style" and the control strategy is preferentially retained within the hard budget. For upstream exceptions such as "Text too long", a one-time fallback retry is provided to maintain the stability of the streaming output. The output of the orchestrator enters the final segment extraction and aggregator in the form of streaming deltas, ensuring that the foreground can obtain the reportable sentence boundary segments as early as possible. Subsequently, the final text undergoes protocol stripping, secondary purification, and security fence determination, and then the speech synthesis is output in the form of streaming PCM and transmitted in parallel with the text events. At the same time, each round of input and output is persistently stored in the diagnostic output directory in an atomic write manner, forming a traceable dialogue pair, metric log, and audio evidence chain, thereby maintaining an engineering closed-loop of "isolation for each user, replayability for each round, and end-to-end testability" under concurrent conditions.

[0060] 4. Local Demo Deployment and Sidecar-style Interruptible Interaction Closed-loop Project In the local computer deployment implementation, the system abstracts the personal computer as a "trendy toy" demonstration carrier. It achieves a closed-loop link of "acquisition-inference-playback-interruptibility" through the separate side vehicle and main control orchestration: The acquisition side continuously detects, acquires and analyzes the microphone audio in an independent process. Based on the noise floor estimation, it uses "three threshold triggers" to complete the voice entry determination (sound pressure threshold, VAD to determine voice, and joint constraint of voice band ratio). It forms stable segmented recordings with pre-roll frames and silence termination strategy. Then, the audio assets are written to disk in a fixed slot rolling manner, and the "this round of input is available" event is published through the atomically written manifest file. At the same time, a playback lock with status and expiration time is written to suppress false triggers during playback. Thus, the visibility consistency of the recording, manifest, and lock is taken as a prerequisite for the reliability of local demonstration.

[0061] The master control unit uses an orchestration process on the same machine to achieve multi-process collaboration. It continuously polls the input list and deduplicates round numbers. When not in the playback / cooling window, it sends a request for "automatically select the latest audio + optional automatic latest image" to the backend server entry point via WebSocket, and uses round-level hard cutoff and receive idle timeout to prevent half-open connections from causing the link to "die". When the backend returns a completion event, the master control unit selects the latest generated audio result from the local output directory and starts the player for playback. At the same time, before playback, it performs playable processing on the output audio and writes the playback metadata to the lock file for other processes to read.

[0062] In engineering, the interruption mechanism is positioned as a monitoring loop that operates in parallel with the recording side, using the lock as the sole arbitration signal. When the lock enters the playing state, the monitor samples the microphone in real time and combines this with the playback waveform to perform file-level echo suppression discrimination, including dual criteria of correlation and spectral similarity, echo baseline estimation, confirmation window, and minimum interruption interval. It only triggers the player to stop and clear the lock under the condition that "non-speaker echo and sufficient voice evidence" are met, thus simultaneously satisfying the sensitivity requirements of presentation interaction and the suppression requirements of "speaker suicide interruption" under local deployment conditions.

[0063] 5. Production-level regression testing and performance acceptance engineering To verify the low latency, high completion rate, and concurrency stability of this embodiment in a trendy multimodal real-time interactive scenario, end-to-end latency testing, tiered concurrency stress testing, and consistency regression verification were performed on the streaming backend kernel. The tests employed a reproducible load-driven regression approach, continuously initiating multiple rounds of interaction requests under the condition of reusing long connections. Statistical analysis was conducted on the arrival time of the first incremental text, the arrival time of the first audio segment, the overall completion time, and the completion rate under different concurrency levels to quantify the actual effectiveness of this invention in terms of early release of streaming segments, tiered concurrency constraints, and asset consistency control.

[0064] In the end-to-end latency test, the system recorded the latency of the first text increment, the latency of the first audio segment, and the latency of the entire round for each request. The test results show that, under local optimized deployment and long connection reuse conditions, this implementation can achieve a first response time of approximately one second: in the optimal round, the shortest latency for the first text increment can reach approximately 0.70 s, and the shortest latency for the first audio segment can reach approximately 1.42 s; in another set of stable test samples, the latency for the first text increment is approximately 1.04 s, the latency for the first audio segment is approximately 1.77 s, and the latency for the entire round of voice output is approximately 4.09 s. Furthermore, in multiple rounds of regression samples, the increment of the first text can stably fall within the range of approximately 0.70 to 0.83 seconds, and the increment of the first audio segment can stably fall within the range of approximately 1.42 to 1.52 seconds. This indicates that this embodiment, through sentence-level boundary aggregation and advance synthesis mechanisms, has shifted the user-perceived waiting time from "starting broadcast after the entire answer is completed" to "broadcasting immediately after the first broadcastable sentence segment is formed," thereby significantly compressing the first packet waiting time.

[0065] In the concurrent stress test, the system submitted interaction rounds in batches under a fixed concurrency level, and the median completion latency, 95th percentile completion latency, and success rate were statistically analyzed. The test results show that: under 5 concurrency levels, the median completion latency was 3125ms, the 95th percentile completion latency was 5266ms, and all 20 / 20 requests succeeded; under 10 concurrency levels, the median completion latency was 3704ms, the 95th percentile completion latency was 4985ms, and all 40 / 40 requests succeeded; under 20 concurrency levels, the median completion latency was 5406ms, the 95th percentile completion latency was 6781ms, and all 80 / 80 requests succeeded. The above results show that, under the condition of progressively increasing concurrent load, the session-level mutual exclusion, system-level in-transit quota, and upstream inference gate adopted in this invention can effectively suppress round accumulation and tail latency expansion, enabling the system to maintain a 100% completion rate within the current test level, and the long-tail latency is still within a controllable range.

[0066] In addition to the quantitative results mentioned above, the test system also conducted regression verification on streaming interruption consistency, upstream failure injection and backoff retries, file disk persistence race conditions and latest asset consistency, trigger word routing, memory window correctness and priority, repeated request idempotency, disconnection reconnection and session recovery, multi-instance pseudo-copy consistency, and real gateway / proxy deployment patterns. The results show that this embodiment can avoid half-write file reading and cross-round mismatch through the "latest and stable assets" filtering mechanism in visual trigger rounds, maintain the injection principle of "current user input takes precedence over historical memory" in short window memory mode, and maintain output boundary stability and perform conservative fallback in streaming interruption and abnormal return scenarios. This indicates that the present invention not only has low latency advantages, but also has consistency, robustness, and reproducibility that match actual deployment.

[0067] 6. LAN Demo: Self-built API and Interruptible Streaming Return Project The API project was built and demonstrated on a local area network using FastAPI (a framework for developing Web APIs) as the service skeleton. By listening on 0.0.0.0, mobile phones and computers can access a unified entry point by placing them on the same Wi-Fi network. The entire interaction protocol of "round request → streaming event return" is carried by a single WebSocket endpoint / ws / chat. The client submits request_id, user_id, text, and optional audio / images / touch payloads via user_message, and declares whether text and audio feedback are needed via output{text,audio}. After the connection is established, the server performs JSON payload validation and type constraints, establishes an authentication subject based on the API key, and adds two types of production-grade protection. One type is token bucket rate limiting based on the key, which returns a 429 error with retry information if the limit is exceeded. The other type is fast failure with a 429 error if the concurrent access limit is exceeded by both the key and the global inflight concurrency gate. This effectively cuts off the cascading backlog caused by "high-frequency clicks on a single device" and "simultaneous requests from multiple devices" at the entry layer.

[0068] When accessing the backend via Inflight protection, the server drives the actual inference chain using a unified Stream_Turn interface, and encapsulates the assistant_delta / assistant_done and assistant_audio_chunk / assistant_audio_done events generated by the backend into WS JSON for frame-by-frame transmission. Here, assistant_delta / assistant_done represents the streaming incremental segment of the assistant's text response and the text response completion marker; assistant_audio_chunk / assistant_audio_done represents the streaming audio chunk of the assistant's voice response and the voice output completion marker. The audio side segments the PCM according to a fixed maximum chunk size and sends it using base64 encoding, while retaining metadata such as sampling rate, format, and sequence number to support real-time client playback and disk-based verification. Metrics and observability are embedded at round boundaries: the server marks the ingress time point after access completion and inbound parsing, and records success / failure and time consumption at the end of the round, providing a unified standard for load testing and online regression testing.

[0069] In this API configuration, the interruption mechanism is explicitly positioned on the "client playback side" rather than the server inference side: the player script decodes and plays the assistant_audio_chunk immediately upon receiving it. If the user needs to interrupt, the client stops playback and initiates a new round or disconnects directly. The server terminates the current event stream by using WebSocket disconnection as the termination signal. Furthermore, the inflight and rate limiter at the entry point ensure that rapid re-entry after an interruption will not drag the system into a state of concurrency out of control.

[0070] The streaming agent backend kernel and its control method proposed in this embodiment for trendy multimodal interactive terminals are applicable to real-time interactive systems that use voice as the primary entry point, conditional vision as a secondary entry point, and require low latency, continuous response, and stable broadcasting. The innovation of this solution lies not in replacing a specific large model, but in the unified engineering design of key aspects such as single-round processing structure, streaming segment release, stable injection of visual assets, short-window memory constraint update, concurrent backpressure control, and output safety shaping. This makes low latency and high consistency no longer dependent on empirical parameter tuning, but rather defineable, verifiable, and reproducible system attributes.

[0071] Specifically, this embodiment uses a "single round" as the smallest unit of consistency processing. Each round receives user text, speech-to-text results, optional image input markers, and session control status, and outputs incremental text, pre-sent speech synthesis segments, audio response control signals, and round status events. At the beginning of the round, the system first standardizes the input and simultaneously completes trigger phrase recognition, current round intent routing, and session status confirmation. Subsequently, speech result confirmation, short-term memory loading, and external asset preparation are executed in parallel paths to compress the critical path length and minimize the release time of the first valid response. In other words, this embodiment does not operate in the traditional serial manner of "unified inference after all preparations are completed," but rather uses parallel preprocessing within the round to move preparation tasks that do not disrupt consistency forward and execute them concurrently, thereby reducing the wait time for the first response.

[0072] In the streaming output stage, this embodiment introduces a sentence-level early release mechanism. The upstream model does not return a complete answer all at once, but rather incremental text arriving continuously. The system continuously aggregates these incremental texts and determines in real time whether the current segment has reached a "playable" state. "Playable" does not simply mean immediately releasing the text upon encountering a period, but rather simultaneously meeting the following conditions: first, the text has an explicit or implicit sentence end signal; second, the end is not in an incomplete state such as an unclosed quotation mark, unclosed parentheses, incomplete enumeration, or hanging conjunction; third, the segment length reaches the minimum threshold for audible perception; and fourth, a semantically complete reading unit has been formed. To achieve the above determination, the system comprehensively uses punctuation detection, character category analysis, local state machines, minimum length thresholds, and pseudo-termination exclusion rules to perform boundary disambiguation on the streaming text. In this way, speech synthesis does not need to wait for the entire answer to end, but can start immediately after the first playable segment is formed, thus changing the initial audio waiting condition from "full text generation complete" to "first stable segment complete," significantly reducing the initial packet latency in the interaction.

[0073] Regarding multimodal injection, this embodiment does not allow visual input to be unconditionally always on, but instead limits visual capabilities to conditional branches driven by specific trigger phrases. The system only enters the image parsing process when the current round explicitly hits the standardized trigger expression and the visual injection flag for this round is valid. The key to this process is not the superficial operation of "getting the latest file", but "getting the latest and stable readable assets". To avoid half-write reads, cross-round erroneous retrieval, and incorrect injection caused by front-end photography, network transmission, asynchronous disk writing, or cache refresh, this embodiment sets up a two-stage stability judgment mechanism: First, the most recent assets are screened in chronological order in the candidate set. Then, the target file is sampled at least twice in a short time interval to compare whether its size, modification time, and readable status are consistent. Only when the continuous sampling results are stable and the generation time of the asset has exceeded the preset safe age threshold is the image considered to have left the writing stage and can be safely injected into the current round. If the latest candidate fails the stability test, the system automatically reverts to the previous candidate that meets the conditions, or directly abandons the visual injection for this round, to ensure "better not to inject than to inject incorrectly". The safety age threshold is not necessarily hardcoded to a fixed value, but can be set according to the terminal's camera resolution, transmission link, average file size, and disk write fluctuation range, thus adapting to different hardware and deployment environments. Through this mechanism, the start and stop boundaries, reading timing, and injection objects of visual input are all clearly constrained, avoiding the asset race problem most common in multimodal systems.

[0074] To address the characteristics of interactive elements in trendy toys, where "recent context is important and distant history should not overshadow current instructions," this embodiment employs a fixed-capacity scrolling window memory mechanism. The system retains only the most recent rounds of user input, system responses, and necessary state summaries directly related to the current interaction, organized chronologically into a finite-length queue. When a new round is written, if the window is not full, it is appended directly; if the window is full, the earliest round is removed before writing the current content, thus ensuring that the memory size always has an upper bound, and loading time and injection costs remain controllable. Furthermore, this embodiment emphasizes the principle that "memory is subordinate to current input": memory is used to supplement the consistency of continuous dialogue, not to dominate the current round. Especially in visually triggered rounds, the system can proactively reduce the intensity of historical memory injection, even retaining only the most recent necessary context, to avoid contaminating current visual judgments with old round text. This memory constraint mechanism allows the system to maintain continuity while prioritizing responses to current user instructions.

[0075] In terms of concurrency control, this embodiment constructs a layered backpressure and deterministic constraint mechanism to prevent real-time systems from experiencing backlog, reentrancy, or tail latency amplification in multi-user, multi-round, or upstream jitter scenarios. The first layer is a session-level mutual exclusion constraint, used to ensure that only one round of the same session enters the critical state area at any given time, preventing concurrent input from the same user from simultaneously modifying shared states. The second layer is a system-level in-transit request quota, used to limit the number of rounds in the active processing phase of the entire service instance at the same time, preventing instantaneous peaks from overwhelming resources. The third layer is an upstream model call gate, used to limit the concurrency of high-cost inference requests, separating inference resources from local lightweight logic for rate limiting. Its coordination logic is as follows: after a request enters, it first attempts to acquire session processing qualifications, and then applies for system active quotas. Only when high-cost model inference is actually needed is the inference gate occupied; when the request ends, is canceled, or exits abnormally, the relevant occupied resources are released in reverse order. If any stage fails to acquire or times out, the system immediately executes fast failure and conservative degradation instead of waiting indefinitely in a queue, in order to avoid deadlock, starvation, and false occupancy. This mechanism ensures the stability of response in interactive trendy toy scenarios, so that "low latency" is no longer just an average performance, but suppresses long tails through concurrency boundary control.

[0076] Regarding output controllability, this embodiment divides the upstream model's generated results into two processing areas: "playable content" and "internal check content." For playable content, the system performs metadata stripping, abnormal control character cleaning, prompt fragment removal, debugging information filtering, and incomplete sentence trimming to ensure that the output text has clear boundaries and is suitable for direct entry into speech synthesis. For the internal check area, it is used to identify risk signals such as prompt word leakage, abnormal echoing, format contamination, and upstream return defects. When the upstream result is detected to be empty, contains control metadata that should not be played, has traces of prompt templates, or exhibits an abnormal pattern that is clearly unsuitable for direct output to the user, the system does not transmit it as is, but instead switches to a conservative fallback response to maintain the consistency and security of the front-end interaction. Thus, the speech synthesis module always only faces playable text that has undergone boundary trimming and risk filtering, avoiding direct exposure of internal model traces to the user side. This embodiment mainly includes the following innovations: 1. Innovations in the upstream client-side link: voice closed loop and trigger-based visual injection The upstream link uses long-lived WebSocket connections as its core carrier, converging the "input-side multimodal acquisition - upstream inference stream - output-side audio stream" into a controllable event sequence within a single session. During the handshake phase, the server uniformly binds request_id (request identifier) ​​and session_id (session identifier), and uses user_id (user identifier) ​​as the key index for session ownership and concurrency isolation, thereby ensuring deterministic timing consistency and resource quota constraints for the same user even in multi-connection scenarios. This routing implements multimodal safety boundaries for input messages: limiting single-frame payload, text length, number of images and size of a single image, upper limit of audio bytes, and scale of haptic payload to avoid amplified jitter caused by multimodal payloads at the transport layer and upstream call layer; at the same time, the link explicitly supports the inclusion of three types of fields—images, audio, and touch—into a unified UserTurn data structure (a stage or node in a dialogue flow where it is the user's turn to speak or provide information), ensuring that "voice-visual-touch" can be combined or omitted within the same round, and directly throwing a diagnostic error when fields do not match to force the interface to evolve in a consistent manner with the orchestration layer.

[0077] The visual branch is strictly attached to the speech semantics trigger rather than an independent input channel: When the ASR transcribed text hits the trigger phrase "Wanzi take a look" after normalization, the visual injection is enabled within the same turn in the link. Specifically, it is to parse the latest and stable image assets from the user isolation directory _diag_in / <user_id> / , and submit this image together with the "transcribed text after trigger (including necessary instruction constraints)" as multimodal input to Doubao 1.6 Flash (large language model); At this time, a verifiable structural change has occurred in the collaborative relationship of the three models: Qwen3 ASR Flash (real-time speech recognition model) still serves as the semantic entry and trigger judgment source, Doubao 1.6 Flash is responsible for the response planning and text generation after the "semantic + visual" fusion, while Qwen3 TTS Flash remains unchanged and is only responsible for converting the final generated text into speech. In order not to introduce additional serial waiting in the visual branch, the link continues to run in parallel for "latest image acquisition" and "transcription convergence / context preparation": Once the trigger word is recognized in the early stage of transcription, the image parsing and stability verification (double stat (status) and safety age threshold) can be started in advance to ensure that the image is in a readable state when the midstream inference really starts, thus avoiding the additional pause caused by "finding the picture after the transcription is completed". Finally, the parallel pipeline is also adopted in the turn output stage: When Doubao1.6 Flash generates text in a streaming incremental manner, the system generates synthesizable segments according to the end-of-sentence boundary in the incremental aggregation and submits them to Qwen3 TTS Flash for synthesis in advance. The audio is transmitted back in chunks and can be played edge-to-edge, enabling the user to obtain continuous responses. This enables the link to maintain the same closed-loop principle whether the vision is triggered or not: taking the voice output as the end point, and organizing ASR, context construction, visual preparation, LLM generation, and TTS synthesis into an overlapping execution pipeline through parallelization, rather than serial stages waiting for each other.

[0078] The key to parallelization and low latency does not lie in "serial calls being made faster", but in stripping the parallelizable waiting time from the critical path: on the one hand, the link continuously produces assistant_delta (the latest generated content segment in the streaming response) from Doubao (Doubao) as streaming events and sets a three-layer insurance of "first token timeout / stream idle timeout / whole-round hard timeout" to avoid the client waiting indefinitely due to a half-open upstream network or a jam in SSE (Server-Sent Events). On the other hand, when voice output is required, the link creates an independent TTS background task before entering the upstream stream, enabling it to run in parallel with the LLM event consumption, and utilizes the tts_ready_text field output by the orchestration layer (plain text that has been preprocessed, formatted, and cleaned and can be directly fed into TTS for synthesis) to achieve "sentence-level early synthesis", that is, when the accumulated text forms a completed sentence, it is immediately segmented and fed into Qwen3 TTS Flash to generate a PCM (Pulse Code Modulation) stream without waiting for the entire segment to be generated, thus significantly reducing the first sound packet latency and improving the time perceptibility of the conversation. On the transmission side, the system buffers and merges the fine-grained PCM shards generated by TTS, reducing the number of WebSocket write calls with a fixed upper limit chunk size and a refresh interval of approximately 120 ms, which not only controls the network and event loop load but also avoids listening perception jamming caused by excessive aggregation. On the concurrency side, through the combination constraint of "per-user single-flight (concurrency control mode) lock + global inflight semaphore + upstream call semaphore", the parallel rounds of the same user across connections are compressed into a single in-flight request, and an independent concurrency gate is maintained for Doubao1.6 Flash calls to prevent queue expansion and tail latency runaway caused by high-concurrency accumulation, so that the parallelization obtains controllable throughput rather than uncontrollable congestion.

[0079] 2. Innovation Points of Fixed Window Rolling Memory: Full Injection of Time Sequence and Gated Priority Constraint The core idea of ​​the fixed-window scrolling memory mechanism is to abstract the "dialogue history" into a strictly controllable, verifiable, and low-latency context buffer: the system does not perform semantic retrieval or sparse recall, but instead uses time sequence as the sole axis, injecting the user-assistant paired information from the most recent N rounds (default 12 rounds) into the model. This allows the model to determine relevance on a unified and continuous dialogue trajectory, thus returning the "memory selection power" to the generative model. At the same time, engineering constraints prevent memory from taking over. Its implementation first relies on round pairing rules: each round of dialogue is represented as a pair of "user-side text + assistant-side text," and is bound using a stable identifier from the same round. Then, the latest timestamp of the pair is used as the sorting basis for that round, ensuring that the scrolling window naturally satisfies the causal narrative structure of "early to late." Based on this, the system defines two strong constraint gating mechanisms to ensure instruction priority. The first gating mechanism is to disable injection for empty inputs: when the user's input in the current round is empty or insufficient to constitute a valid instruction, the system directly skips memory concatenation, avoiding the historical context from dominating the output in the absence of a current intent anchor. However, if the system detects that the user explicitly makes a recall request such as "review / recap / remember," it allows injection into the window even with very short inputs to maintain the functional consistency of "providing memory when asked." The second gating mechanism is to disable injection for visual triggers. When a visual interaction trigger word is detected, such as a specific phrase used to request the latest visual evidence, the system forcibly disables memory injection and performs minimal cleaning and completion of the input, ensuring that the multimodal round is based solely on the "latest observation," avoiding false associations or misleading biases in visual question-and-answer sessions caused by historical text. The organization of memory blocks also reflects the design philosophy of "controllability first." Each round of history is not stacked with long paragraphs of the original text, but compressed into a single-line summary. For example, the first sentence of the user and the assistant's key points are retained at the same time, and the summary length is evenly distributed across rounds through budget allocation. If the overall length exceeds the allowed context limit, the system will gradually shrink the single-round summary budget and rebuild the memory block until the hard limit is met. Only in extreme cases will the final truncation be performed. In any case, preserving the user input in this round is the priority, fulfilling the priority commitment of "user instruction > memory > style" in terms of mechanism.

[0080] Furthermore, to enable the model to recover temporal relationships without additional retrieval and inference, the system generates relative time labels for each round, explicitly encoding "order" and "recency" into the context. This allows the model to directly reference temporal clues rather than guess when answering recall questions or explaining causal chains. To maintain low latency in real-world interactions, the mechanism also introduces robustness strategies for I / O (input / output): signatures are created only for the minimum set of files relevant to memory, and window results are cached within a short TTL to avoid repeated scanning in high-frequency rounds. Simultaneously, signature calculations deliberately ignore irrelevant growth items such as large-volume audio files to prevent frequent cache invalidation due to directory expansion, thus stabilizing the time complexity of memory management within a range that is "approximately linear with the number of rounds and approximately independent of the external data size." Finally, the "rolling" of the rolling window is not a logical concept but a strict state maintenance: when the number of historical rounds exceeds the window limit, the oldest rounds are evicted, and consistency cleanup, including byproducts bound to that round, is performed when configuration allows, to ensure constant memory capacity, stable sorting, and predictable injection costs. Overall, this is a memory mechanism that uses time-series full injection as its main component, gating and budgeting as its safeguards, and caching and minimum signature as its accelerators. It does not pursue "smarter retrieval" but rather "more stable context modeling," downgrading memory from "potentially usurping decision sources" to "interpretable background evidence," and ensuring through hard constraints that it will never overwrite the current user's intent.

[0081] 3. Innovations in backend services and round-based orchestration: Protocol-constrained streaming pipelines and deterministic guardrails. The backend and server adopt an overall architecture of "protocol layer decoupling + round-based streaming orchestration + deterministic guardrails". The principle is to first constrain multimodal payloads into verifiable round objects at the network entry point, then complete the end-to-end closed loop of ASR-MLLM-TTS through asynchronous pipeline, and introduce budget, security, and concurrency boundaries at key nodes to suppress tail latency and runaway risks. The server maintains long-lived connections via WebSocket, binding stable user_id and request_id to each round, ensuring consistent parsing of audio and images under the same identity semantics. At the entry stage, type and size validation is performed on text / image / audio / haptic feedback respectively. It also supports "automatically mounting the latest audio" reference-based input to reduce upload overhead and applies stronger gating to visual injection, allowing the latest image to be mounted only when the text matches the trigger phrase, thus preventing unconditional visual inflation at the protocol layer. Once the round enters the engine, the system parallelizes "semantic acquisition" and "context preparation": when text is missing but audio bytes are present, ASR transcription is immediately invoked to obtain the main semantics, while a fixed window timeline memory is constructed. However, the memory is strictly downgraded to a secondary context and directly suppressed in visual / tactile rounds to ensure that the current perception takes priority. To prevent upstream rejection of excessively long prompts, the system applies a hard budget to system_prompt and memory_block and prunes them according to priority. If necessary, a minimal rollback retry is performed to downgrade length errors to recoverable events. The multimodal generation phase is driven by streaming events. The output is limited to a playable area and a deterministic filter removes protocol markers and metadata. TTS synthesis is then initiated after the text passes through the guardrail. Audio is continuously produced in PCM blocks and sliced ​​into controlled-size base64 (an encoding method) frames for transmission back to reduce network jitter and event loop burden. Simultaneously, the entire link measures key moments such as ASR completion, first delta (increment), generation completion, first audio packet, and TTS completion, enabling latency to be stably decomposed and attributed for optimization. Overall, the key difference of this design lies not in model stacking, but in using stable session identity, trigger-based visual gating, parallel pipeline, budget pruning, and deterministic guardrails to solidify the collaboration of the three models into a repeatable, observable, and deployable multimodal round mechanism.

[0082] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A streaming processing method for multimodal interaction, characterized in that, Includes the following steps: S1. Receive input data from a user client, wherein the input data includes at least audio data; S2. Perform automatic speech recognition (ASR) processing on the audio data to generate the user text for the current round; S3. Detect trigger words in the user text and determine whether it matches the preset visual trigger conditions; If the visual trigger condition is met, the visual injection sub-process is executed: retrieve the image file with the latest timestamp from the storage directory associated with the user identifier, perform a stability check on the image file, and determine it as the visual input for the current round after the check passes; The stability check includes: sampling the file attributes and readable status of the candidate image file at least twice within a short time interval, and determining it as stable only when the consecutive sampling results are consistent and the file generation time exceeds the safe age threshold; if the visual triggering condition is not met, the current round is determined as a plain text round. S4. Based on the user text, visual input determination results, and context information obtained from the fixed scrolling memory window, construct a multimodal prompt context; wherein, if the current round is a visually triggered round, suppress or reduce the injection intensity of historical information in the fixed scrolling memory window; the fixed scrolling memory window is used to store user input and system response pairs from the most recent multiple rounds of dialogue in chronological order; when a new round is written, if the window is full, remove the earliest round of dialogue before writing; S5. Input the multimodal prompt context into the multimodal large language model MLLM and receive the streaming text increment returned by it; perform real-time aggregation and sentence boundary analysis on the streaming text increment to determine and extract the playable sentence segments; the playable sentence segments must simultaneously meet the following conditions: contain a sentence segment end signal, have no unclosed grammatical structure at the end, have a length exceeding the minimum perceptual threshold, and form a complete reading unit semantically; S6. Send the playable sentence segment into the text-to-speech (TTS) synthesis module to generate streaming audio data; and return the streaming text increment and / or streaming audio data to the user client.

2. The method according to claim 1, characterized in that, The method further includes a concurrency control step: for a received user request, first attempt to obtain session-level processing eligibility corresponding to the user session identifier; if successful, request a system-level active processing quota. The model call level concurrency quota is requested only when the multimodal large language model MLLM needs to be invoked; if the qualification or quota fails to be obtained or a timeout occurs at any stage, the current request processing is immediately terminated and a failure response is returned.

3. An electronic device, characterized in that, include: processor; A memory for storing a computer program executed by the processor; wherein, when executed by the processor, the computer program implements the method as described in any one of claims 1 to 2.

4. A multimodal interaction system employing the streaming processing method for multimodal interaction as described in any one of claims 1 to 2, characterized in that, include: The input receiving module is configured to receive input data from the user client. The speech recognition module is configured to recognize audio data and generate user text. The trigger routing module is configured to parse user text and determine whether to trigger visual injection. The vision processing module is configured to retrieve and verify image files from storage when vision injection is triggered. The memory management module is configured to maintain and provide a fixed capacity of historical dialogue context; the context construction module is configured to integrate user text, visual input information, and historical context to generate multimodal prompts. The inference generation module is configured to receive multimodal cues and return streaming text increments; the output processing module is configured to aggregate text increments, determine playable segments, and perform cleanup; the speech synthesis module is configured to synthesize playable segments into streaming audio data. And a response return module, configured to return text and audio streams to the client.

5. The system according to claim 4, characterized in that, The visual processing module is further configured to: filter candidate image files in reverse chronological order; sample the file attributes of the target candidate file at least twice within a short time interval, the file attributes including file size and modification time; compare the two sampling results, and if they are consistent and the file generation time exceeds the safe age threshold, then the image file is determined to be stable and usable.

6. The system according to claim 4, characterized in that, The memory management module uses a fixed-length first-in-first-out queue to store dialogue rounds, and actively suppresses the injection intensity of historical context or retains only the most recent round context during visually triggered rounds.

7. The system according to claim 4, characterized in that, The system also includes a concurrency control module, configured to manage session-level mutexes, system-level active request semaphores, and model call-level concurrent semaphores, and to trigger a fast failure mechanism when acquisition fails.

8. A humanoid robot, characterized in that, The system includes a multimodal interaction system as described in any one of claims 4 to 7, wherein the system is configured as an upper-level cognitive interaction module of the humanoid robot, for processing audio and visual inputs collected through the microphone and camera mounted on the robot, and generating interactive voice and control commands.

9. An AI interactive toy, characterized in that, The system includes a multimodal interaction system as described in any one of claims 4 to 7, wherein the system is configured as the core interaction control module of the AI ​​interactive toy, for processing audio and visual input collected through the microphone and camera mounted on the toy body, and generating interactive voice and feedback commands with companionship attributes.

Images