Multimodal interaction methods, devices and electronic devices, and storage media

Abstract

This application discloses a multimodal interaction method, device, electronic device, and storage medium. The method includes responding to multimodal input from a front end by matching a corresponding intelligent agent through a dynamic processor; obtaining a text segment to be synthesized using a streaming text slicing method based on the text content generated by the intelligent agent; and returning the text segment and the synthesized audio to the front end in real time. This application solves the problems of delayed voice feedback, rigid semantic slicing, and asynchronous interaction rhythm in traditional systems, significantly improving the naturalness and real-time experience of intelligent voice interaction. This application provides a real-time multimodal voice interaction method with streaming transmission capabilities, a semantically aware slicing mechanism, a synchronization control strategy, and multi-scenario adaptive capabilities.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal interaction method, device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, voice interaction systems have been widely used in various scenarios such as intelligent customer service, bank tutoring, and educational guidance.

[0003] Existing voice interaction systems typically include speech recognition, speech synthesis, and dialogue management modules. Their basic workflow is as follows: user inputs speech → ASR module recognizes it as text → dialogue module generates response text → TTS module converts the text into speech and reads it to the user. This workflow achieves the basic functions of human-computer voice interaction, but it still has significant shortcomings in real-time performance, streaming response capabilities, and semantic integrity processing. Especially in multi-turn dialogues and streaming scenarios, the traditional system's piece-by-piece processing method cannot meet the requirements for real-time feedback. Summary of the Invention

[0004] This application provides a multimodal interaction method, device, electronic device, and storage medium to meet the needs of highly interactive, multimodal scenarios.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a multimodal interaction method, the method comprising:

[0007] In response to multimodal input from the front end, the corresponding intelligent agent is matched through a dynamic processor;

[0008] Based on the text content generated by the intelligent agent, the text segment to be synthesized is obtained by using a streaming text slicing method;

[0009] The text fragment and the synthesized audio are returned to the front end in real time.

[0010] In some embodiments, the response to front-end multimodal input includes:

[0011] Based on the voice input interface, it receives real-time audio streams from the front end, calls the Automatic Speech Recognition (ASR) engine for streaming recognition, and converts the voice signal into segmented text data in real time.

[0012] Based on the text input interface, plain text data input by the user can be received directly via HTTP or WebSocket.

[0013] Based on the speech-to-text interface, pre-recorded speech files are processed and converted into text by the speech recognition module;

[0014] The text results from the different channels are formatted and standardized in a unified manner to form a unified text stream input.

[0015] In some embodiments, matching the corresponding agent via a dynamic processor includes:

[0016] Classify and identify the input data to determine the input type, and combine this with a preset business scenario to identify the business scenario type.

[0017] Based on the input type and business scenario, select the optimal processing logic and the corresponding intelligent agent instance.

[0018] In some embodiments, the agent is implemented based on a Large Language Model (LLM), and the operations performed by the agent include:

[0019] Perform semantic understanding on user input and parse the semantic information of the text corresponding to multimodal input;

[0020] Dynamically infer the user's task objective based on the user's voice or text content to complete intent recognition;

[0021] The semantic context caching mechanism retains session history information, ensuring the continuity of multi-round interaction context, and dynamically adjusts the generation strategy according to the context.

[0022] Based on the results of semantic parsing and intent recognition, response text and corresponding control tags are generated to complete natural language generation.

[0023] In some embodiments, obtaining the text segment to be synthesized by using a streaming text slicing method based on the text content generated by the intelligent agent includes:

[0024] The text stream output by the intelligent agent is segmented in real time to ensure the semantic integrity of the text and adapt to the requirements of simultaneous text-to-speech (TTS) broadcasting.

[0025] By controlling the flag-driven slice, a dynamic threshold is used to determine whether to immediately send the buffer.

[0026] The step of driving the slice via control markers includes:

[0027] When explicit control tags appear in a text stream, the text is divided into text before the tag, text after the control tag, and text after the tag.

[0028] The text before tagging is filtered and appended to the buffer, and the contents of the buffer are sent to the text-to-speech (TTS) service in real time.

[0029] Control flags are used to indicate whether to broadcast immediately or end the current round of broadcasting. After being flagged, the text is put back into the buffer to await further processing.

[0030] The step of determining whether to immediately send the buffer based on a dynamic threshold includes:

[0031] The dynamic threshold is dynamically adjusted based on text type, semantic complexity, network latency, and system load. If the text in the buffer reaches the minimum fragment length and encounters a punctuation mark, it will be sent immediately.

[0032] If the conditions are not met, the text continues to accumulate and waits for the next round of slicing.

[0033] In some embodiments, returning the text fragment and the speech-synthesized audio to the front end in real time includes:

[0034] The text-to-speech (TTS) engine is deployed as a microservice of a multilingual speech synthesis system based on a large language model, and the text fragments are received via the WebSocket protocol.

[0035] The text-to-speech (TTS) engine generates an audio stream and plays it simultaneously from the text fragment, outputting it to the front end in a synchronized manner.

[0036] In some embodiments, the step of returning the text fragment and the speech-synthesized audio to the front end in real time further includes:

[0037] Based on the publish-subscribe model, communication between modules is realized through an event bus, which listens for intelligent agent output events and text-to-speech (TTS) status events and triggers corresponding linkage operations.

[0038] Real-time tracking of the agent's text generation progress and text-to-speech (TTS) broadcast status, maintenance of synchronization signals between modules, and ensuring consistent interaction.

[0039] Secondly, embodiments of this application also provide a multimodal interaction device, the device comprising:

[0040] The response matching module is used to respond to multimodal input from the front end and match the corresponding agent through a dynamic processor;

[0041] The slicing module is used to obtain text segments to be synthesized from the text content generated by the intelligent agent using a streaming text slicing method.

[0042] The feedback module is used to return the text fragments and the synthesized audio to the front end in real time.

[0043] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the above-described method.

[0044] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the above-described method.

[0045] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: In response to multimodal input from the front end, a corresponding intelligent agent is matched through a dynamic processor. Then, based on the text content generated by the intelligent agent, a text segment to be synthesized is obtained using a streaming text slicing method. Finally, the text segment and the audio synthesized by speech are returned to the front end in real time. Through the synergy of streaming parallelism, semantic-aware slicing, generation-playback synchronization, and multimodal adaptive strategies, latency is reduced while ensuring natural playback, and it outperforms traditional solutions in terms of robustness and scene generalization. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a schematic diagram of the system architecture of the multimodal interaction method in the embodiments of this application;

[0048] Figure 2 This is a flowchart illustrating the multimodal interaction method in the embodiments of this application;

[0049] Figure 3 This is a schematic diagram of the structure of the multimodal interaction device in the embodiments of this application;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The technical terms used in this application are as follows:

[0053] ASR (Automatic Speech Recognition): Speech recognition refers to a technology module that converts user speech into text information in real time to capture user intent input.

[0054] TTS (Text-To-Speech): Speech synthesis refers to the output module that converts text content into natural human voice. This system achieves high-naturalness speech synthesis based on the CosyVoice deep learning model.

[0055] WebSocket: A network protocol that supports full-duplex communication for real-time audio data interaction between the front end and the speech engine, the speech engine and the ASR module, and the speech engine and the TTS module.

[0056] SSE (Server-Sent Events): A one-way server push mechanism used for streaming text communication between the voice engine and the agent.

[0057] Agent: refers to an intelligent agent with specific business logic and dialogue capabilities, responsible for handling semantic analysis, task planning, and response generation.

[0058] The inventors discovered that existing mainstream voice interaction systems are typically implemented based on the following technical architecture:

[0059] (1) Speech Recognition Module (ASR): Uses acoustic and language models to convert speech signals into text. Some systems support real-time recognition, but the output is still segmented.

[0060] (2) Dialogue module: It is usually based on deep learning or large language model to generate complete response text, and generally uses sentences or paragraphs as the smallest output unit.

[0061] (3) Text-to-Speech (TTS) module: responsible for converting text into playable speech. Usually, it is necessary to wait for the complete text input before synthesizing and playing it at once.

[0062] Based on communication mechanisms: Existing systems mostly use HTTP long connections or WebSockets to achieve communication between the Agent and the front end. Some systems also introduce Server-Sent Events (SSE) to support streaming, but this is usually only used for text-level display and has not yet been integrated with TTS broadcasting.

[0063] Within the existing framework, voice interaction only supports a single input mode (voice or text), making flexible switching between voice and text input impossible. Furthermore, the interaction flow is "complete generation—complete synthesis—complete playback," resulting in typically high feedback latency. Additionally, it cannot flexibly switch between different types of agents or processing logic based on different business scenarios, leading to poor system scalability and a subpar user experience.

[0064] There are also voice interaction systems based on SSE streaming. This solution utilizes the SSE mechanism to progressively push the results generated by a large model, enabling real-time display of text content. Its basic steps are as follows:

[0065] User voice input is recognized as text by ASR; the recognition result is sent to the large model; the large model generates response text and gradually returns it through SSE streaming; the front end displays the text content in real time, and after the complete text is received, the TTS module is called to synthesize speech.

[0066] With the development of voice interaction and large model technology, traditional voice systems generally adopt a serial processing structure of "speech recognition - text generation - speech synthesis". However, through research and analysis of existing solutions, the following shortcomings can be deduced:

[0067] (1) Large response delay and insufficient real-time performance. In general, traditional systems transmit the complete text to the TTS module for speech synthesis and playback only after the large model has generated it. Due to the lack of a streaming transmission mechanism, the system has to wait for the text generation to finish before it can start the broadcast, resulting in a significant speech response delay and making it impossible to achieve natural and smooth real-time interaction.

[0068] (2) The text slicing method is too simple and lacks semantic understanding. Existing technologies mostly use fixed length, punctuation marks or time intervals as the basis for text slicing, without combining semantic structure for intelligent analysis. This "static slicing" method is prone to semantic breaks, context confusion or incoherent tone, thereby reducing the naturalness of voice broadcasting and user experience.

[0069] (3) Lack of synchronization control mechanism between Agent output and TTS broadcast. In the traditional architecture, there is no effective synchronization signal between the text stream output by the Agent and the TTS broadcast module, which makes it impossible for the system to distinguish between the segment that should be broadcast immediately and the context that is still being generated, resulting in a mismatch between the broadcast and output rhythm, which is particularly prominent in asynchronous generation scenarios.

[0070] (4) Insufficient multimodal input fusion capability. Current interactive systems typically only support a single input mode (voice or text), and cannot achieve flexible switching between voice and text input, nor can they achieve parallel processing and unified management of voice and text, thus limiting the application of the system in multiple scenarios.

[0071] (5) The system's adaptive capability is insufficient, making it difficult to meet the needs of multiple scenarios. Existing solutions generally adopt fixed parameter slicing and broadcasting strategies, which cannot be dynamically adjusted according to scenario type (such as bank tutoring, intelligent customer service), semantic complexity or network latency, thus limiting the system's generalization and flexibility.

[0072] In summary, existing voice interaction systems have significant shortcomings in terms of real-time performance, semantic integrity, control synchronization, multimodal input fusion, and adaptive capabilities, making it difficult to meet the needs of banks in highly interactive and multimodal scenarios.

[0073] To address the aforementioned shortcomings, this application provides a multimodal interaction method, the overall goal of which is to construct a system architecture that can achieve low latency, strong semantics, collaboration, and adaptability in multimodal voice interaction.

[0074] Specifically, the present invention aims to achieve the following technical objectives:

[0075] (1) Improve end-to-end response real-time performance. By introducing an SSE-based streaming transmission mechanism, the text generated by the Agent can be transmitted while it is being generated. Combined with streaming TTS broadcasting, low-latency voice feedback of "generation and broadcasting" can be achieved.

[0076] (2) Realize intelligent slicing based on control tags and semantic analysis. A hybrid slicing method combining control tags (e.g., [SEND], [DONE]) and semantic rules is proposed. This method can be driven by the Agent output to force slicing, and can also automatically determine the appropriate slice boundaries based on semantic integrity, thereby ensuring natural and coherent broadcasting.

[0077] (3) Achieve synchronous control of Agent output and TTS playback. Construct a cross-module "text slicing control mechanism" to establish a collaborative channel between Agent generation and TTS playback, enabling the system to dynamically trigger speech synthesis and playback according to the rhythm of Agent output, thus achieving true generation-playback synchronization.

[0078] (4) Construct a unified multimodal input layer to support the dynamic fusion of speech recognition (ASR) input, text input and speech-to-text input, and meet the needs of multiple scenarios such as intelligent tutoring, voice assistant and customer service;

[0079] (5) Implement multi-scenario adaptive broadcasting strategy. Through the dynamic threshold and strategy adjustment module, the system can adjust the slice length, broadcasting rate and pause interval in real time according to semantic complexity, network conditions and application scenarios (such as bank tutoring or embody robot interaction), thereby improving the system's adaptability and intelligence.

[0080] In summary, the purpose of this application is to construct a real-time multimodal voice interaction system with streaming capabilities, a semantic-aware slicing mechanism, a synchronous control strategy, and multi-scenario adaptive capabilities, fundamentally solving problems such as delayed voice feedback, rigid semantic slicing, and asynchronous interaction rhythm in traditional systems, and significantly improving the naturalness and real-time experience of intelligent voice interaction.

[0081] The technical solutions provided in the various embodiments of this application are described in detail below with reference to the accompanying drawings. Through a WebSocket full-duplex connection, bidirectional real-time transmission of audio and text streams is achieved, significantly reducing interaction latency. Each module has an anomaly recovery mechanism to ensure continuity of interaction in complex network environments. The modular design allows for independent upgrades and replacements of speech synthesis, speech recognition, and intelligent agent business logic, facilitating system iteration.

[0082] like Figure 1 The diagram shown is a system architecture diagram of the multimodal interaction method in this application embodiment. It is mainly divided into five core parts: front-end, speech engine, speech synthesis module, speech recognition module and intelligent agent module. The collaboration between modules is achieved through different communication protocols.

[0083] 1. The front-end (user interaction entry point) is responsible for direct interaction with the user and is divided into two sub-modules:

[0084] Text interaction includes: text input: receiving text content input by the user; message sending: sending text or speech-transcribed content to the backend; result display: presenting the text or synthesized speech results returned by the agent to the user; and state synchronization: maintaining consistency with the backend's interactive state.

[0085] Voice Interaction: Audio Input: Capture user voice via, for example, a microphone. Audio Transmission: Transmit audio data to the speech engine. Audio Playback: Play the audio returned by the speech synthesis module. State Synchronization: Maintain consistency with the backend's voice interaction state.

[0086] 2. The voice engine (core scheduling hub), as the core control layer of the system, is responsible for coordinating the entire interaction process and includes three sub-modules:

[0087] Session Management: Handles voice / text input with standardized processing. Provides session lifecycle management including initialization, playback feedback, and hang-up control. Voice Management: Task scheduling, text management, and voice management. Activity detection, intelligent slicing, and streaming synchronization optimize voice processing efficiency. Synchronization Control and Task Scheduling: Event scheduling, status tracking, and exception handling ensure stable and coordinated operation of all modules.

[0088] 3. The text-to-speech (TTS) module converts text into speech, providing real-time synthesis and streaming playback. Multi-tone tone correction and anomaly recovery mechanisms ensure natural and stable speech output.

[0089] 4. The speech recognition module (ASR) converts speech into text, providing: speech transcription, streaming output, multi-source recognition, anomaly recovery, and support for speech input in complex scenarios.

[0090] 5. The Intelligent Agent module (core of business logic) handles business logic and knowledge reasoning, providing: process orchestration, multi-turn question answering, tool invocation, and knowledge base access to enable complex dialogues and task execution.

[0091] Specifically, the data flow process includes the entire interaction process, which is divided into two main lines based on the input method: text interaction and voice interaction.

[0092] (1) Text interaction process

[0093] Step S1. The user enters text on the front end.

[0094] Step S2. The front end sends the text message to the speech engine's session management module via a WebSocket long connection.

[0095] Step S3. The voice engine's synchronization control and task scheduling module sends text to the intelligent agent module via HTTP hypertext transfer request.

[0096] Step S4. The agent module performs process orchestration, multi-round question and answer or tool calls, and generates response text.

[0097] Step S5. The agent returns the response text to the speech engine in the form of streaming text.

[0098] Step S6. The speech engine pushes the response text back to the front end via WebSocket, and the front end displays the results.

[0099] (2) Voice interaction process

[0100] Step S1. The user inputs audio on the front end, and the front end transmits the audio data to the speech engine's session management module via WebSocket.

[0101] Step S2. The voice engine sends audio data to the voice recognition module via WebSocket.

[0102] Step S3. The speech recognition module performs speech transcription, generates text, and returns it to the speech engine in the form of streaming output.

[0103] Step S4. The speech engine sends the transcribed text to the agent module via an HTTP request.

[0104] Step S5. After processing, the agent module returns the response text to the speech engine in the form of streaming text.

[0105] Step S6. The speech engine sends the response text to the speech synthesis module via WebSocket.

[0106] Step S7. The speech synthesis module converts the text into audio and performs real-time synthesis or streaming.

[0107] Step S8. The synthesized audio is returned to the speech engine via WebSocket, and then pushed to the front end for audio playback by the speech engine.

[0108] (3) Communication method

[0109] Full-duplex long-connection communication (WebSocket): Used for real-time interaction between the front-end and the speech engine, and between the speech engine and the speech synthesis / recognition module, supporting bidirectional transmission of audio and text streams. Hypertext Transfer Request (HTTP): Used for text data exchange between the speech engine and the agent module, suitable for non-real-time business logic processing. Implements synchronized control of Agent output and TTS playback. Constructs a cross-module text slicing control mechanism, establishing a collaborative channel between Agent generation and TTS playback, enabling the system to dynamically trigger speech synthesis and playback according to the Agent output rhythm, achieving true generation-playback synchronization.

[0110] like Figure 1 The multimodal interaction system shown includes: a front-end module for receiving user text or voice input and displaying or playing the interaction results; a voice engine module, serving as the system's core scheduling hub, which interacts with the front-end, voice synthesis module, and voice recognition module in real time via a full-duplex long-duplex communication protocol, and interacts with the agent module for business data via a hypertext transfer protocol; a voice synthesis module for converting text data into a voice stream and pushing the voice stream to the voice engine via a full-duplex long-duplex communication protocol; a voice recognition module for converting the voice stream into text data and pushing the text data to the voice engine via a full-duplex long-duplex communication protocol; and an agent module for performing business logic processing based on the input text, generating response text, and pushing the response text to the voice engine via a hypertext transfer protocol. Real-time bidirectional communication between the front-end and back-end, and between the voice engine and the synthesis / recognition modules, is achieved through a WebSocket full-duplex connection, while efficient processing of business logic is achieved through the HTTP protocol, thereby improving the smoothness and response speed of the interaction.

[0111] This application provides a multimodal interaction method, such as... Figure 2 The diagram shows a flowchart of a multimodal interaction method in an embodiment of this application. The method includes at least the following steps S210 to S230:

[0112] Step S210: In response to the front-end multimodal input, the corresponding agent is matched through the dynamic processor.

[0113] The front-end multimodal input serves as the system's unified input layer, receiving and integrating various input formats to ensure natural and continuous input processing across different interaction scenarios. A dynamic processor selects the optimal processing logic and corresponding agent based on the input type or business scenario, ensuring efficient response and optimal resource allocation in multimodal and multi-tasking environments.

[0114] It's understandable that multimodal input includes text, speech, and speech-to-text conversion. Dynamic processors have routing capabilities, selecting appropriate processing logic and agents based on the input type and business scenario.

[0115] Step S220: Based on the text content generated by the intelligent agent, obtain the text segment to be synthesized by using a streaming text slicing method.

[0116] The large-scale agent, based on a large language model (LLM), can perform semantic understanding, intent recognition, and natural language generation on user input. Furthermore, it enables intelligent decision-making and streaming text generation in end-to-end real-time multimodal interaction, a key aspect in enhancing the system's naturalness and real-time performance. Streaming text slicing further segments the text generated by the agent in real time, ensuring semantic integrity and supporting synchronized playback with the TTS module. This module employs a hybrid slicing strategy based on control tags and semantic analysis, balancing real-time generation-playback with semantic coherence.

[0117] Step S230: The text fragment and the audio synthesized from speech are returned to the front end in real time.

[0118] The text segments output by the large model are synthesized into a speech stream in real time and returned to the front end for display.

[0119] The method in this application employs an end-to-end streaming interaction mechanism of ASR (Speech Recognition) - Agent - TTS (Speech Synthesis). By establishing segmented transmission and parallel processing links between speech recognition, Agent generation, and speech synthesis, a low-latency speech response process is achieved, enabling simultaneous recognition, generation, and playback. This mechanism achieves millisecond-level real-time text-to-speech conversion through asynchronous streaming event control. This significantly reduces overall response latency and enables near real-time natural speech dialogue.

[0120] The method described in this application constructs a cross-module text slicing control and synchronization mechanism. A collaborative control channel is established between Agent generation and TTS (text-to-speech) playback, dynamically triggering speech synthesis and playback based on the generation rhythm and control signals output by the Agent, achieving precise synchronization of "generating a segment and playing a segment." This mechanism effectively solves the problems of asynchronous generation and playback, and speech repetition or lag in traditional systems, achieving true collaborative generation and playback.

[0121] The method described in this application constructs a unified multimodal input system that supports the fusion processing of multiple information types, including voice input (ASR recognition results), text input, and speech-to-text input. Furthermore, it achieves intelligent adaptation and dynamic optimization for different application scenarios, enhancing the system's generalization ability and user experience consistency. Employing a state tracking and caching mechanism, it can automatically resume unfinished text playback when the TTS or ASR module malfunctions or the network is interrupted, ensuring the continuity and stability of the interaction process.

[0122] Unlike related technologies that suffer from large response latency and insufficient real-time performance, the above method achieves end-to-end low-latency response. Through the end-to-end streaming processing and fragmented parallel transmission mechanism of ASR-Agent-TTS, it achieves "recognition, generation, and broadcasting simultaneously," significantly reducing system response latency and meeting near real-time voice interaction requirements.

[0123] Unlike related technologies that lack a synchronization control mechanism between Agent output and TTS broadcasting, the method described above achieves synchronization between generation and broadcasting. Through synchronization control and task scheduling mechanisms, Agent generation progress is strictly synchronized with TTS broadcasting, resolving the issues of asynchronous, repetitive, or delayed generation and broadcasting in traditional systems, thus improving the interactive experience.

[0124] Unlike related technologies that suffer from insufficient multimodal input fusion capabilities, the above method supports unified processing of voice input, text input, and speech-to-text input. Combined with dynamic processor selection and strategy scheduling, it enables adaptive responses for various application scenarios such as intelligent tutoring, customer service, and voice assistants.

[0125] In one embodiment of this application, the response to front-end multimodal input includes: receiving a real-time audio stream from the front end via a voice input interface, calling an automatic speech recognition (ASR) engine for streaming recognition, and converting the voice signal into segmented text data in real time; receiving plain text data input by the user directly via HTTP or WebSocket via a text input interface; processing a pre-recorded voice file via a speech-to-text interface and converting it into text via a speech recognition module; and uniformly formatting and standardizing the text results from the different channels to form a unified text stream input.

[0126] The voice input interface receives real-time audio streams from the front end (transmitted via WebSocket) and calls the Automatic Speech Recognition (ASR) engine for streaming recognition, converting the speech signal into segmented text data in real time. The text input interface supports receiving plain text data directly from the user via HTTP or WebSocket, suitable for text-based interaction scenarios. The speech-to-text interface processes pre-recorded voice files, converting them into text through the speech recognition module, facilitating offline or playback-based input scenarios. Furthermore, through input fusion, text results from different channels are uniformly formatted and standardized to form a unified text stream input, providing a consistent input source for subsequent intelligent slicing, semantic analysis, and speech synthesis modules.

[0127] Through multimodal input, dynamic fusion of voice, text, and voice-to-text input is achieved, significantly improving the flexibility of input methods and cross-scenario adaptability.

[0128] In one embodiment of this application, the step of matching the corresponding intelligent agent through a dynamic processor includes: classifying and identifying the input data to determine the input type, identifying the business scenario type in conjunction with a preset business scenario, and selecting the optimal processing logic and the corresponding intelligent agent instance based on the input type and the business scenario.

[0129] The dynamic processor is used to select the optimal processing logic and corresponding agent based on the input type or business scenario, ensuring efficient response and optimal resource allocation of the system in a multimodal and multi-task environment.

[0130] Input type recognition: Classifies and identifies input data to determine its source type (speech-to-text recognition, direct text input, or speech-to-text result). Scene recognition: Identifies the business scene type based on preset business scenarios (such as intelligent tutoring, customer service, voice assistant, etc.). Dynamic agent scheduling: Selects the corresponding processing logic and Agent instance based on input type and scene characteristics, enabling flexible invocation and task routing. Policy management: Maintains the mapping relationship between input type, scene, and Agent, and supports dynamic policy updates to ensure rapid integration of new scenes or new Agents.

[0131] The technical implementation scheme of this module is as follows: In terms of system implementation, the dynamic processor selection module can achieve dynamic selection through the AgentFactory pattern: (1) Agent Factory Pattern (AgentFactory): The system defines a unified interface AgentHandler, and all specific types of Agents implement this interface. Through the factory class AgentFactory, the corresponding AgentHandler instance is dynamically returned according to the Agent type in the user request (chatRequest.getAgentType()). (2) Operation mechanism: The multimodal input layer receives user input and passes it to the dynamic processor selection module; the module determines the required Agent type through input type and scenario recognition; calls AgentFactory.getHandler(chatRequest.getAgentType()) to obtain the corresponding AgentHandler instance; executes business logic through the returned AgentHandler, including text slicing, semantic understanding, TTS triggering, etc.; when the system adds a new Agent type or business scenario, it only needs to register a new AgentHandler implementation class in the factory, without modifying the original system logic, thus achieving high scalability.

[0132] By matching the corresponding intelligent agent with a dynamic processor, multimodal and cross-scenario agent dynamic scheduling can be achieved, providing core support for low-latency streaming interaction, text slicing and broadcast control of the entire system architecture.

[0133] In one embodiment of this application, the intelligent agent is implemented based on a Large Language Model (LLM). The operations performed by the intelligent agent include: semantic understanding of user input and parsing the semantic information of the text corresponding to the multimodal input; dynamically inferring the user's task objective based on the user's speech or text content to complete intent recognition; retaining session history information through a semantic context caching mechanism to ensure the continuity of multi-turn interaction context, while dynamically adjusting the generation strategy according to the context; and generating response text and corresponding control tags based on the semantic parsing and intent recognition results to complete natural language generation.

[0134] The Agent is used in real-time interactive systems to conduct voice and text interactions. The Large Model Agent is the core intelligent processing unit of the system, based on a Large Language Model (LLM), responsible for semantic understanding, intent recognition, and natural language generation of user input. This module enables intelligent decision-making and streaming text generation in end-to-end real-time multimodal interaction, and is a key component in improving the system's naturalness and real-time performance. It mainly includes the following capabilities:

[0135] Semantic understanding and dynamic intent recognition: Semantic parsing of text information provided by the multimodal input module; dynamic inference of task objectives based on user voice or text content, such as bank consultation, intelligent tutoring, financial advice, etc.; support for context-related reasoning to understand implicit intents in multi-turn dialogues.

[0136] Multi-turn dialogue context management: Retains conversation history information through a semantic context caching mechanism; ensures the continuity of context in multi-turn interactions, avoiding semantic breaks or misaligned responses; can dynamically adjust the generation strategy according to the context to achieve coherent and natural interaction.

[0137] Natural Language Generation: Generates response text based on the parsing results, supports streaming output, and generates control tags (such as [SEND], [DONE]) to ensure semantic integrity.

[0138] It's understandable that large-model agent modules can be implemented in various ways, ensuring compatibility with different platforms. One approach is to leverage the Dify low-code platform to build agents, integrating large language models with the dialogue flow; supporting API interfaces to receive text input and streaming results via SSE; and configuring multi-turn context management and dynamic intent recognition strategies to achieve intelligent dialogue.

[0139] The above methods improve end-to-end real-time response. By introducing a streaming mechanism based on SSE, the text generated by the agent can be transmitted while it is being generated. Combined with streaming TTS broadcasting, low-latency voice feedback of "generation as broadcasting" is achieved. By constructing a unified multimodal input, the system supports the dynamic fusion of speech recognition (ASR) input, text input, and speech-to-text input, meeting the needs of multiple scenarios such as intelligent tutoring, voice assistants, and customer service. Multi-scenario adaptive broadcasting strategies are implemented. Through a dynamic threshold and strategy adjustment module, the system can adjust the slice length, broadcasting rate, and pause interval in real time according to semantic complexity, network conditions, and application scenarios, improving the system's adaptability and intelligence.

[0140] In one embodiment of this application, obtaining the text segment to be synthesized by using a streaming text slicing method based on the text content generated by the intelligent agent includes: real-time segmentation of the text stream output by the intelligent agent to ensure the semantic integrity of the text and adapt to the synchronous broadcasting requirements of text-to-speech (TTS); driving slicing through control tags and determining whether to immediately send the buffer based on a dynamic threshold; driving slicing through control tags includes: when an explicit control tag appears in the text stream, the text is divided into pre-tag text, control tag, and post-tag text; the pre-tag text is filtered and appended to the buffer, and the buffer content is immediately sent to the TTS; the control tag is used to indicate immediate broadcasting or end of the current broadcast, and the post-tag text is put back into the buffer to wait for subsequent processing; determining whether to immediately send the buffer based on a dynamic threshold includes: the dynamic threshold is dynamically adjusted according to the text type, semantic complexity, network latency, and system load; if the text in the buffer reaches the minimum segment length and encounters a punctuation mark, it is sent immediately; if the conditions are not met, the text continues to accumulate and waits for the next round of slicing.

[0141] Streaming text slicing is used to segment text streams generated by large models in real time to ensure semantic integrity and support synchronized playback with the TTS module. This module employs a hybrid slicing strategy based on control tags and semantic analysis, balancing real-time generation and playback with semantic coherence. Core functionalities include:

[0142] Control flags drive slicing, specifically:

[0143] When explicit control markers (such as [SEND] or [DONE]) appear in the text, the module divides the text into three parts according to the marker: the text before the marker, the marker itself, and the text after the marker.

[0144] The text before the tag is filtered and appended to the buffer; the contents of the buffer are immediately refreshed and sent to TTS; then the tagging command is sent: [SEND] indicates immediate broadcast (fixed text script), [DONE] indicates end of this round of broadcasting; the text after the tag is put back into the buffer, waiting for the next round of processing.

[0145] Semantic integrity slicing, specifically: For plain text without control markers, the module first performs text cleaning, removing empty content or invalid characters. It then determines whether to send immediately based on a dynamic threshold `ttsTextMinLength`: if the minimum slice length is reached and punctuation is encountered, the buffer content is sent; otherwise, text continues to accumulate, waiting for the next round of slicing. Punctuation marks serve as slice boundaries, ensuring that the sliced ​​text is as semantically complete as possible.

[0146] Dynamic threshold adjustment dynamically adjusts the slice length and transmission strategy based on text type, semantic complexity, network latency, and system load to balance real-time performance and natural playback. Specifically, it employs a streaming text slicing method to obtain the text segments to be synthesized, including: text stream reception: receiving the real-time generated text stream from the agent module; buffer management: maintaining a text buffer and triggering slices based on control markers or punctuation; slice judgment and transmission: determining slice boundaries by combining control markers, semantic analysis, and dynamic thresholds, and sending the sliced ​​text to the TTS module for streaming playback; and generation-playback synchronization: ensuring that TTS playback is triggered immediately after each slice is generated, achieving low-latency synchronous interaction of "generating a slice and playing a slice."

[0147] This paper proposes a hybrid slicing strategy based on control tags and semantic analysis for real-time segmentation of streaming text output from large models. The system comprehensively considers semantic boundaries, syntactic structure, and explicit tags (such as [SEND], [DONE]) for slicing, avoiding semantic breaks caused by traditional fixed-length or punctuation-based slicing methods, thus improving the naturalness and contextual coherence of TTS playback. The hybrid text slicing strategy based on control tags and semantic analysis ensures that each text segment is as semantically complete as possible, avoiding semantic breaks and achieving natural and coherent speech output. This method achieves intelligent slicing based on control tags and semantic analysis. A hybrid slicing method combining control tags (such as [SEND], [DONE]) and semantic rules is proposed, which can be driven by agent output to force slicing, and can also automatically determine appropriate slice boundaries based on semantic integrity, thereby ensuring natural and coherent playback.

[0148] In one embodiment of this application, the step of returning the text fragment and the synthesized audio to the front end in real time includes: deploying the text-to-speech (TTS) engine as a microservice of a multilingual speech synthesis system based on a large language model, receiving the text fragment via the WebSocket protocol; and having the TTS engine generate an audio stream and play it simultaneously based on the text fragment, and output it to the front end synchronously.

[0149] Text slices are converted into speech and read aloud in real time through speech synthesis, supporting streaming processing to achieve low latency and natural speech output. Specifically, this includes:

[0150] Streaming text-to-speech conversion: Receives text slices, generates and reads them simultaneously, and outputs an audio stream; Highly natural speech generation: Based on the CosyVoice speech synthesis engine, it supports multiple languages, multiple timbres, and adjustable speech rate and pauses to achieve natural and fluent human voice synthesis; Microservice deployment and robustness: The TTS engine is deployed as a CosyVoice microservice, receiving text slices via WebSocket and supporting exception caching and recovery.

[0151] In one embodiment of this application, the step of returning the text fragment and the audio synthesized by speech to the front end in real time further includes: based on the publish-subscribe model, realizing inter-module communication through an event bus, listening to the intelligent agent output event and the text-to-speech (TTS) status event, and triggering corresponding linkage operations; tracking the progress of intelligent agent text generation and the TTS broadcast status in real time, maintaining synchronization signals between modules, and ensuring interaction consistency.

[0152] The synchronization control and task scheduling blocks coordinate Agent generation and TTS broadcasting to ensure timing synchronization and robustness. Specifically, this includes an event scheduler: listening to Agent output events and TTS status events and triggering corresponding operations (such as starting broadcasting); a status tracker: tracking Agent generation progress and TTS broadcasting status in real time and maintaining synchronization signals; and an exception handler: detecting module exceptions (such as network interruptions) and triggering recovery mechanisms.

[0153] Specifically, the implementation scheme is as follows: The event scheduler is based on a publish-subscribe pattern and uses an event bus (such as RedisPub or Sub) for inter-module communication. The state tracker uses a shared state machine to record the generation and broadcasting status of each text slice. The exception handler identifies exceptions through a timeout mechanism and invokes the cache recovery process.

[0154] This application also provides a multimodal interaction device 300, such as... Figure 3 As shown, a structural schematic diagram of a multimodal interaction device in an embodiment of this application is provided. The multimodal interaction device 300 includes at least: a response matching module 310, a slicing module 320, and a feedback module 330, wherein:

[0155] In one embodiment of this application, the response matching module 310 is specifically used to: in response to front-end multimodal input, match the corresponding intelligent agent through a dynamic processor.

[0156] The front-end multimodal input serves as the system's unified input layer, receiving and integrating various input formats to ensure natural and continuous input processing across different interaction scenarios. A dynamic processor selects the optimal processing logic and corresponding agent based on the input type or business scenario, ensuring efficient response and optimal resource allocation in multimodal and multi-tasking environments.

[0157] It's understandable that multimodal input includes text, speech, and speech-to-text conversion. Dynamic processors have routing capabilities, selecting appropriate processing logic and agents based on the input type and business scenario.

[0158] In one embodiment of this application, the slicing module 320 is specifically used to: obtain the text segment to be synthesized by using a streaming text slicing method based on the text content generated by the intelligent agent.

[0159] The large-scale agent, based on a large language model (LLM), can perform semantic understanding, intent recognition, and natural language generation on user input. Furthermore, it enables intelligent decision-making and streaming text generation in end-to-end real-time multimodal interaction, a key aspect in enhancing the system's naturalness and real-time performance. Streaming text slicing further segments the text generated by the agent in real time, ensuring semantic integrity and supporting synchronized playback with the TTS module. This module employs a hybrid slicing strategy based on control tags and semantic analysis, balancing real-time generation-playback with semantic coherence.

[0160] In one embodiment of this application, the feedback module 330 is specifically used to: return the text fragment and the audio synthesized from speech to the front end in real time.

[0161] The text segments output by the large model are synthesized into a speech stream in real time and returned to the front end for display.

[0162] In one embodiment of this application, the response matching module 310 is further configured to:

[0163] Based on the voice input interface, it receives real-time audio streams from the front end, calls the Automatic Speech Recognition (ASR) engine for streaming recognition, and converts the voice signal into segmented text data in real time.

[0164] Based on the text input interface, plain text data input by the user can be received directly via HTTP or WebSocket.

[0165] Based on the speech-to-text interface, pre-recorded speech files are processed and converted into text by the speech recognition module;

[0166] The text results from the different channels are formatted and standardized in a unified manner to form a unified text stream input.

[0167] In one embodiment of this application, the response matching module 310 is further configured to:

[0168] Classify and identify the input data to determine the input type, and combine this with a preset business scenario to identify the business scenario type.

[0169] Based on the input type and business scenario, select the optimal processing logic and the corresponding intelligent agent instance.

[0170] In one embodiment of this application, the slicing module 320 is further configured to:

[0171] Perform semantic understanding on user input and parse the semantic information of the text corresponding to multimodal input;

[0172] Dynamically infer the user's task objective based on the user's voice or text content to complete intent recognition;

[0173] The semantic context caching mechanism retains session history information, ensuring the continuity of multi-round interaction context, and dynamically adjusts the generation strategy according to the context.

[0174] Based on the results of semantic parsing and intent recognition, response text and corresponding control tags are generated to complete natural language generation.

[0175] In one embodiment of this application, the slicing module 320 is further configured to:

[0176] The text stream output by the intelligent agent is segmented in real time to ensure the semantic integrity of the text and adapt to the requirements of simultaneous text-to-speech (TTS) broadcasting.

[0177] By controlling the flag-driven slice, a dynamic threshold is used to determine whether to immediately send the buffer.

[0178] The step of driving the slice via control markers includes:

[0179] When explicit control tags appear in a text stream, the text is divided into text before the tag, text after the control tag, and text after the tag.

[0180] The text before tagging is filtered and appended to the buffer, and the contents of the buffer are sent to the text-to-speech (TTS) service in real time.

[0181] Control flags are used to indicate whether to broadcast immediately or end the current round of broadcasting. After being flagged, the text is put back into the buffer to await further processing.

[0182] The step of determining whether to immediately send the buffer based on a dynamic threshold includes:

[0183] The dynamic threshold is dynamically adjusted based on text type, semantic complexity, network latency, and system load. If the text in the buffer reaches the minimum fragment length and encounters a punctuation mark, it will be sent immediately.

[0184] If the conditions are not met, the text continues to accumulate and waits for the next round of slicing.

[0185] In one embodiment of this application, the feedback module 330 is further configured to:

[0186] The text-to-speech (TTS) engine is deployed as a microservice of a multilingual speech synthesis system based on a large language model, and the text fragments are received via the WebSocket protocol.

[0187] The text-to-speech (TTS) engine generates an audio stream and plays it simultaneously from the text fragment, outputting it to the front end in a synchronized manner.

[0188] In one embodiment of this application, a synchronization control and task scheduling module is further included.

[0189] Based on the publish-subscribe model, communication between modules is realized through an event bus, which listens for intelligent agent output events and text-to-speech (TTS) status events and triggers corresponding linkage operations.

[0190] Real-time tracking of the agent's text generation progress and text-to-speech (TTS) broadcast status, maintenance of synchronization signals between modules, and ensuring consistent interaction.

[0191] It is understood that the above-mentioned multimodal interaction device can realize each step of the multimodal interaction method provided in the foregoing embodiments. The relevant explanations of the multimodal interaction method are applicable to the multimodal interaction device, and will not be repeated here.

[0192] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0193] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0194] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0195] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a multimodal interactive device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0196] In response to multimodal input from the front end, the corresponding intelligent agent is matched through a dynamic processor;

[0197] Based on the text content generated by the intelligent agent, the text segment to be synthesized is obtained by using a streaming text slicing method;

[0198] The text fragment and the synthesized audio are returned to the front end in real time.

[0199] The above is as stated in this application. Figure 2 The methods executed by the multimodal interaction device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0200] The electronic device can also perform Figure 2 A method for executing multimodal interaction devices, and implementation of multimodal interaction devices in... Figure 2 The functions of the embodiments shown are not described again in this application.

[0201] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 2 The method executed by the multimodal interaction device in the illustrated embodiment is specifically used to perform:

[0202] In response to multimodal input from the front end, the corresponding intelligent agent is matched through a dynamic processor;

[0203] Based on the text content generated by the intelligent agent, the text segment to be synthesized is obtained by using a streaming text slicing method;

[0204] The text fragment and the synthesized audio are returned to the front end in real time.

[0205] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0207] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0209] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0210] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0211] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0212] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0213] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multimodal interaction method, characterized in that, The method includes: In response to multimodal input from the front end, the corresponding intelligent agent is matched through a dynamic processor; Based on the text content generated by the intelligent agent, the text segment to be synthesized is obtained by using a streaming text slicing method; The text fragment and the synthesized audio are returned to the front end in real time.

2. The multimodal interaction method as described in claim 1, characterized in that, The response to the front-end multimodal input includes: Based on the voice input interface, it receives real-time audio streams from the front end, calls the Automatic Speech Recognition (ASR) engine for streaming recognition, and converts the voice signal into segmented text data in real time. Based on the text input interface, plain text data input by the user can be received directly via HTTP or WebSocket. Based on the speech-to-text interface, pre-recorded speech files are processed and converted into text by the speech recognition module; The text results from the different channels are formatted and standardized in a unified manner to form a unified text stream input.

3. The multimodal interaction method as described in claim 1, characterized in that, The process of matching the corresponding agent through a dynamic processor includes: Classify and identify the input data to determine the input type, and combine this with a preset business scenario to identify the business scenario type. Based on the input type and business scenario, select the optimal processing logic and the corresponding intelligent agent instance.

4. The multimodal interaction method as described in claim 2, characterized in that, The agent is implemented based on a large language model (LLM), and the operations performed by the agent include: Perform semantic understanding on user input and parse the semantic information of the text corresponding to multimodal input; Dynamically infer the user's task objective based on the user's voice or text content to complete intent recognition; The semantic context caching mechanism retains session history information, ensuring the continuity of multi-round interaction context, and dynamically adjusts the generation strategy according to the context. Based on the results of semantic parsing and intent recognition, response text and corresponding control tags are generated to complete natural language generation.

5. The multimodal interaction method as described in claim 1, characterized in that, The step of obtaining the text segment to be synthesized by using a streaming text slicing method based on the text content generated by the intelligent agent includes: The text stream output by the intelligent agent is segmented in real time to ensure the semantic integrity of the text and adapt to the requirements of simultaneous text-to-speech (TTS) broadcasting. By controlling the flag-driven slice, a dynamic threshold is used to determine whether to immediately send the buffer. The step of driving the slice via control markers includes: When explicit control tags appear in a text stream, the text is divided into text before the tag, text after the control tag, and text after the tag. The text before tagging is filtered and appended to the buffer, and the contents of the buffer are sent to the text-to-speech (TTS) service in real time. Control flags are used to indicate whether to broadcast immediately or end the current round of broadcasting. After being flagged, the text is put back into the buffer to await further processing. The step of determining whether to immediately send the buffer based on a dynamic threshold includes: The dynamic threshold is dynamically adjusted based on text type, semantic complexity, network latency, and system load. If the text in the buffer reaches the minimum fragment length and encounters a punctuation mark, it will be sent immediately. If the conditions are not met, the text continues to accumulate and waits for the next round of slicing.

6. The multimodal interaction method as described in claim 1, characterized in that, The step of returning the text fragment and the synthesized audio to the front end in real time includes: The text-to-speech (TTS) engine is deployed as a microservice of a multilingual speech synthesis system based on a large language model, and the text fragments are received via the WebSocket protocol. The text-to-speech (TTS) engine generates an audio stream and plays it simultaneously from the text fragment, outputting it to the front end in a synchronized manner.

7. The multimodal interaction method as described in claim 1, characterized in that, The step of returning the text fragment and the synthesized audio to the front end in real time also includes: Based on the publish-subscribe model, communication between modules is realized through an event bus, which listens for intelligent agent output events and text-to-speech (TTS) status events and triggers corresponding linkage operations. Real-time tracking of the agent's text generation progress and text-to-speech (TTS) broadcast status, maintenance of synchronization signals between modules, and ensuring consistent interaction.

8. A multimodal interaction device, characterized in that, The device includes: The response matching module is used to respond to multimodal input from the front end and match the corresponding agent through a dynamic processor; The slicing module is used to obtain text segments to be synthesized from the text content generated by the intelligent agent using a streaming text slicing method. The feedback module is used to return the text fragments and the synthesized audio to the front end in real time.

9. An electronic device, characterized in that, It includes a processor and a memory; the memory stores computer-executable instructions, which, when executed by the processor, implement the multimodal interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multimodal interaction method according to any one of claims 1 to 7.

Images