A speech interaction method and system based on multi-agent orchestration

Abstract

This application relates to the field of voice interaction technology, and provides a voice interaction method and system based on multi-agent orchestration. The method includes: generating a task processing request based on user voice commands and associated voice turn identifiers; obtaining user intent through user intent recognition by a main agent based on the task processing request; determining a target sub-agent based on the processing type of the user intent; processing the task processing request through the target sub-agent to obtain a task processing result and returning it to the main agent; generating a corresponding structured block sequence by the main agent; sending multiple structured blocks to corresponding output channels according to block type and voice turn identifier; storing the processing details of the task processing request in the runtime state layer during the processing of the task processing request; and storing a snapshot of the runtime state layer at a preset write time in the checkpoint state layer. This application can improve the accuracy and efficiency of task processing.

Description

Technical Field

[0001] This application belongs to the field of voice interaction technology, and in particular relates to a voice interaction method and voice interaction system based on multi-agent orchestration. Background Technology

[0002] With the development of speech recognition, natural language processing, and artificial intelligence technologies, voice interaction systems have been widely used in smart terminals, in-vehicle devices, smart homes, and information service platforms. As application scenarios become increasingly complex, the types of tasks involved in user voice commands are constantly increasing, and different tasks may differ in terms of processing logic, required resources, and feedback formats.

[0003] Currently, user speech or text with different tasks is usually directly fed into a single large model, which then performs the same task's intent understanding, knowledge answering, and response generation. In other words, all tasks share the same reasoning and processing chain, which can easily lead to low task processing efficiency or inaccurate processing results. Summary of the Invention

[0004] This application provides a voice interaction method and system based on multi-agent orchestration, which can improve the accuracy and efficiency of processing tasks corresponding to user voice commands.

[0005] In a first aspect, embodiments of this application provide a voice interaction method based on multi-agent orchestration, including: A task processing request is generated based on the user's voice commands and the voice turn identifiers associated with the user's voice commands. The main intelligent agent identifies user intent based on task processing requests to obtain user intent. Based on the processing type of the user intent, a target sub-intelligent agent is determined from multiple specialized sub-intelligent agents. The target sub-intelligent agent processes the task processing request to obtain the task processing result, and the task processing result is returned to the main intelligent agent. The main intelligent agent generates a structured block sequence corresponding to the task processing results; the structured block sequence includes multiple structured blocks, each of which includes a block type and a speech round identifier; Based on the block type and voice round identifier, multiple structured blocks are sent to the corresponding output channels to output the task processing results through the corresponding output channels; During the processing of a task processing request, the processing details of the task processing request are stored in the runtime state layer, and a snapshot of the runtime state layer at a preset write time is stored in the checkpoint state layer; the checkpoint state layer is used to restore the processing state of the task processing request in case of an exception.

[0006] In this embodiment, a task processing request is generated based on the user's voice command and the voice turn identifier associated with the command. The main agent identifies the user's intent based on the task processing request, and determines the target sub-agent from multiple specialized sub-agents based on the processing type of the intent. The target sub-agent processes the task processing request to obtain the task processing result, which is then returned to the main agent. This enables collaboration between the main agent and specialized sub-agents, reduces the coupling of voice interaction capabilities, avoids all tasks sharing the same inference and processing chain, and improves the accuracy and efficiency of processing tasks corresponding to user voice commands. Furthermore, the main agent generates a structured block sequence corresponding to the task processing result. Based on the block type and voice turn identifier, multiple structured blocks are sent to their respective output channels to output the task processing result. This achieves a unified expression mechanism of one-time generation and multi-channel consumption, resulting in diverse response forms. During the processing of task processing requests, by storing the processing details of the task processing requests in the runtime state layer and storing a snapshot of the runtime state layer at a preset write time in the checkpoint state layer, the processing state of the task processing requests can be restored through the checkpoint state layer in case of an exception. This achieves hierarchical management of the runtime state layer and the checkpoint state layer and maintains session recovery capabilities.

[0007] Secondly, embodiments of this application provide a voice interaction system based on multi-agent orchestration, comprising: The request generation module is used to generate task processing requests based on the user's voice commands and the voice turn identifiers associated with the user's voice commands. The task processing module is used to identify user intent based on task processing requests through the main intelligent agent, obtain user intent, determine the target sub-intelligent agent from multiple professional sub-intelligent agents based on the processing type of the user intent, process the task processing request through the target sub-intelligent agent, obtain the task processing result, and return the task processing result to the main intelligent agent. The sequence generation module is used to generate a structured block sequence corresponding to the task processing result through the main intelligent agent; the structured block sequence includes multiple structured blocks, and each structured block includes a block type and a speech round identifier; The result output module is used to send multiple structured blocks to the corresponding output channels according to the block type and voice round identifier, so as to output the task processing results through the corresponding output channels; The information storage module is used to store the processing details of the task processing request to the runtime state layer during the processing of the task processing request, and to store a snapshot of the runtime state layer at a preset write time to the checkpoint state layer; the checkpoint state layer is used to restore the processing state of the task processing request in case of an exception.

[0008] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the computer device implements the voice interaction method as described in any of the first aspects above.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the voice interaction method as described in any of the first aspects above.

[0010] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the voice interaction method as described in any of the first aspects above to be executed.

[0011] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a five-layer overall architecture diagram of a voice interaction system based on multi-agent orchestration provided in an embodiment of this application; Figure 2 This is a first flowchart illustrating the voice interaction method based on multi-agent orchestration provided in an embodiment of this application; Figure 3 This is a second flowchart illustrating the voice interaction method based on multi-agent orchestration provided in an embodiment of this application; Figure 4 This is a third flowchart illustrating the voice interaction method based on multi-agent orchestration provided in the embodiments of this application; Figure 5 This is a schematic diagram of the fourth process of the voice interaction method based on multi-agent orchestration provided in the embodiments of this application; Figure 6 This is a schematic diagram of the fifth process of the voice interaction method based on multi-agent orchestration provided in the embodiments of this application; Figure 7 This is a flowchart of multi-agent collaboration, tool invocation, and dual-channel distribution of structured blocks provided in the embodiments of this application; Figure 8This is a request-level real-time voice interaction main flow and interruption control logic diagram provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of a voice interaction system based on multi-agent orchestration provided in an embodiment of this application; Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0015] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0016] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0019] In voice interaction systems designed for smart home and smart device control scenarios, the relevant technologies adopt an architecture that separates the voice access layer, orchestration service layer, and device service layer. User requests are received via WebSocket or Hypertext Transfer Protocol (HTTP), and then the device control interface or knowledge retrieval service is invoked to complete the execution, returning the results to the client in text format. This technology typically relies on a single orchestration logic, lacking mechanisms for clear division of responsibilities between the main agent and specialized sub-agents, structured response protocols, request-level tracing, strong consistency interruption control, and hierarchical state management.

[0020] The above-mentioned technologies have the following main drawbacks: 1. High coupling of capabilities, low task processing efficiency, and inaccurate processing results: All tasks are stacked in the same orchestration logic, resulting in excessively long prompts, unclear tool selection boundaries, easy mis-calls, missed calls, and difficulties in expansion and maintenance, leading to problems such as low task processing efficiency or inaccurate processing results.

[0021] 2. Limited response format: It only outputs plain text and cannot simultaneously carry different response formats in the same round of results.

[0022] 3. Insufficient low latency and interruptibility: Automatic Speech Recognition (ASR), orchestration, Text-to-Speech (TTS) and front-end rendering are separated, lacking incremental text bridging, request-level isolation and unified interrupt control mechanisms. This can easily lead to high latency of the first packet, failure to stop broadcasting in time, and old round messages being inserted into the new round of conversation.

[0023] 4. Weak session consistency and recovery capabilities: Connection states, intermediate states, and persistent history are usually not managed in a hierarchical manner, and some systems rely on digest-style context compression, which can easily lead to the loss of key tool parameters, device identifiers, and clarification conclusions, resulting in errors in subsequent rounds of execution.

[0024] Based on this, embodiments of this application provide a voice interaction method and system based on multi-agent orchestration. Its innovations are mainly reflected in: 1. A collaborative orchestration structure of main intelligent agent and specialized sub-intelligent agents is proposed. The main intelligent agent is responsible for overall scheduling and result aggregation, while specialized sub-intelligent agents such as installation, troubleshooting, and scenario are responsible for vertical domain task processing under the constraints of tool whitelist. This clarifies the capability boundaries, reduces the capability coupling of voice interaction, and improves the accuracy and efficiency of processing tasks corresponding to user voice commands.

[0025] 2. A structured block protocol is proposed, which organizes a round of response into an ordered list of blocks, so that text blocks are used for text-to-speech broadcasting and non-text blocks are used for client-side graphical rendering, thereby realizing a unified expression mechanism of "one-time generation and dual-channel consumption".

[0026] 3. A request-level tracing method based on session link identifier and voice round identifier is proposed, and combined with incremental text-to-speech bridging and a two-stage interruption judgment mechanism, to achieve low-latency response and high-reliability interruption switching.

[0027] 4. A layered management approach is proposed, including runtime state layer, checkpoint state layer, and persistent state layer. This approach can reduce the risk of losing key parameters caused by traditional digest compression while maintaining session recovery capabilities.

[0028] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0029] Please see Figure 1 , Figure 1 This paper illustrates a five-layer overall architecture diagram of a voice interaction system based on multi-agent orchestration provided in an embodiment of this application. The voice interaction system adopts a five-layer collaborative structure, which, from bottom to top, includes an infrastructure layer, a tool service layer, an agent orchestration layer, a gateway layer, and a user interaction layer.

[0030] The infrastructure layer provides support for large language models, voice services, log tracing, caching (such as Redis), and databases (such as MySQL).

[0031] The tool service layer provides device control, knowledge retrieval, and management control capabilities (such as prompt word management).

[0032] The tool service layer can use Model Context Protocol (MCP), Representational State Transfer (REST), gRPC, or equivalent protocols to complete device control, historical queries, scene operations, and knowledge retrieval. After the tool execution results are returned, the main intelligent agent will summarize them to generate the final task processing result.

[0033] The agent orchestration layer is responsible for intent recognition, main agent scheduling, specialized sub-agent handover, tool routing, and result aggregation.

[0034] The gateway layer is responsible for WebSocket access, audio stream management, voice activity detection, speech recognition, text-to-speech synthesis, protocol conversion, session management, and large language model routing.

[0035] The user interaction layer carries real-time voice streams, HTTP, voice terminals, apps, or other clients that support text / voice input / output.

[0036] Through the above structure, the voice interaction system provided in this application forms an end-to-end closed loop of "access—understanding—orchestration—execution—distribution—feedback". Unlike a single orchestration link, this application does not only focus on the large language model's response itself, but also integrates request-level identification, professional intelligent agent division of labor, tool boundary constraints, structured expression of response, voice broadcast link, client-side graphical rendering, session checkpoints and persistent auditing into the same technical system.

[0037] The aforementioned voice interaction system can be integrated into computer devices such as servers and clients. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or any of the following: cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), big data, or artificial intelligence platforms. No specific limitation is made here.

[0038] Please see Figure 2 , Figure 2 The diagram illustrates a flowchart of a voice interaction method based on multi-agent orchestration provided in an embodiment of this application. This voice interaction method is executed by a computer device, which can be a client, a server, or both. As an example and not a limitation, the application of this voice interaction method to a server is used as an example. Figure 2 The process shown is explained in detail. This voice interaction method includes the following steps: Step 201: Generate a task processing request based on the user's voice command and the voice turn identifier associated with the user's voice command.

[0039] The voice turn identifier associated with a user's voice command refers to the turn-level identifier assigned to that user's voice command within the user session associated with it. It's understood that a user session can include one or more rounds of dialogue, with each round having a voice turn identifier. One voice turn identifier corresponds to one user voice command, and one user voice command corresponds to one independent interaction. For example, in user session A (the session link identifier for this user session is A), in the first round of dialogue (voice turn identifier 001): the user says, "Turn on the living room light for me," and the computer device executes: voice recognition - understanding intent - device activation - voice response. In the second round of dialogue (voice turn identifier 002): the user says, "Turn the brightness down to 50%," and the computer device executes: voice recognition - understanding intent - device activation - voice response.

[0040] Task processing requests can refer to internal processing encapsulation built for user voice commands, which can transform user voice commands into programmable, resumable, and traceable execution units.

[0041] Step 202: The main intelligent agent identifies the user intent based on the task processing request, obtains the user intent, determines the target sub-intelligent agent from multiple specialized sub-intelligent agents based on the processing type of the user intent, processes the task processing request through the target sub-intelligent agent, obtains the task processing result, and returns the task processing result to the main intelligent agent.

[0042] In this embodiment, the task processing request can be input into the intent classifier in the main intelligent agent to identify the user intent of the task processing request and output the user intent (e.g., casual conversation, device control, installation guidance, troubleshooting, scene management, etc.). Optionally, the above-mentioned intent classifier can be a lightweight model or a rule, and this application does not limit the specific type of intent classifier.

[0043] The processing type of user intent can refer to the path label of the task processing request. Since user intent is the semantic understanding result of the task processing request, determining the target sub-agent from multiple specialized sub-agents based on the processing type of user intent can make the determined target sub-agent more suitable for the task processing request.

[0044] In this context, a specialized sub-agent can refer to a sub-processing unit that is divided according to a business domain and encapsulates the complete processing capabilities of the corresponding business domain. Each specialized sub-agent is bound to a specific business domain and is used to handle task processing requests routed to that business domain.

[0045] As an example and not a limitation, multiple specialized sub-agents may include agents corresponding to the following business domains: a device control agent corresponding to the device control business domain, an installation guidance agent corresponding to the installation guidance business domain, a fault diagnosis agent corresponding to the fault diagnosis business domain, a scene management agent corresponding to the scene management business domain, and a knowledge retrieval agent corresponding to the knowledge retrieval business domain. That is, this application can be used in business scenarios such as device control (e.g., smart home device control), installation guidance, fault diagnosis, scene management, and knowledge question answering.

[0046] The target sub-agent can refer to a specialized sub-agent selected from multiple specialized sub-agents to handle task processing requests. For example, if the task processing request is for requesting device control, the target sub-agent can be a device control agent to achieve device control.

[0047] In this embodiment, the master agent acts as the central scheduling hub. It does not directly handle all types of task processing requests. Instead, it selects the most suitable target sub-agent from multiple specialized sub-agents based on the request content. This ensures that each task is handled by the sub-agent most proficient in its domain. This reduces the risk of knowledge overload and efficiency bottlenecks caused by a single agent attempting to cover all domains, improving the accuracy and professionalism of task processing. Each specialized sub-agent only needs to focus on its own domain's logic and tools, with clear functional boundaries and mutual isolation. When it is necessary to add or optimize the processing capabilities of a certain business domain, only the corresponding specialized sub-agent needs to be adjusted or added, without modifying the master agent's core scheduling logic or other sub-agents. This significantly reduces the complexity and risk of system iteration and improves the flexibility and maintainability of the overall architecture.

[0048] Step 203: Generate a structured block sequence corresponding to the task processing result through the main intelligent agent; the structured block sequence includes multiple structured blocks, and each structured block includes a block type and a voice round identifier.

[0049] Each structured block also includes a content field, which carries the actual data to be sent. For example, if a structured block is of type text, then in addition to the text type and voice round identifier, it also includes the text to be played. Each structured block may also include at least one of the following: a content field, a block number, a timestamp, and a status field. The block number is used for sorting within the same voice round, and the status field indicates the generation, transmission, or consumption status of the structured block.

[0050] Optionally, the block type can be one or more of the following: text type, task step type, icon type, device status type, function call result type, image type, button type, list type, etc. Task step type, icon type, device status type, function call result type, image type, button type, list type, etc., are non-text types.

[0051] Step 204: Based on the block type and voice round identifier, send multiple structured blocks to the corresponding output channels to output the task processing results through the corresponding output channels.

[0052] In this embodiment, the output channels correspond to the block types, with different block types corresponding to different output channels. By organizing the task processing results into a structured block sequence carrying block types and voice round identifiers within the main agent, and distributing each structured block to its matching output channel for processing according to the block type, different block types can be output simultaneously within the same round. Furthermore, because each structured block is bound to its respective round (i.e., voice round identifier), the output has clear round affiliation and manageability (e.g., cancellation by round, sorting by round, isolation by round). This improves the clarity, controllability, and maintainability of multimodal output while ensuring the continuity of voice interaction.

[0053] It is understandable that the processes corresponding to user voice commands—including speech recognition, multi-agent orchestration, tool invocation, TTS synthesis, and client rendering—all share the same speech round identifier and are completed stream-wise on a single link, rather than being processed asynchronously in segments. By sharing the same speech round identifier and completing these processes continuously in a streaming manner within the same link, this technology enables request-level end-to-end consistency management. In other words, computer devices can accurately identify the complete processing flow of each round of voice interaction, from input, understanding, execution to output, avoiding mismatches in recognition results, tool execution results, voice broadcast content, or interface rendering content between different rounds. Because the processing steps are not executed asynchronously in segments but flow continuously around the same speech round identifier, subsequent TTS synthesis and client rendering can be triggered immediately after generating partial text or structured results, thereby reducing initial packet response latency and overall interaction latency, and improving the smoothness of real-time voice interaction.

[0054] Meanwhile, when a user initiates a new voice input before the previous round of voice broadcast has ended, the system can accurately locate and stop the TTS playback corresponding to the old round based on the voice round identifier, clear the waiting queue of the old round, and switch to the new round for processing. This avoids issues such as residual broadcasts of old voice, crosstalk, and the mixing of old round results into the new round's interface display. It also facilitates system log tracking, anomaly location, and state recovery. The recognized text, agent decisions, tool call parameters, tool return results, TTS status, and client rendering status generated by each module can all be associated with the same round, thereby improving the system's traceability, auditability, and operational reliability.

[0055] Step 205: During the processing of the task processing request, the processing details of the task processing request are stored in the runtime state layer, and a snapshot of the runtime state layer at a preset write time is stored in the checkpoint state layer; the checkpoint state layer is used to restore the processing state of the task processing request in case of an exception.

[0056] The processing details information refers to the process status data generated during the processing of a task processing request. After a single round of task processing request processing is completed (successfully or unsuccessfully), the computer device can release the processing details information of that round of task processing request from the runtime state layer. This avoids the backlog problem caused by data from completed rounds continuing to occupy the runtime state layer, and also prevents subsequent newly created rounds from mistakenly reading data from completed rounds. The preset write time refers to the pre-set time for writing a snapshot to the runtime state layer.

[0057] In some embodiments, the preset write time includes at least one of the following: after the main agent completes its corresponding task, after the target sub-agent completes its corresponding task, after generating the structured block sequence corresponding to the task processing result, before the user pauses the user session associated with the user's voice command, and before the user disconnects. Here, user disconnection can refer to the user disconnecting the connection between the client and the gateway layer in the server. Restoring the processing state of the task processing request can refer to restoring the execution context of the task processing request. This processing state can reflect the system progress at the time of snapshot generation, enabling the main agent and specialized sub-agents to continue executing subsequent logic based on this context, thereby achieving breakpoint resumption rather than restarting the entire request processing flow.

[0058] This embodiment constructs a hierarchical state management system by maintaining detailed processing information during task processing in the runtime state layer and saving the complete state as a snapshot to the checkpoint state layer at key nodes (preset write times). This design enables the system to accurately restore the processing state before the task interruption based on the snapshot saved in the checkpoint state layer when encountering abnormal situations such as unexpected network connection interruptions or service instance restarts. This ensures the continuity and reliability of complex, multi-step voice interaction tasks (such as device installation guidance and troubleshooting processes). Users do not need to repeat completed steps, and the system can seamlessly resume from the interruption point, significantly improving the success rate of long-duration interactive tasks and the user experience.

[0059] Computer devices manage the state of task processing requests in a hierarchical manner, including a runtime state layer, a checkpoint state layer, and a persistent state layer. Different state layers have different lifecycles and storage media. The runtime state layer is stored in memory and is used to save the current message list, tool call results, intermediate calculation results, and a queue of structured blocks to be consumed during a single request processing. The runtime state is released after the current request processing round is completed. The checkpoint state layer is stored in Redis (an in-memory key-value database / caching system). Preferably, AsyncRedisSaver (an asynchronous Redis state saver, a state saving component that asynchronously writes session state or process checkpoints to Redis) is used to asynchronously write state snapshots, with a predetermined lifespan, such as 24 hours. AsyncRedisSaver is used to load the most recent state snapshot based on the session link identifier when a user disconnects and reconnects, continues interaction across requests, or recovers from an abnormal service, thereby restoring the dialogue context and task execution progress.

[0060] The persistent state layer is stored in MySQL or other persistent databases to permanently store session metadata, historical dialogues, tool call records, audit logs, and data required for model tuning, supporting subsequent auditing, traceability, and model optimization. Furthermore, the checkpoint state layer does not use large language model digest compression to save the context; instead, it preserves the complete tool call chain through a secure message window, including the tool call request, call parameters, tool return results, and corresponding message relationships. This avoids the loss of critical information such as device identification, tool parameters, and clarification results caused by digest compression, ensuring state consistency in multi-turn voice interactions and the accuracy of subsequent tool calls.

[0061] In this embodiment, a task processing request is generated based on the user's voice command and the voice turn identifier associated with the command. A main agent selects a target sub-agent from multiple specialized sub-agents based on the task processing request. The target sub-agent processes the request to obtain the task processing result, which is then returned to the main agent. This enables collaboration between the main agent and specialized sub-agents, reduces the coupling of voice interaction capabilities, avoids all tasks sharing the same inference and processing chain, and improves the accuracy and efficiency of processing tasks corresponding to user voice commands. Furthermore, the main agent generates a structured block sequence corresponding to the task processing result. Based on the block type and voice turn identifier, multiple structured blocks are sent to their respective output channels to output the task processing result. This achieves a unified expression mechanism of one-time generation and multi-channel consumption, resulting in diverse response forms. During the processing of task processing requests, by storing the processing details of the task processing requests in the runtime state layer and storing a snapshot of the runtime state layer at a preset write time in the checkpoint state layer, the processing state of the task processing requests can be restored through the checkpoint state layer in case of an exception. This achieves hierarchical management of the runtime state layer and the checkpoint state layer and maintains session recovery capabilities.

[0062] In some embodiments of this application, the method may further include, before generating the task processing request: The gateway layer performs voice activity detection on the received continuous audio frames. If a user voice command is detected, the speech recognition service is invoked, and the user voice command is recognized by the speech recognition service to obtain the user's voice text. The session link identifier corresponding to the user voice command is detected. If no session link identifier corresponding to the user voice command is detected, a user session associated with the user voice command is established, and a session link identifier for the user session is generated. The voice round identifier corresponding to the user voice command is generated based on the session link identifier. Specifically, the computer device generates a task processing request based on the user's voice command and the voice turn identifier associated with the user's voice command. This can be achieved by encapsulating the user's voice text, session link identifier, voice turn identifier, context information of the user's voice command, and the current state associated with the user's session to obtain the task processing request.

[0063] The session link identifier refers to a unique identifier assigned to a user session. The session link identifier is created when the user first initiates an interaction and remains unchanged throughout the entire session lifecycle (which may include multiple rounds of question-and-answer). In this embodiment, the gateway layer can maintain a real-time connection with the client via WebSocket. After continuously receiving consecutive audio frames from the client in a voice scenario, it performs voice activity detection on the consecutive audio frames. To avoid concurrent round misalignment, a unique voice round identifier is assigned to each round of requests and is used throughout subsequent processing stages such as speech recognition, agent orchestration, tool invocation, TTS synthesis, and client rendering. Giving each round of requests a unique number (i.e., a voice round identifier) ​​ensures that all data carries this number, avoiding confusion.

[0064] In this embodiment, when a WebSocket connection is established, the client can complete authentication through an authentication mechanism (such as APIKey, OAuth token, or SwitchBot account authorization) and bind that identity to the session link identifier. When submitting a task processing request to the main agent, identity information such as user identifier and family identifier are read as session metadata from the checkpoint state layer or persistent state layer and transmitted together with the session link identifier.

[0065] In this embodiment, a voice round identifier can be generated for each round of voice interaction, and after binding it with the session link identifier, it is encapsulated together with the user's voice text, the context information of the user's voice command, and the current state associated with the user session to obtain a task processing request.

[0066] Optionally, before performing voice activity detection on continuous audio frames, the gateway layer can first perform noise suppression, echo cancellation, and other processing on the continuous audio frames to improve their quality.

[0067] In this embodiment, before request processing, the gateway layer uniformly performs voice activity detection and speech recognition, and systematically generates session link identifiers and voice turn identifiers. This encapsulates the user's voice text, session link identifiers, voice turn identifiers, instruction context information, and the current state associated with the user session into a unified task processing request, achieving end-to-end traceable request initialization. This step ensures that each round of user input is accurately associated with the correct continuous session, carrying all the historical and contextual information needed to understand the intent of this round, thus laying a solid foundation for backend agent orchestration and state management. The current state associated with the user session refers to the structured state information associated with the user session, used to maintain the continuity of multi-turn dialogues, task execution progress, and device context consistency. This current state can include one or more of the following: runtime state from the runtime state layer, checkpoint state from the checkpoint state layer, persistent state from the persistent state layer, and real-time device context.

[0068] The runtime state is stored in memory and persists throughout the lifecycle of the current request. It stores temporary data generated during the current voice round, including the current message window, the currently active agent, intermediate inference steps, and a queue of structured blocks to be consumed. The current message window includes complete dialogue messages from the most recent rounds and retains tool call requests, tool call parameters, and tool return results to prevent the loss of critical information such as device identifiers, tool parameters, or clarification results due to digest compression. The checkpoint state is stored in a high-speed cache system, such as Redis, and its lifespan can be set. It is used for recovery across requests or after a disconnection and reconnection, and includes complete dialogue history from the most recent rounds, the currently active agent identifier, device context, and user clarification records. Therefore, when a user re-initiates a request or the connection is restored, the system can read the most recent checkpoint state based on the session link identifier to restore the dialogue progress and task context. The persistent state is stored in a persistent database to permanently store session metadata, complete message records, prompt word versions, tool call records, and audit logs. Session metadata may include session link identifiers, user identifiers, family identifiers, and the currently active agent identifier. For historical content that exceeds the current message window, the system can mark its time range or compress its range, but the original message content is still stored in the persistent database to support subsequent auditing, historical backtracking, and model optimization.

[0069] Furthermore, before submitting the task processing request to the agent orchestration layer, the system can also attach real-time device context. Real-time device context includes dynamic information such as the current online status of the device, the most recent device control result, and the timestamp of voice activity detection corresponding to the user's voice input. By submitting the above status information along with the current request to the agent orchestration layer, the system can maintain contextual consistency in multi-turn voice interactions, improving task execution accuracy and anomaly recovery capabilities.

[0070] In some embodiments of this application, a target sub-agent is determined from multiple specialized sub-agents based on the processing type of the user's intent, and the task processing request is processed by the target sub-agent to obtain the task processing result, which may include steps 301 to 302.

[0071] Step 301: If the processing type of the user intent is the sub-agent distribution type, then the master agent in the agent orchestration layer determines the target sub-agent from multiple specialized sub-agents in the agent orchestration layer based on the user intent, and generates a task handover request based on the task processing request.

[0072] The processing type of user intent can include sub-agent distribution type and direct response generation type. Sub-agent distribution type refers to types that need to be transferred to a specialized sub-agent for processing. Direct response generation type refers to types where the response is directly generated by the main agent. In this embodiment, the task processing request can be trimmed and repackaged to obtain a task transfer request. For example, the data required by the target sub-agent can be extracted from the task processing request, and then this data can be repackaged to obtain the task transfer request.

[0073] As examples, not limitations, if the user intent explicitly expresses "install" or "add device," then the processing type for the user intent can be determined as the sub-agent distribution type, with the target sub-agent being the installation guidance agent; if the user intent expresses "broken" or "not working," then the processing type for the user intent can be determined as the sub-agent distribution type, with the target sub-agent being the troubleshooting agent; if the user intent expresses "scenario" or "automation," then the processing type for the user intent can be determined as the sub-agent distribution type, with the target sub-agent being the scenario management agent; if the user intent represents simple casual conversation or that sufficient information has been obtained, then the processing type for the user intent can be determined as the direct response generation type, where a response can be generated directly without transferring to a specialized sub-agent.

[0074] In some embodiments, a declarative handover tool based on LangGraph Swarm can be used to determine the target sub-agent. Specifically, a set of handover tools and a direct response tool can be injected into the main agent. The user intent is input into the agent selection model through the main agent. The agent selection model selects the tool to be invoked from the set of handover tools and the direct response tool based on the system prompt and the user intent. If the tool to be invoked is a handover tool, the professional sub-agent corresponding to that handover tool is determined as the target sub-agent, and the task processing request is handed over to the target sub-agent for processing by invoking that handover tool. If the tool to be invoked is a direct response tool, a response is directly generated by invoking the direct response tool. Here, the system prompt is used to constrain the agent selection model to autonomously decide whether to invoke the direct response tool or a handover tool based on the user intent. Optionally, the agent selection model can be a large language model, and this application does not limit the specific type of agent selection model.

[0075] Step 302: The main intelligent agent transfers the task handover request to the target sub-intelligent agent, and the target sub-intelligent agent calls the processing tool within its corresponding whitelist tool range to process the task handover request and obtain the task processing result.

[0076] In this system, each specialized sub-agent can correspond to a whitelist of tools, and the whitelist of tools for different specialized sub-agents is usually not exactly the same. The whitelist of tools can refer to the set of tools that the corresponding specialized sub-agent can call. In this embodiment, the whitelist of tools can constrain the capability boundaries of the corresponding specialized sub-agent, reduce interference from irrelevant tools and the risk of phantom calls, thereby reducing the probability of calling the wrong tools and achieving permission isolation.

[0077] In this embodiment, within the lifecycle of a single voice round identifier, the target sub-agent can reuse its connection with the processing tool through a connection manager or equivalent mechanism, avoiding the need to re-establish the connection for each call and improving the efficiency of the processing tool's invocation. If the tool call times out or returns an error, the target sub-agent can record an error log and determine its subsequent behavior based on a preset strategy (e.g., retrying once or directly generating a failure message).

[0078] In this embodiment, through the orchestration structure of the main intelligent agent and specialized sub-intelligent agents working together, the main intelligent agent is responsible for overall scheduling and result aggregation, while specialized sub-intelligent agents such as device control, installation guidance, fault diagnosis, and scene management complete vertical domain task processing under the constraints of the whitelist tool scope. This clarifies the capability boundaries of different sub-intelligent agents, decouples the determination of target sub-intelligent agents from the processing of task handover requests, and improves the accuracy and efficiency of processing tasks corresponding to user voice commands.

[0079] In this embodiment, the system processes user voice commands using a collaborative orchestration approach between a main intelligent agent and specialized sub-intelligent agents. The main intelligent agent is responsible for overall scheduling of task processing requests, agent handover, and result aggregation. Specialized sub-intelligent agents, including equipment installation agents, troubleshooting agents, and scene management agents, are used to handle tasks within their respective business domains. Each specialized sub-intelligent agent can only invoke tools within a whitelist that matches its scope of responsibility. This reduces interference from irrelevant tools on agent decision-making and lowers the risks of incorrect tool calls, prompt word overload, and permission diffusion.

[0080] In some embodiments of this application, the task handover request is handed over to the target sub-agent by the main agent, and the target sub-agent calls a processing tool within its corresponding whitelist tool scope to process the task handover request and obtain the task processing result. This may include: when the first handover request sub-block is obtained by the main agent through block-level splitting processing of the task handover request, the first handover request sub-block is handed over to the target sub-agent in real time; the main agent sequentially hands over subsequent handover request sub-blocks generated based on the task handover request to the target sub-agent until all handover request sub-blocks generated based on the task handover request are sent to the target sub-agent; the subsequent handover request sub-blocks are... A handover request sub-block is generated after the first handover request sub-block. When the target sub-agent receives the first handover request sub-block, it identifies and processes the first handover request sub-block, and then sequentially identifies and processes subsequent handover request sub-blocks to obtain a task handover request. Based on the task handover request, the target processing tool is determined from the whitelist of tools corresponding to the target sub-agent. The target sub-agent determines the parameters required to call the target processing tool based on the task handover request, and calls the target processing tool based on the required parameters to process the task handover request and obtain the task processing result.

[0081] Specifically, the target sub-agent identifies and processes the first handover request sub-block, and then sequentially identifies and processes subsequent handover request sub-blocks to obtain the task handover request. This can be achieved by the target sub-agent parsing each received handover request sub-block, extracting its payload, sub-block number, and boundary information (such as sub-block length or end marker). Upon receiving the first handover request sub-block, the target sub-agent begins constructing the initial framework of the task handover request. As subsequent handover request sub-blocks are received sequentially, for each subsequent handover request, the payload of that subsequent handover request is incrementally filled into the initial framework based on its block number and boundary information. The complete task handover request is restored synchronously when the last handover request sub-block is received.

[0082] In this embodiment, by first performing block-level traffic splitting on the task handover request on the main intelligent agent side and immediately issuing the first handover request sub-block, followed by sequentially handing over subsequent handover request sub-blocks, the handover process can be transformed from full packet handover to incremental handover. This allows the target intelligent agent to perform identification and processing immediately upon receiving the first handover request sub-block, and to restore the complete request in a streaming manner when subsequent handover request sub-blocks arrive. This reduces the waiting overhead of task handover requests and allows for finer-grained control of the already handed-over portion during network fluctuations or interruptions. Furthermore, determining the parameters required for calling the target processing tool based on the task handover request allows the parameters to be dynamically adapted based on the specific content of the task handover request, improving the accuracy and flexibility of tool invocation.

[0083] In some embodiments of this application, the target sub-agent determines the parameters required to invoke the target processing tool based on the task handover request, and invokes the target processing tool based on the parameters required to invoke the tool, which may include steps 401 to 403.

[0084] Step 401: The target sub-agent extracts the parameters required to call the target processing tool from the task handover request through the task recognition model.

[0085] It should be understood that the parameters required for invocation refer to the parameters needed to invoke the target processing tool. For example, for a device control tool, the required parameters may include the device name, control action, and control value; for a knowledge query tool, the required parameters may include the query text and the number of returned results. Since these parameters are usually obtained by the task recognition model based on user semantic parsing, there may be cases of missing parameters, incorrect types, invalid values, or unclear device names. Therefore, the required parameters should be verified before invoking the actual tool. This application does not limit the specific type of the task recognition model mentioned above. For example, the task recognition model can be a large language model such as Qwen or GPT-4o. As an example and not a limitation, if the task handover request indicates device control, the extracted required parameters may include: device name = living room light, action = turn on, brightness = 80. If the task handover request indicates knowledge query, the extracted required parameters may include: query = how to connect to a certain device, and return a maximum of 3 most relevant results.

[0086] Step 402: Perform validity validation on the required parameters for the call and obtain the validation result. If the validation result indicates that the required parameters for the call have passed the validation, convert the type of the validated required parameters for the call into a standard format that the target processing tool can recognize, and obtain the type-converted required parameters for the call.

[0087] The aforementioned validation of the required parameters includes, but is not limited to, checking for missing, incorrect, or invalid parameters, checking whether parameter values ​​are within preset allowed ranges, checking whether the control action belongs to a predefined action set, and checking whether the device name exists in the current user's device list. Type conversion may include: converting string-type numerical values ​​output by the model to integers or floating-point numbers, and mapping device aliases or natural language device names to internal system device identifiers.

[0088] The standard format that the target processing tool can recognize can refer to a pre-defined parameter format that can be directly recognized by the target processing tool.

[0089] For example, the parameters required for a successful call include brightness "80" and device name "living room light", both of which are strings. The target processing tool is a device control class, so the string "80" needs to be converted into an integer "80" that the device control class tool can recognize. The device name "living room light" needs to be converted into a device identifier (e.g., device_id="light_001") that the device control class tool can recognize using the device cache.

[0090] Step 403: Based on the parameters required for the call after type conversion, call the target processing tool.

[0091] In this embodiment, the target sub-agent extracts the parameters required to call the target processing tool from the task handover request through a task recognition model, and performs validity checks and type conversions on the parameters in sequence. Finally, the target processing tool is called in the converted standard format. This ensures that the parameters required for the call are precisely aligned with the target processing tool in terms of data format (e.g., converting string values ​​to integer types and mapping device names to device identifiers that the tool can recognize). This eliminates execution anomalies caused by missing parameters, out-of-bounds errors, or type mismatches before the call, thereby improving the success rate of tool calls and the reliability of task processing results.

[0092] In some embodiments of this application, validating the parameters required for the call and obtaining the validation result may include steps 501 to 503.

[0093] Step 501. If the target processing tool is of type device control and the required parameters for the call are missing device control parameters, then read the total list of smart devices of the device control class from the device cache.

[0094] Among them, device control parameters can refer to parameters required when calling device control tools, which may include at least one of device identifier, device type, control action, and control value. Device identifier (e.g., light_001) is used to uniquely identify the device to be controlled.

[0095] The total smart device list can refer to a list of devices of the same type as the device type currently awaiting clarification, read from the device cache. The total smart device list may include the device identifiers and location information of multiple devices, and may also include data such as device names. The device type currently awaiting clarification can refer to the device type corresponding to the missing device control parameters needed to retrieve the required parameters.

[0096] For example, when a user says "turn on the lights" and there are three lights in the house, the current device type to be clarified is "lights." The total smart device list can include: the device identifier for each light (e.g., light_001, light_002, light_003), the location information for each light (e.g., living room, bedroom, study), and the device name for each light (e.g., living room light, bedroom light, study light). In this embodiment, the tool service layer can use a short-term caching strategy to store the total smart device list (e.g., the lifespan of the total device category in the device cache is 5 minutes) to reduce the latency caused by repeatedly fetching the total smart device list.

[0097] Step 502: Generate a clarification block based on the total list of smart devices and the voice turn identifier, and output the clarification block to the user. The clarification block includes the device identifier and location information of each device in the total list of smart devices.

[0098] The clarification block can also include the device name of each device in the total list of smart devices.

[0099] It should be understood that this application does not limit the type of the clarification block. For example, the type of the clarification block can be text or list. As an example and not a limitation, the content of the clarification block could be: "You have three lights in your house, one in the living room, one in the bedroom, and one in the study. Which one would you like to turn on?"

[0100] Step 503: Receive feedback information from the user regarding the clarification block and send the feedback information to the target sub-agent. The target sub-agent corrects the required parameters based on the feedback information and generates a verification result indicating that the required parameters have passed verification.

[0101] The feedback information refers to the user's selections for all devices in the clarification block. Feedback information may include device names and / or location information. For example, after seeing or hearing "You have three lights in your house, one in the living room, one in the bedroom, and one in the study. Which one would you like to turn on?", and answering "the one in the living room," the feedback information is the location selection result of "the one in the living room," which determines the specific device the user is referring to (e.g., light_001).

[0102] In this embodiment, when a device control tool is found to lack device control parameters during the parameter validity verification process, the system automatically reads the total list of similar smart devices from the device cache and generates a clarification block containing device identifiers and location information, which is then output to the user. The system then receives user feedback on this clarification block, and the target sub-agent corrects the required parameters accordingly. This allows ambiguous commands that would otherwise be impossible to execute due to incomplete parameters (such as "turn on the light") to be accurately interpreted as control of a specific device through a single lightweight interaction. Thus, without increasing the user's learning curve, parameter validity verification and multi-device clarification are organically integrated. This avoids tool call failures or misoperations due to missing parameters and improves the accuracy of device control commands and the user experience.

[0103] In one implementation, before invoking external tools, the system verifies, converts, and maps the parameters required for the invocation generated by the agent through a tool wrapping layer. When a user request involves multiple devices of the same type and the target device is unclear, the system generates a clarification block and sends it to the client, writing the user-confirmed clarification result into the runtime state layer or checkpoint state layer. In subsequent rounds, when a similar device control request occurs again, the system can reuse the clarification result to directly determine the target device, thereby reducing repeated queries and improving the continuity of multi-round voice interactions.

[0104] In some embodiments of this application, the processing details of the task processing request include one or more of the following: verified call parameters, which are stored in a first storage field of the runtime state layer; a clarification block, which is stored in a second storage field of the runtime state layer; and context information of the clarification block, which is stored in the second storage field.

[0105] The context information of the clarification block includes, but is not limited to, the current device type to be clarified and the total list of smart devices. In this embodiment, by storing the validated call parameters in the first storage field of the runtime state and storing the clarification block and its context information in pairs in the second storage field of the runtime state layer, multi-device clarification is no longer a one-off interaction, but a controlled state that can be recorded by the system and continued across rounds. When the user gives a selection in the next round (e.g., "living room"), the existence of the first and second storage fields can be detected during the intent recognition stage. The record can be directly read, and the location description given by the user can be mapped to a specific device identifier, thereby skipping the steps that should be repeated, such as "parameter validity verification → reading the total list of smart devices → generating the clarification block", and directly executing the tool call. This mechanism avoids repeated queries for the same device type in the same user session, significantly improving the efficiency and fluency of voice interaction in multi-device scenarios.

[0106] In some embodiments, after multi-device clarification is completed, the device mapping finally confirmed by the user (e.g., the mapping between the device identifiers of a lamp and a living room lamp) can be stored in a third storage field in the checkpoint state layer. Based on this, in subsequent rounds of the same user session, if the user again states the device type to be clarified (e.g., turning off the light), the third storage field will be queried during the parameter validity verification phase. If a corresponding mapping exists, the device identifier in that mapping will be used directly, without repeatedly asking the user. This mapping is also stored in the persistent state layer for long-term memory. This reduces repeated confirmation steps, improves the continuity of multi-round voice interaction, and enhances device control efficiency.

[0107] In some embodiments, the processing details of a task processing request may further include a current message list, intermediate computation results, tool call results, and a queue of blocks to be consumed. The current message list may include the task processing request for the current round and complete messages from the previous N rounds of dialogue (each message includes the sender's identifier, message content, tool call request, tool call representation, timestamp, etc.) to preserve a complete tool call chain, where N is a positive integer. Intermediate computation results may include the next step plan inferred by the main agent, partial summary results, and a list of parameters for tools to be executed. Tool call results may include the processing results returned by each processing tool and the parsed structured data. The queue of blocks to be consumed may include structured blocks generated based on the task processing results that have not yet been sent to the text-to-speech module or the client.

[0108] In some embodiments, data in the runtime state layer can be directly read during the main agent decision-making or target sub-agent decision-making stage, the parameter validity verification stage, and the structured block generation stage.

[0109] Understandably, the agent orchestration phase is used to decouple different business capabilities. After receiving a user request, the computer device first performs initial intent recognition to determine the processing type of the user's intent, such as device control, device installation, troubleshooting, scene management, knowledge-based question answering, or casual conversation. Subsequently, it determines whether the processing type of the user's intent falls under the sub-agent dispatch type. If not, the main agent determines to directly generate a response based on the initial intent recognition result, the current session context, the device context, and historical interaction states; if so, the current request is transferred to the corresponding specialized sub-agent for processing.

[0110] During the agent handover process, control parameters such as the maximum number of handovers between agents, the handover timeout, and whether to automatically return to the main agent can be set. Once a specialized sub-agent completes its processing, its result is returned to the main agent, which then aggregates the results and generates the final response. If the number of handovers between agents exceeds a preset limit or the handover process exceeds a preset timeout, the handover is terminated, and the currently obtained processing result or a preset fallback message is returned, thus avoiding repeated handovers or prolonged periods of unresponsiveness between agents.

[0111] In some embodiments of this application, the method further includes the following during the processing of task processing requests: If a user disconnects and reconnects, based on the user session associated with the user's voice command, a snapshot of the runtime state layer that was most recently successfully written is loaded from the checkpoint state layer; the snapshot of the most recently successfully written runtime state layer is deserialized to obtain runtime state recovery data; the runtime state layer is reconstructed or updated based on the runtime state recovery data, so that task processing requests can continue to be processed from the task processing progress corresponding to the runtime state recovery data.

[0112] In this embodiment, when storing a snapshot of the runtime state layer at a preset write time to the checkpoint state layer, the timestamp, session link identifier, and voice turn identifier corresponding to the snapshot can also be stored in the checkpoint state layer. There is a mapping relationship between these data and between these data and the snapshot. Based on this, the snapshot of the runtime state layer that was most recently successfully written can be loaded from the checkpoint state layer based on the session link identifier of the user session associated with the user's voice command.

[0113] In this embodiment, when a user disconnects and reconnects, a snapshot of the runtime state layer that was most recently successfully written is loaded from the checkpoint state layer based on the user session. This snapshot is then deserialized to obtain runtime state recovery data. The runtime state layer is then reconstructed or updated based on this recovery data. This allows the system to continue processing task requests from the task processing progress corresponding to the runtime state recovery data. As a result, the entire task processing flow does not need to be restarted after the connection is interrupted, and the user does not need to repeat the completed interaction steps or the provided instruction information. This achieves breakpoint resume-type session recovery, improves the continuity and completion rate of long-process or multi-round interactive tasks in unstable network environments, and avoids session state loss and duplicate processing overhead caused by connection interruption.

[0114] In some embodiments of this application, the computer device may also perform the following steps: Store the session metadata of the user session associated with the user's voice command to the persistent state layer; and / or, after executing at least one key operation for the task processing request, store the log of at least one key operation to the persistent state layer; and / or, store the version number of the prompt word currently used by the main agent and the content snapshot of the target sub-agent to the persistent state layer; and / or, store the message list of the task processing request to the persistent state layer.

[0115] The session metadata associated with the user's voice command includes, but is not limited to, the session link identifier of the user's session, the voice turn identifier associated with the user's voice command, the user identifier, the family identifier of the application voice interaction (for example, if the application scenario of this application is the family of user A, then the family identifier of the application voice interaction can be the identifier of the family where user A lives), the currently used intelligent agent (including the main intelligent agent and the target sub-intelligent agent), the creation time of the user session, and the last active time of the user session (i.e., the timestamp of the last request processed by the user session), etc.

[0116] Optionally, at least one key operation can be set according to actual needs. For example, at least one key operation may include voice input operation, main agent decision, tool call start operation, tool call return operation, fault event detection operation (such as parameter validity detection, call anomaly detection, etc.), reliability detection operation (such as call timeout), etc. The corresponding log (i.e., audit log) may include: user voice text, agent decision results, tool call input and output, error stack, timeout event, etc.

[0117] The message list for task processing requests can be a complete record of each message in the task processing request (with the same message list format as the runtime state layer), and additionally includes compression start time and compression end time fields to mark the historical range that has been moved out of the window, but without summary compression.

[0118] In this embodiment, session metadata can be stored in the persistent state layer when a user session is established. When the prompt word used by the main agent and the target sub-agent changes, the version number and content snapshot of the changed prompt word can be stored in the persistent state layer. At the end of each round of requests, newly added messages and tool call records can be asynchronously written to the message list. Audit logs can also be stored in the persistent state layer in real time or in batches. When a user session is restored, if the data in the checkpoint state layer has expired (i.e., exceeded its lifespan, for example, more than 24 hours), the most recently successfully stored message can be loaded from the persistent state layer and used as runtime state restoration data to rebuild or update the runtime state layer. This allows task processing requests to continue processing from the task processing progress corresponding to the runtime state restoration data, thus restoring the user session. Data can be read from the persistent state layer when querying historical conversations, audit logs, or analyzing model behavior in the management console.

[0119] In this embodiment, the persistence of these data is achieved by storing session metadata, logs of key operations, prompt word version numbers and content snapshots, and message lists in the persistent state layer. The persistence of session metadata allows for long-term traceability of the basic information of each interaction. The persistence of logs of key operations provides end-to-end auditing capabilities, facilitating problem localization and accountability. The persistence of prompt word version numbers and content snapshots ensures the reproducibility and compliance of agent behavior versions. The persistence of the message list provides a reliable initial context for downgrade recovery after snapshot expiration. These storage operations collectively form the foundational support for the system in terms of auditability, traceability, compliance, and downgrade recovery capabilities, enabling the system to not only be fully recorded during normal operation but also to perform effective analysis and recovery based on persistent data in abnormal scenarios.

[0120] In one implementation, during the processing of a task processing request, the processing details of the task processing request are stored and managed in a hierarchical manner. Specifically, the processing details of the task processing request are stored in the runtime state layer, and a snapshot of the runtime state layer is stored in the checkpoint state layer at a preset write time. The checkpoint state layer is used to restore the processing state of the task processing request in case of an anomaly, such as when a user connection is lost, the service restarts abnormally, processing continues across requests, or the user re-initiates interaction. The context and processing progress of the task processing request are restored based on the snapshot stored in the checkpoint state layer. The runtime state layer is used to save temporary state data of the task processing request during the current processing. The processing details of the task processing request may include one or more of the following: current message list, intermediate calculation results, tool call results, queue of structured blocks to be consumed, parameters required for a valid call, clarification blocks, and context information of the clarification blocks. The current message list may include complete messages of the current request and several previous rounds of dialogue. Each message may include a role, content, tool call information, tool call identifier, and timestamp to preserve a complete tool call chain. Once the tool wrapper layer validates the parameters required for invoking the target tool and confirms that the validation is successful, it stores the validated parameters in the first storage field of the runtime state layer. The required parameters may include device identifier, control action, control value, knowledge query text, and the number of returned results. Subsequently, when executing a tool call, generating a structured response, or performing anomaly recovery, the system can read the validated parameters from the first storage field.

[0121] The preset write times can include one or more of the following: after each agent node execution, before and after each tool call, after each structured block generation, before the user actively pauses, and before the user connection is disconnected. At each preset write time, a snapshot of the runtime state layer is asynchronously written to the checkpoint state layer. The checkpoint state layer can include a complete snapshot of the runtime state layer, a checkpoint identifier, a timestamp, a session link identifier, and a voice turn identifier. The checkpoint state layer can be stored in a cache system and can be set with a predetermined lifespan, such as 24 hours, to enable automatic reclamation of inactive sessions. Session metadata associated with user voice commands is stored in the persistent state layer. Session metadata can include a session link identifier, user identifier, family identifier, currently active agent, creation time, and update time. The persistent state layer provides long-term data storage to support historical backtracking, audit analysis, and recovery after system failures.

[0122] After executing at least one key operation in response to the task processing request, the logs of at least one key operation are stored in the persistent state layer. Key operations may include user request reception, agent decision-making, agent handover, tool invocation, tool return, structured block generation, speech synthesis, exception handling, and timeout handling. Logs may include the original user request text, agent decision results, tool invocation input, tool invocation output, error stack, timeout event, and operation timestamp. The system also stores the current prompt word version number and content snapshot of the main agent and target sub-agent in the persistent state layer. This allows for the determination of the prompt word version corresponding to each task processing request during subsequent troubleshooting, effect analysis, version rollback, or model optimization. The message list for the task processing request is also stored in the persistent state layer. The message list may have the same or corresponding data format as the current message list in the runtime state layer and includes role, content, tool invocation information, tool invocation identifier, and timestamp. For historical messages that have been moved out of the current message window, the range can be marked by compression start time and compression end time, but the original message content is still stored in the persistent state layer to avoid loss of tool parameters, device identification or clarification results due to lossy digest.

[0123] In this embodiment, through the hierarchical management of the runtime state layer, checkpoint state layer, and persistent state layer, as well as the complete tool call chain retention strategy, the consistency of key parameters and execution context can be maintained during disconnection recovery and multi-turn dialogue.

[0124] In some embodiments, before storing the logs of critical operations to the persistent state layer, sensitive fields in the logs of critical operations can be anonymized. Sensitive fields include at least one of user tokens, passwords, device serial numbers, and device identifiers. Alternatively, when the gateway layer calls the agent orchestration layer, or the agent orchestration layer calls the processing tool, authentication information can be included in the service call request, and the called party can verify the authentication information. This achieves identity authentication and authorization isolation between services, preventing unauthorized requests from entering the internal system links.

[0125] In some embodiments of this application, generating a structured block sequence corresponding to the task processing result through the main intelligent agent may include steps 601 to 603.

[0126] Step 601: The main intelligent agent inputs the task processing result and the task type associated with the task processing request into the structured partitioning model. The structured partitioning model performs structured partitioning of the task processing result based on constraint prompts to obtain the output result. The constraint prompts are used to constrain the structured partitioning model to partition the task processing result according to a predetermined structured data format.

[0127] The predetermined structured data format can refer to a predefined output data organization specification that the structured partitioning model must adhere to. The core of the predetermined structured data format can be a JSON Schema, which stipulates that the output contains an array of structured blocks. Each element of the structured block array is a structured block object, and each structured block object includes at least a block type field, and selectively includes, depending on the block type: when the block type is text, it includes a content field carrying the text block to be broadcast; when the block type is non-text, it includes a data field carrying the non-text block for rendering.

[0128] Step 602: If the output result conforms to the predetermined structured data format, then the output result is determined as the structured block sequence corresponding to the task processing result.

[0129] For example, the output results conforming to the predetermined structured data format are as follows: Structured block array: [{block type field: text type, content field: ...}, {block type field: task step type, data field: ...}].

[0130] Step 603: If the output result does not conform to the predetermined structured data format, the main intelligent agent performs structured division of the task processing result based on preset keywords or preset templates to obtain a structured block sequence; both preset keywords and preset templates are used to identify the block type and start and end positions of the structured blocks.

[0131] Preset keywords can be words or symbols with pre-defined structural meanings. There can be at least one preset keyword. For example, preset keywords could be: "Step 1:", "Step 2:", "Tip:", "Note:", etc.

[0132] In this embodiment, when the text of a certain line in the output result matches a certain preset keyword (here, the starting position), it is determined that the block type corresponding to the subsequent content led by that line is the block type associated with the matched preset keyword. Taking the position of the preset keyword in the line as the starting point and the next similar keyword or the end position of the paragraph as the ending point, the start and end positions of the corresponding structural block are determined, thereby dividing the output result into a structured block sequence that conforms to the predetermined structured data format.

[0133] Preset templates can refer to pre-defined sentence matching rules (such as regular expressions or fixed prefix patterns). There must be at least one preset template. For example, preset templates could be: "Please follow these steps:", "Operation steps are as follows:", "Specific steps are as follows:", "Query results are as follows:", "Including the following points:", "Notes:", etc.

[0134] In this embodiment, when the text of a certain line in the output result matches the matching pattern of a certain preset template, it is determined that the block type corresponding to the subsequent content led by that line is the block type associated with the matched preset template. The start and end positions of the corresponding structural block are determined with the position of the preset template in the line as the starting point and the end position of the paragraph as the ending point, thereby dividing the output result into a structured block sequence that conforms to the predetermined structured data format.

[0135] In this embodiment, by inputting the task processing results and associated task types into a structured partitioning model, and using constraint prompts to constrain the model to output according to a predetermined structured data format, a structured block sequence can be directly obtained in an efficient manner with a single generation that conforms to the format when the model output meets expectations. Simultaneously, for abnormal cases where the model output does not conform to the predetermined format, post-processing partitioning of the task processing results is performed using preset keywords or preset templates. This ensures that even when the model output is not standardized, the ability to identify the block type and start / end positions of the structured blocks based on keywords or templates can still restore the free text into a structured block sequence conforming to the predetermined structured data format. This enhances the robustness in the face of model output uncertainty, ensuring that the generation of structured blocks does not fail due to a single deviation in the model's output format, thereby improving the reliability and stability of the voice interaction system in multimodal output scenarios.

[0136] In some embodiments of this application, the response results generated by the main intelligent agent or specialized sub-intelligent agents are organized into an ordered list of structured blocks. Specifically, based on the block type and voice turn identifier, multiple structured blocks are sent to corresponding output channels to output the task processing results through the corresponding output channels, including: If there are text blocks of type text among multiple structured blocks, the text blocks are written into the incremental text-to-speech queue of the text-to-speech module according to the speech round identifier. The text-to-speech module converts the text blocks into audio data at the same time as receiving them, and sends the audio data to the client for real-time playback. If there are non-text blocks of non-text type among multiple structured blocks, then the non-text blocks are sent to the client for graphical rendering based on the voice round identifier.

[0137] The incremental text-to-speech queue refers to a block-level cache queue arranged in the arrival order of text blocks, including text blocks to be played. For each text block, the gateway acts as the producer, writing it into the incremental text-to-speech queue. The text-to-speech module acts as the consumer, receiving, synthesizing, and transmitting the text in real time. This reduces the latency of the first character being played, enabling simultaneous generation and playback, rather than waiting for the entire sentence or paragraph to return before playing the text block. A text block carries a piece of natural language text suitable for playback (e.g., "The living room lights have been turned on for you"), and its output channel is the text-to-speech channel. Non-text blocks carry renderable structured information data (e.g., installation guides, troubleshooting steps, device status such as on / off / brightness / mode). Non-text blocks can include task step blocks, device status blocks, chart blocks, button blocks, or list blocks, etc.

[0138] In this embodiment, text blocks are written to the incremental text-to-speech queue of the text-to-speech module based on speech round identifiers. This allows text blocks to be converted in parallel and audio data to be sent to the client for playback in real time upon arrival, reducing the latency of the first character. Simultaneously, non-text blocks are sent to the client for graphical rendering based on the same speech round identifier, enabling the same round of response to simultaneously support voice playback and interface display, avoiding inconsistencies between the played content and the content displayed on the client. Furthermore, different output formats within the same round are bound to the same round. When a user interrupts or cancels, all structured blocks in that round can be identified and processed uniformly based on the speech round identifier, preventing interference between different rounds. Since the structured blocks carry round identifiers and are generated sequentially, the output order of structured blocks within the same round is guaranteed to remain consistent, thereby improving the controllability of multimodal output and the continuity of the user experience. In some embodiments, structured blocks can be transmitted simultaneously with their generation, reducing waiting time and improving the response speed to structured blocks.

[0139] like Figure 7 The diagram shows a flowchart of multi-agent collaboration, tool invocation, and dual-channel distribution of structured blocks provided in this application embodiment. The flowchart mainly includes two parts: multi-agent collaboration and tool invocation, and dual-channel distribution of structured blocks. In the multi-agent collaboration and tool invocation part, the master agent can delegate task processing requests to the device control agent, fault diagnosis agent, scene management agent, etc. These specialized sub-agents can call processing tools from their respective whitelisted tool ranges and return the corresponding tool execution results. The master agent summarizes all tool execution results and generates a structured block sequence. In the dual-channel distribution part, for each structured block in the structured block sequence, if it is a text block, audio data is played through a text-to-speech module; if it is a non-text block, it is rendered graphically.

[0140] In some embodiments of this application, the computer device may also perform the following steps: During the real-time transmission of audio data to the client for real-time playback, if a new user voice activity is detected and the new user voice activity meets the preset interruption conditions, the corresponding audio playback will be stopped based on the voice round identifier corresponding to the currently played audio data. Clear the incremental text-to-speech queue corresponding to the voice turn identifier, and generate a new task processing request based on the new user voice activity.

[0141] In this embodiment, when a user interruption is detected, the system stops the audio playback corresponding to the old playback round based on the voice round identifier and clears the pending playback queue (i.e., the incremental text-to-speech queue), before switching to the new task processing request. This reduces issues such as crosstalk, residual playback, and mismatched results between different rounds.

[0142] In a preferred embodiment, the system employs a two-stage interruption determination mechanism to balance interruption response speed and false trigger suppression capability. The first stage is a rapid initial screening stage for voice activity, which performs real-time voice activity detection on the acquired audio stream to determine whether user voice activity exists in the current audio stream. When the detection result meets preset trigger conditions, such as continuously detecting human voices exceeding a preset duration threshold, the client generates an interruption event and sends it to the server, enabling the server to promptly stop the TTS playback corresponding to the current voice turn or clear the waiting queue. This reduces the response latency after a user initiates an interruption. The preset trigger conditions may include one or more of the following: a voice activity confidence threshold, a duration threshold, and an energy threshold. By adjusting these thresholds, a balance can be struck between interruption response speed and false trigger probability.

[0143] The second stage is the semantic confirmation stage. Simultaneously with the client triggering the interruption event, the client continues to send the audio stream to the speech recognition module, which then recognizes the audio stream and generates recognized text. The system determines whether the current voice activity constitutes a valid user command based on the recognized text. Preferably, when the speech recognition result is empty, contains only invalid interjections, or does not meet the preset semantic validity conditions, the system determines that the voice activity is not a valid interruption and discards the corresponding interruption result; when the speech recognition result is not empty and meets the semantic validity conditions, the system confirms that the current voice activity is a valid interruption and switches to a new voice cycle for subsequent processing. Through this two-stage interruption determination mechanism, the system can first achieve low-latency response using local voice activity detection, and then use the speech recognition result for semantic confirmation, thereby avoiding false interruptions caused by background noise, coughing, or meaningless interjections, improving the accuracy of interruptions and user experience in real-time voice interaction.

[0144] like Figure 8The diagram shown is a main flow diagram and interruption control logic diagram of request-level real-time voice interaction provided in an embodiment of this application. Figure 8 The steps shown may include the following: Step 801: The user performs voice input or text input on the client.

[0145] Step 802: The gateway layer establishes a connection with the client via WebSocket.

[0146] Step 803: The gateway layer performs audio preprocessing, recognizes user voice commands, and binds session link identifiers during this process to generate voice round identifiers for this request.

[0147] Step 804: The agent orchestration layer performs user intent recognition, the main agent transfers the request to the target sub-agent and makes tool calls, and outputs a structured block sequence.

[0148] Step 805: For text blocks, write them into an incremental text-to-speech queue, synthesize them while receiving them, and transmit and broadcast them while sending them.

[0149] Step 806: For non-text blocks, the client renders them, displays them graphically, and merges them according to the voice round identifier.

[0150] Step 807: It was detected that the user spoke again during the broadcast.

[0151] Step 808: Execute the two-stage interruption judgment mechanism to confirm that the new input is valid.

[0152] Step 809: Stop the old task processing request, clear the incremental text-to-speech queue, and terminate the current broadcast.

[0153] In some processes, during the handling of task requests by the target agent and its sub-agents, corresponding timeout thresholds can be set. These thresholds can be set for one or more of the following processes: agent orchestration, agent model invocation, agent tool invocation, agent structured block generation, and agent text-to-speech. When any process reaches its corresponding timeout threshold, the process terminates or is downgraded, and a corresponding timeout prompt block is generated based on the speech round identifier. Downgraded processing includes skipping the timeout process, invoking a backup model or tool, returning partial results, using a templated response, or outputting only text without speech synthesis.

[0154] In some embodiments, the reliability of the entire voice interaction process can be ensured through mechanisms such as global execution timeout, maximum number of graph steps, maximum number of tool call rounds, model call timeout, tool call timeout, survival timeout during TTS synthesis, and heartbeat keep-alive.

[0155] Optionally, the upper limits for global execution timeout, number of graph steps, number of tool calls, model call timeout, tool call timeout, TTS compositing survival time, and heartbeat interval can be set according to actual needs or experience. For example, the upper limit for global execution timeout is 90 seconds, the upper limit for number of graph steps is 50 steps, the upper limit for number of tool calls is 8 times, the upper limit for model call timeout is 30 seconds, the upper limit for tool call timeout is 10 seconds, the TTS compositing survival time is 5 seconds, and the heartbeat interval is every 30 seconds.

[0156] In some embodiments, a timer can be started when the main agent begins recognizing user intent. Before each node, the global execution time is checked for timeout. If a global execution timeout occurs, the corresponding operation is immediately terminated, and a fallback message, such as "Processing timed out, please try again later," is returned to the client. Before each entry into the main agent or target sub-agent, it can be checked whether the graph step limit has been reached. If the graph step limit is reached, graph execution is forcibly terminated, and partial results or a timeout message are returned. Before each call to a processing tool, it can be checked whether the number of tool calls has reached the tool call round limit. If the tool call round limit is reached, the call to the processing tool is rejected, and the currently summarized results are returned. When each large model inference is called, it can be checked whether the model call has timed out. If the model call times out, it is retried once; if it still times out, the default response is returned. When each processing tool is called, it can be checked whether the call time of the processing tool has timed out. If the tool call times out, the processing tool call is marked as failed, and an error message is returned to the target agent, which decides whether to retry or use another tool. During TTS synthesis, the system can detect if the lifespan has expired. If it has, the corresponding text block is skipped, and a default message, such as "Sorry, speech synthesis failed," is played directly. During WebSocket connection idle periods, the system can detect if the heartbeat keep-alive interval has exceeded a certain time limit. If it has, and there is no heartbeat response after two consecutive attempts, the connection is actively disconnected, and the client needs to reconnect.

[0157] It should be noted that the main intelligent agent, specialized sub-intelligent agents, intent recognition, and speech recognition and text-to-speech modules in this application are not limited to specific vendors or specific models. Any multimodal model, large language model, rule model, or combination thereof that can achieve equivalent semantic understanding, task orchestration, speech processing, and structured result generation can be used as an alternative implementation.

[0158] The orchestration structure of the main intelligent agent and specialized sub-intelligent agents in this application can be replaced by equivalent tree-like intelligent agents, graph workflow intelligent agents, rule orchestrators and large model hybrid schedulers, or multi-stage node routing mechanisms. As long as it can realize task distribution, result collection and boundary control among multiple specialized processing units, it falls under the protected ideas of this application.

[0159] The device control and knowledge service in this application can adopt MCP, REST, gRPC, message queue calls or other standardized tool interfaces; the connection reuse, long connection, short connection or connection pool mechanism in the call process can be adjusted according to the deployment environment.

[0160] The block type, field name, and serialization format of the structured blocks in this application can be expressed in JSON, binary blocks, event streams, or other equivalent ways; as long as it can achieve separate consumption of text blocks and non-text blocks and unified distribution in the same round, it falls within the scope of this application.

[0161] The runtime state layer, checkpoint state layer, and persistent state layer in this application can be hosted in memory, Redis, a document database, a relational database, or a distributed caching system, respectively; message window management can also adopt rule pruning, tag pruning, or other equivalent strategies that can retain key toolchain information.

[0162] The multi-agent-based voice interaction method provided in this application can be deployed in the cloud or in a local private environment; it can run on a single machine or be deployed in a multi-instance cluster, container orchestration or edge-cloud collaborative deployment without affecting the core technical ideas of this application.

[0163] The multi-agent-based voice interaction method provided in this application supports both single-machine verification and multi-instance production deployment. It can adapt to different computing resources and deployment environments while keeping the core technology unchanged, thus improving the engineering implementation capability.

[0164] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0165] Corresponding to the voice interaction method based on multi-agent orchestration described in the above embodiments, Figure 9 The diagram shows a schematic of the structure of a voice interaction system based on multi-agent orchestration provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0166] Reference Figure 9 The voice interaction system includes: The request generation module 901 is used to generate a task processing request based on the user's voice command and the voice turn identifier associated with the user's voice command. The task processing module 902 is used to identify user intent based on task processing request through the main intelligent agent, obtain user intent, determine target sub-intelligent agent from multiple professional sub-intelligent agents based on the processing type of user intent, process task processing request through target sub-intelligent agent, obtain task processing result, and return task processing result to the main intelligent agent. The sequence generation module 903 is used to generate a structured block sequence corresponding to the task processing result through the main intelligent agent; the structured block sequence includes multiple structured blocks, and each structured block includes a block type and a speech round identifier; The result output module 904 is used to send multiple structured blocks to the corresponding output channels according to the block type and voice round identifier, so as to output the task processing results through the corresponding output channels; The information storage module 905 is used to store the processing details of the task processing request to the runtime state layer during the processing of the task processing request, and to store a snapshot of the runtime state layer at a preset write time to the checkpoint state layer; the checkpoint state layer is used to restore the processing state of the task processing request in case of an exception.

[0167] In some embodiments, before generating a task processing request, the voice interaction system further includes: The voice processing module is used to perform voice activity detection on the received continuous audio frames through the gateway layer. If a user voice command is detected, the voice recognition service is called and the user voice command is recognized by the voice recognition service to obtain the user voice text. The session establishment module is used to detect the session link identifier corresponding to the user's voice command. If no session link identifier corresponding to the user's voice command is detected, a user session associated with the user's voice command is established, and a session link identifier for the user session is generated. The identifier generation module is used to generate voice round identifiers corresponding to user voice commands based on the session link identifier; Request generation module 901 is specifically used for: The task processing request is obtained by encapsulating the user's voice text, session link identifier, voice turn identifier, context information of the user's voice command, and the current state associated with the user session.

[0168] In some embodiments, the task processing module 902 is specifically used for: If the user intent processing type is the sub-agent distribution type, then the master agent in the agent orchestration layer determines the target sub-agent from multiple specialized sub-agents in the agent orchestration layer based on the user intent, and generates a task handover request based on the task processing request. The main agent transfers the task handover request to the target sub-agent, which then calls the processing tool within its corresponding whitelist of tools to process the task handover request and obtain the task processing result.

[0169] In some embodiments, the task processing module 902 is specifically used for: When the first sub-block of the task handover request is obtained by the main agent through block-level diversion processing of the task handover request, the first sub-block of the handover request is handed over to the target sub-agent in real time. The main agent sequentially transfers subsequent handover request sub-blocks generated based on the task handover request to the target sub-agent until all handover request sub-blocks generated based on the task handover request are sent to the target sub-agent; subsequent handover request sub-blocks are handover request sub-blocks generated after the first handover request sub-block; When the target sub-agent receives the first handover request sub-block, it identifies and processes the first handover request sub-block, and then identifies and processes the subsequent handover request sub-blocks in turn to obtain the task handover request. Based on the task handover request, the target processing tool is determined from the whitelist of tools corresponding to the target sub-agent to process the task handover request. The target sub-agent determines the parameters required to invoke the target processing tool based on the task handover request, and invokes the target processing tool based on the required parameters, so that the target processing tool can process the task handover request and obtain the task processing result.

[0170] In some embodiments, the task processing module 902 is specifically used for: The target sub-agent extracts the parameters required to call the target processing tool from the task handover request through the task recognition model; The validity of the required parameters for the call is validated, and the validation result is obtained. If the validation result indicates that the required parameters for the call have passed the validation, the type of the validated required parameters for the call is converted into a standard format that the target processing tool can recognize, and the converted required parameters for the call are obtained. Based on the parameters required for the type conversion, the target processing tool is invoked.

[0171] In some embodiments, the task processing module 902 is specifically used for: If the target processing tool is of the device control class and the required parameters for the call are missing device control parameters, then the total list of smart devices of the device control class is read from the device cache. A clarification block is generated based on the total list of smart devices and the voice turn identifier, and the clarification block is output to the user. The clarification block includes the device identifier and location information of each device in the total list of smart devices. Receive user feedback on the clarification block and send the feedback to the target sub-agent. The target sub-agent then corrects the required parameters based on the feedback and generates a verification result indicating that the required parameters have passed verification.

[0172] In some embodiments, the processing details information of a task processing request includes one or more of the following; The parameters required for the call that pass the verification are stored in the first storage field of the runtime state layer. Clarification blocks are stored in the second storage field of the runtime state layer; The context information of the clarification block is stored in the second storage field.

[0173] In some embodiments, the voice interaction system further includes: The data storage module is used to store the session metadata of the user session associated with the user's voice commands to the persistent state layer; And / or, after executing at least one critical operation for a task processing request, store the log of at least one critical operation to the persistent state layer; And / or, store the current prompt word version number and content snapshot of the main agent and the target sub-agent to the persistent state layer, respectively; And / or, store the list of task processing request messages to the persistent state layer.

[0174] In some embodiments, the sequence generation module 903 is specifically used for: The main intelligent agent inputs the task processing results and the task type associated with the task processing request into the structured partitioning model. The structured partitioning model then performs structured partitioning of the task processing results based on constraint prompts, and obtains the output results. The constraint prompts are used to constrain the structured partitioning model to output the task processing results according to a predetermined structured data format. If the output conforms to the predetermined structured data format, the output will be determined as the structured block sequence corresponding to the task processing result. If the output does not conform to the predetermined structured data format, the main intelligent agent will perform structured division of the task processing results based on preset keywords or preset templates to obtain a sequence of structured blocks; the preset keywords and preset templates are used to identify the block type and start and end positions of the structured blocks.

[0175] In some embodiments, the result output module 904 is specifically used for: If there are text blocks of type text among multiple structured blocks, the text blocks are written into the incremental text-to-speech queue of the text-to-speech module according to the speech round identifier. The text-to-speech module converts the text blocks into audio data at the same time as receiving them, and sends the audio data to the client for real-time playback. If there are non-text blocks of non-text type among multiple structured blocks, then the non-text blocks are sent to the client for graphical rendering based on the voice round identifier.

[0176] In some embodiments, the preset writing time includes at least one of the following: after the main agent completes the corresponding task, after the target sub-agent completes the corresponding task, after generating the structured block sequence corresponding to the task processing result, before the user pauses the user session associated with the user's voice command, and before the user disconnects.

[0177] In some embodiments, the voice interaction system further includes: The interruption determination module is used to stop the corresponding audio playback based on the voice round identifier corresponding to the currently played audio data if a new user voice activity is detected and the new user voice activity meets the preset interruption conditions during the process of real-time transmission of audio data to the client. The queue clearing module is used to clear the incremental text-to-speech queue corresponding to the voice turn identifier and generate new task processing requests based on new user voice activities.

[0178] In some embodiments, during the processing of a task processing request, the voice interaction system further includes: The snapshot loading module is used to load a snapshot of the runtime state layer that was most recently successfully written from the checkpoint state layer, based on the user session associated with the user's voice command, if a user is detected to have disconnected and reconnected. The deserialization module is used to deserialize the most recently successfully written snapshot of the runtime state layer to obtain runtime state recovery data; The update module is used to rebuild or update the runtime state layer based on the runtime state recovery data, so as to continue processing task processing requests from the task processing progress corresponding to the runtime state recovery data.

[0179] It should be noted that the information interaction and execution process between the above systems / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0180] Figure 10This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 10 As shown, the computer device 10 of this embodiment includes: at least one processor 1000 ( Figure 10 (Only one is shown in the diagram), memory 1001, and computer program 1002 stored in said memory 1001 and executable on said at least one processor 1000, wherein when said processor 1000 executes said computer program 1002, it causes computer device 10 to perform the steps in any of the above method embodiments.

[0181] The computer device may include, but is not limited to, a processor 1000 and a memory 1001. Those skilled in the art will understand that... Figure 10 The computer device 10 is merely an example and does not constitute a limitation on the computer device 10. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0182] The processor 1000 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0183] In some embodiments, the memory 1001 may be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 1001 may be an external storage device of the computer device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10. Further, the memory 1001 may include both internal and external storage units of the computer device 10. The memory 1001 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 1001 can also be used to temporarily store data that has been output or will be output.

[0184] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0185] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A voice interaction method based on multi-agent orchestration, characterized in that, include: A task processing request is generated based on the user's voice commands and the voice turn identifier associated with the user's voice commands; The main intelligent agent identifies user intent based on the task processing request to obtain user intent. Based on the processing type of the user intent, a target sub-intelligent agent is determined from multiple specialized sub-intelligent agents. The target sub-intelligent agent processes the task processing request to obtain the task processing result, and the task processing result is returned to the main intelligent agent. The main intelligent agent generates a structured block sequence corresponding to the task processing result; the structured block sequence includes multiple structured blocks, each of which includes a block type and the voice round identifier; According to the block type and the voice round identifier, the multiple structured blocks are sent to the corresponding output channels respectively, so as to output the task processing results through the corresponding output channels; During the processing of the task processing request, the processing details of the task processing request are stored in the runtime state layer, and a snapshot of the runtime state layer at a preset write time is stored in the checkpoint state layer; the checkpoint state layer is used to restore the processing state of the task processing request in case of an anomaly.

2. The voice interaction method according to claim 1, characterized in that, Prior to generating the task processing request, the method further includes: The gateway layer performs voice activity detection on the received continuous audio frames. If a user voice command is detected, the voice recognition service is invoked, and the user voice command is recognized by the voice recognition service to obtain the user voice text. The session link identifier corresponding to the user's voice command is detected. If no session link identifier corresponding to the user's voice command is detected, a user session associated with the user's voice command is established, and a session link identifier for the user session is generated. Generate the voice round identifier corresponding to the user's voice command based on the session link identifier; The process of generating a task processing request based on the user's voice commands and the voice turn identifier associated with those commands includes: The user's voice text, the session link identifier, the voice turn identifier, the context information of the user's voice command, and the current state associated with the user session are encapsulated to obtain a task processing request.

3. The voice interaction method according to claim 1, characterized in that, The process based on the user intent determines a target sub-agent from multiple specialized sub-agents, and the target sub-agent processes the task processing request to obtain the task processing result, including: If the processing type of the user intent is a sub-agent distribution type, then the main agent in the agent orchestration layer determines the target sub-agent from multiple specialized sub-agents in the agent orchestration layer based on the user intent, and generates a task handover request based on the task processing request. The main intelligent agent transfers the task handover request to the target sub-intelligent agent, and the target sub-intelligent agent calls the processing tool within its corresponding whitelist tool range to process the task handover request and obtain the task processing result.

4. The voice interaction method according to claim 3, characterized in that, The process involves the main intelligent agent transferring the task handover request to the target sub-intelligent agent, and the target sub-intelligent agent invoking a processing tool within its corresponding whitelist of tools to process the task handover request and obtain the task processing result, including: When the first handover request sub-block is obtained by the main intelligent agent through block-level diversion processing of the task handover request, the first handover request sub-block is handed over to the target sub-intelligent agent in real time; The main agent sequentially transfers subsequent handover request sub-blocks generated based on the task handover request to the target sub-agent until all handover request sub-blocks generated based on the task handover request are sent to the target sub-agent; each subsequent handover request sub-block is a handover request sub-block generated after the first handover request sub-block. When the target sub-agent receives the first handover request sub-block, it identifies and processes the first handover request sub-block, and then sequentially identifies and processes the subsequent handover request sub-blocks to obtain the task handover request. Based on the task handover request, it determines the target processing tool for processing the task handover request from the whitelist tool range corresponding to the target sub-agent. The target sub-agent determines the parameters required to invoke the target processing tool based on the task handover request, and invokes the target processing tool based on the parameters, so as to process the task handover request through the target processing tool to obtain the task processing result.

5. The voice interaction method according to claim 4, characterized in that, The step of determining the parameters required for invoking the target processing tool based on the task handover request by the target sub-agent, and invoking the target processing tool based on the required parameters, includes: The target sub-agent extracts the parameters required to invoke the target processing tool from the task handover request using a task recognition model; The validity of the parameters required for the call is validated, and the validation result is obtained. If the validation result indicates that the parameters required for the call have passed the validation, the type of the validated parameters required for the call is converted into a standard format that the target processing tool can recognize, and the converted parameters required for the call are obtained. Based on the parameters required for the type conversion, the target processing tool is invoked.

6. The voice interaction method according to claim 5, characterized in that, The validation of the parameters required for the call, and the resulting validation, includes: If the target processing tool is of the device control class and the required parameters for the call lack device control parameters, then the total list of smart devices of the device control class is read from the device cache. A clarification block is generated based on the total list of smart devices and the voice round identifier, and the clarification block is output to the user. The clarification block includes the device identifier and location information of each device in the total list of smart devices. The system receives feedback from the user regarding the clarification block and sends the feedback to the target sub-agent. The target sub-agent then modifies the parameters required for the call based on the feedback and generates a verification result indicating that the parameters required for the call have passed verification.

7. The voice interaction method according to claim 6, characterized in that, The processing details of the task processing request include one or more of the following: The parameters required for the successful call are stored in the first storage field of the runtime state layer. The clarification block is stored in the second storage field of the runtime state layer; The context information of the clarification block is stored in the second storage field.

8. The voice interaction method according to any one of claims 1 to 7, characterized in that, The method further includes: The session metadata of the user session associated with the user's voice command is stored in the persistent state layer; And / or, after executing at least one critical operation for the task processing request, the log of the at least one critical operation is stored in the persistent state layer; And / or, store the current prompt word version number and content snapshot of the main agent and the target sub-agent to the persistent state layer, respectively; And / or, store the message list of the task processing request to the persistent state layer.

9. The voice interaction method according to any one of claims 1 to 7, characterized in that, The step of generating the structured block sequence corresponding to the task processing result through the main intelligent agent includes: The main intelligent agent inputs the task processing result and the task type associated with the task processing request into a structured partitioning model. The structured partitioning model then performs structured partitioning of the task processing result based on constraint prompts to obtain the output result. The constraint prompts are used to constrain the structured partitioning model to partition the task processing result according to a predetermined structured data format. If the output result conforms to the predetermined structured data format, then the output result is determined as the structured block sequence corresponding to the task processing result; If the output does not conform to the predetermined structured data format, the main intelligent agent performs a structured division of the task processing result based on preset keywords or preset templates to obtain the structured block sequence; the preset keywords and the preset templates are used to identify the block type and start and end positions of the structured blocks.

10. The voice interaction method according to any one of claims 1 to 7, characterized in that, The step of sending the multiple structured blocks to their respective output channels according to the block type and the voice round identifier, so as to output the task processing result through the corresponding output channels, includes: If there is a text block of type text among the multiple structured blocks, then according to the speech round identifier, the text block is written into the incremental text-to-speech queue of the text-to-speech module. When the text block is received, the text-to-speech module converts the text block into audio data and sends the audio data to the client for real-time playback. If there is a non-text block among the multiple structured blocks whose block type is not text, then based on the voice round identifier, the non-text block is sent to the client for graphical rendering.

11. The voice interaction method according to claim 10, characterized in that, The method further includes: During the process of real-time transmission of the audio data to the client for real-time broadcasting, if a new user voice activity is detected and the new user voice activity meets the preset interruption conditions, the corresponding audio broadcasting is stopped based on the voice round identifier corresponding to the currently broadcast audio data. Clear the incremental text-to-speech queue corresponding to the voice turn identifier, and generate a new task processing request based on the new user voice activity.

12. The voice interaction method according to any one of claims 1 to 7, characterized in that, The preset writing time includes at least one of the following: after the main agent completes the corresponding task, after the target sub-agent completes the corresponding task, after generating the structured block sequence corresponding to the task processing result, before the user pauses the user session associated with the user's voice command, and before the user disconnects.

13. The voice interaction method according to any one of claims 1 to 7, characterized in that, During the processing of the task processing request, the method further includes: If the user is detected to have disconnected and reconnected, a snapshot of the runtime state layer that was most recently successfully written is loaded from the checkpoint state layer based on the user session associated with the user's voice command. Deserialize the most recently successfully written snapshot of the runtime state layer to obtain runtime state recovery data; The runtime state layer is rebuilt or updated based on the runtime state recovery data, so as to continue processing the task processing request according to the task processing progress corresponding to the runtime state recovery data.

14. A voice interaction system based on multi-agent orchestration, characterized in that, include: The request generation module is used to generate a task processing request based on the user's voice command and the voice turn identifier associated with the user's voice command. The task processing module is used to identify user intent based on the task processing request through the main intelligent agent, obtain user intent, determine target sub-intelligent agent from multiple professional sub-intelligent agents based on the processing type of the user intent, process the task processing request through the target sub-intelligent agent, obtain task processing result, and return the task processing result to the main intelligent agent. A sequence generation module is used to generate a structured block sequence corresponding to the task processing result through the main intelligent agent; the structured block sequence includes multiple structured blocks, and each structured block includes a block type and the voice round identifier; The result output module is used to send the multiple structured blocks to the corresponding output channels according to the block type and the voice round identifier, so as to output the task processing result through the corresponding output channels; The information storage module is used to store the processing details of the task processing request to the runtime state layer during the processing of the task processing request, and to store a snapshot of the runtime state layer at a preset write time to the checkpoint state layer; the checkpoint state layer is used to restore the processing state of the task processing request in case of an anomaly.

Images