A safe voice intelligent customer service system and an interaction method based on a multi-agent architecture

Abstract

This invention discloses a secure voice-based intelligent customer service system and interaction method based on a multi-agent architecture. The system includes an ASR (Automatic Speech Recognition) module, multi-layered security barriers, a main retrieval agent, a specialized retrieval agent cluster, a parallel retrieval and result aggregation module, an LLM (Limited Language Modeling) synthesis engine, and a TTS (Text-to-Speech) module. Through multi-agent system architecture and methodological design, this invention achieves modular processing, parallel retrieval tasks, and closed-loop security control. By enabling independent calls between input, output, agents, and inference stages via multi-layered security barriers, the system effectively mitigates risks such as prompt word injection, indirect injection attacks, harmful content generation, and information leakage. Sensitive information identification, access permission verification, and content compliance review are embedded in each stage of input, retrieval, information synthesis, and output to prevent sensitive data leakage and meet the application requirements of high-security business scenarios (such as finance, telecommunications, and government services).

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, natural language processing and voice interaction, in particular to a safe voice intelligent customer service system and interaction method based on a multi-agent architecture. BACKGROUND

[0002] With the development of artificial intelligence technology, intelligent customer service is increasingly popular in many fields. Currently, the mainstream intelligent customer service system mainly adopts the following four technical solutions:

[0003] 1. Single round QA system based on rules or retrieval: The system relies on a pre-defined knowledge base and rule templates, and retrieves pre-set answers from the FAQ library through keyword matching or simple semantic similarity calculation (such as TF-IDF, BM25). This solution has a simple and direct processing flow, but has poor flexibility and generalization ability, cannot handle complex, multi-round or semantically variable user queries, and lacks a security audit mechanism to effectively identify malicious inputs or unauthorized requests.

[0004] 2. Question and answer system based on a single large language model (LLM): Directly use the internal knowledge and generation ability of a large language model (such as the GPT series) to answer user questions. This solution generates natural and fluent answers, but the LLM has a "hallucination" problem, which may generate factually incorrect information, and cannot access external knowledge that is proprietary, real-time or structured, resulting in low accuracy in specialized fields. At the same time, the system lacks effective security controls for inputs and outputs, making it vulnerable to malicious queries that induce the generation of inappropriate content, and has no cost control mechanism, resulting in resource waste.

[0005] 3. Question and answer system based on traditional RAG: Introduce retrieval augmented generation (RAG) technology based on LLM, obtain external knowledge through a static retrieval pipeline (such as a vector database), and input it as context to the LLM to generate answers. This solution alleviates the hallucination problem and introduces real-time data, but its retrieval and generation process is fixed (retrieval-reading), which has weak context integration ability, poor multi-step reasoning ability and extension delay problems. The system is difficult to perform deep reasoning on the retrieval results or iterative retrieval based on intermediate answers, resulting in fragmented and incoherent answers to complex queries. In addition, this system also has security risks such as data pollution and information leakage, and has obvious shortcomings in data security and system protection.

[0006] 4. Single-Agent RAG System: This is an evolution of traditional RAG, introducing a central agent to dynamically coordinate query understanding, tool selection (such as databases, search APIs), and answer generation. This architecture achieves a certain degree of dynamic routing (such as choosing to call Text-to-SQL or semantic search), which is superior to static RAG. However, its core bottleneck lies in the fact that all tasks are scheduled and processed serially by a single agent. When faced with complex queries that require simultaneous access to multi-source heterogeneous data (such as databases + documents + APIs), this architecture becomes a bottleneck in performance and flexibility, with limited scalability. In addition, the single-agent architecture has significant security flaws: as the core decision-making point of the system, the single agent is prone to becoming a single point of failure. Once maliciously attacked (such as injection hints, unauthorized access) or misled, the entire system may be manipulated, leading to data leaks, the generation of harmful content, or service abuse.

[0007] Based on the practical effects of the above-mentioned specific application methods, existing intelligent customer service systems generally have the following shortcomings:

[0008] 1. Limited and rigid processing capabilities: Traditional RAG systems have static and rigid processes, lacking the ability to perform multi-step reasoning and iterative retrieval for complex queries, resulting in insufficient information integration and fragmented answers; although single-agent RAG systems achieve dynamic routing, all tasks rely on serial processing by a single agent, which becomes a system bottleneck when facing multi-source heterogeneous data queries, resulting in poor scalability and high response latency.

[0009] 2. Insufficient architectural scalability: The system's functions are tightly coupled with the data source. Adding new data sources or processing capabilities requires significant modifications to the entire system, making upgrades and maintenance difficult and costly. Furthermore, it lacks a plug-and-play design for security modules.

[0010] 3. Lack of Security and Cost Control Mechanisms: Most existing systems lack effective content security auditing mechanisms (Guardrails) for user input and system output. Due to the open interaction between the Agent and the Large Language Model (LLM), the control over generated content is reduced to some extent, creating a risk of harmful content generation. Even if the LLM has built-in security protection mechanisms, it may still generate violent, pornographic, hate speech, and factual hallucination content through jailbreak attacks and adversarial vulnerabilities, or even leak sensitive information. Even if the user input is malicious or invalid, it will still trigger the costly LLM inference and retrieval process, posing a risk of abuse and causing a huge waste of computing resources.

[0011] 4. Unnatural interaction and poor real-time performance: Most systems are text-based and fail to achieve end-to-end voice interaction. Furthermore, due to architectural latency issues, they struggle to support smooth, real-time streaming voice output, resulting in a poor user experience.

[0012] 5. Multi-agent security challenges: In multi-agent environments, existing systems lack security safeguards for communication and data exchange between agents, making them vulnerable to man-in-the-middle attacks, data corruption, or privilege escalation attacks, which affect the overall security and reliability of the system.

[0013] Therefore, the design and improvement of voice-based intelligent customer service systems and methods are needed. Summary of the Invention

[0014] To address the aforementioned problems, this invention provides a secure voice-based intelligent customer service system based on a multi-agent architecture, comprising an ASR (Automatic Speech Recognition) module, a multi-layered security barrier, a main retrieval agent, a specialized retrieval agent cluster, a parallel retrieval and result aggregation module, an LLM (Limited Language Modeling) synthesis engine, and a TTS (Text-to-Speech) module.

[0015] The speech recognition module (ASR) converts user voice input into text-based user queries.

[0016] The multi-layered safety railing is used to provide layered security protection for key aspects of the secure voice intelligent customer service system.

[0017] The main retrieval agent is used to receive user queries after they have been filtered by the input security barrier layer, and to perform intent recognition and task decomposition.

[0018] The specialized retrieval agent cluster is used to call the most suitable LLM base from the base library to perform its exclusive task based on its task characteristics and performance requirements.

[0019] The parallel retrieval and result aggregation module is used to aggregate the retrieval results of the specialized retrieval agent cluster to obtain multi-source information;

[0020] The LLM synthesis engine is used to integrate information, perform logical reasoning and natural language generation on the multi-source information, and output a comprehensive answer that is clear in structure, semantically coherent and context-related.

[0021] The text-to-speech (TTS) module is used to convert the comprehensive answer into natural speech using text-to-speech (TTS) technology.

[0022] As a further explanation of the present invention, the multi-layer safety barrier includes, but is not limited to, input layer protection, intermediate layer protection, and output layer protection, wherein,

[0023] The input layer protection is used to provide security protection at the system input level;

[0024] The intermediate layer protection is used to provide security protection at the intermediate level of the system;

[0025] The output layer protection is used to provide security protection at the system output level.

[0026] Furthermore, the specialized retrieval agent cluster includes structured data agents, semantic retrieval agents, and real-time information agents, wherein,

[0027] The structured data agent is used to process queries involving structured data. It uses Text-to-SQL technology to convert natural language queries into SQL statements, access relational databases (such as PostgreSQL and MySQL), and obtain accurate results.

[0028] The semantic retrieval agent is used to process queries involving structured data. It uses Text-to-SQL technology to convert natural language queries into SQL statements, access relational databases (such as PostgreSQL and MySQL), and obtain accurate results.

[0029] The real-time information agent is used to obtain real-time public information (such as weather, stock prices, and policy updates) by calling external APIs or performing web searches (such as Google Search API and news API).

[0030] Furthermore, the various agents in the specialized retrieval agent cluster work in parallel and are logically isolated from the main retrieval agent.

[0031] On the other hand, the present invention also provides a secure voice-based intelligent customer service interaction method based on a multi-agent architecture, comprising the following steps:

[0032] S1, user voice input, the ASR module converts voice into text, and security protection is provided at the input level;

[0033] S2, the main retrieval agent receives the user query, performs intent recognition and task decomposition, obtains several types of sub-tasks, and distributes the sub-tasks to the corresponding specialized retrieval agents;

[0034] S3, the specialized retrieval agent performs parallel retrieval of the corresponding sub-tasks to obtain retrieval results, aggregates the retrieval results to obtain multi-source information, and performs security protection at the intermediate level;

[0035] S4, perform information integration, logical reasoning and natural language generation on the multi-source information, and output a comprehensive answer that is clear in structure, coherent in semantics and relevant in context, and perform security protection at the output level;

[0036] S5, the comprehensive answer is converted into voice and played to the user to realize a secure voice-based intelligent customer service dialogue.

[0037] Furthermore, the security protection at the input level involves deploying a lightweight model to perform content security detection before the user queries and enters the main system. This model identifies and blocks content containing sensitive words, offensive language, unauthorized requests, or content that violates public order and good morals. If a risk is detected, the process is immediately terminated and a compliance prompt is returned to avoid the execution of subsequent high-cost modules.

[0038] Furthermore, the intermediate layer provides security protection by setting up a secure isolation proxy mechanism in the interaction between the main retrieval agent and the specialized retrieval agent. This mechanism logically isolates control plane information (system commands, prompts) from data plane information (tool response content). Data plane information is only used for planning and reasoning through the isolated proxy in necessary scenarios to prevent indirect injection attacks. At the same time, Guardrails detection is embedded in each necessary link (such as tool description and response content, the agent's final response to the user, and message passing between agents).

[0039] Furthermore, the output layer security protection involves performing a security review on the response content generated by the LLM synthesis engine to detect security risks such as privacy leaks, factual inconsistencies, violence, pornography, and hate speech; if any are found, content correction or blocking is triggered to ensure the final output is compliant and secure.

[0040] Furthermore, the input format for the multi-source information synthesis is: [Search Result 1] + [Search Result 2] + ... + Original User Query;

[0041] The search results are from various specialized search agents.

[0042] Furthermore, in S5, the TTS module converts the answer stream into speech and gradually plays the speech during the LLM generation process, significantly reducing user perception latency and improving the interactive experience.

[0043] The beneficial effects of this invention are:

[0044] Through the system architecture and methodological design of this invention, modularization of the processing flow, parallelization of retrieval tasks, and closed-loop security control are achieved. By independently calling multi-layered security barriers in the input, output, agent, and inference stages, the system effectively mitigates risks such as prompt injection, indirect injection attacks, harmful content generation, and information leakage. Sensitive information identification, access permission verification, and content compliance review are embedded in each stage of input, retrieval, information synthesis, and output to prevent sensitive data leakage and meet the application requirements of high-security business scenarios (such as finance, telecommunications, and government services). By employing a collaborative approach between a main retrieval agent and multiple specialized retrieval agents, complex queries are decomposed into parallelizable subtasks, enabling multi-source fusion and inference of structured, unstructured, and real-time information. Semantic routing is also utilized. The system employs a routing and task decomposition mechanism to achieve dynamic task allocation and load balancing, significantly improving system processing efficiency and scalability. Security barriers are deployed at the system input layer, agent collaboration layer, and output layer to achieve multi-layered closed-loop security protection, including input security filtering, agent-to-agent communication isolation, and output content compliance review. The protection mechanism covers various security threats such as prompt injection, unauthorized access, privacy breaches, hallucinations, and the generation of hate and illegal content. A "control plane-data plane isolation" design ensures that the main retrieval agent does not directly access sensitive data, effectively reducing the attack surface. A modular and scalable architecture can be adopted, with all agent modules supporting plug-and-play functionality. Adding or replacing data sources or agents does not require large-scale modifications to the main system structure. Both the retrieval agent and security barriers are expandable, facilitating future integration with more industry scenarios (such as healthcare, finance, and government). Combined with high-performance ASR and TTS modules, a closed-loop interaction is achieved from voice input, semantic understanding, multi-source information integration to streaming voice output, supporting real-time voice recognition and streaming playback, significantly reducing response latency and improving user experience. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the secure voice-based intelligent customer service system and method based on a multi-agent architecture, as described in this invention.

[0046] Figure 2 The following is a schematic diagram illustrating the engineering design of this invention. Detailed Implementation

[0047] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0048] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", "first", "second", etc., indicate the orientation, position or order relationship based on the orientation, position or order relationship shown in the drawings, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0049] The invention discloses a secure voice intelligent customer service system based on a multi-agent architecture, including an ASR speech recognition module, a multi-layer security barrier, a main retrieval agent, a professional retrieval agent cluster, a parallel retrieval and result aggregation module, an LLM synthesis engine, and a TTS speech synthesis module.

[0050] The speech recognition module (ASR) converts user voice input into text-based user queries; preferably, it supports real-time streaming speech recognition to ensure low-latency response. This module integrates voice activity detection (VAD) and noise suppression to improve the quality of voice input.

[0051] The multi-layered safety railing is used to provide layered security protection for key aspects of the secure voice intelligent customer service system.

[0052] The main retrieval agent is used to receive user queries after they have been filtered by the input security barrier layer, and to perform intent recognition and task decomposition.

[0053] The specialized retrieval agent cluster is used to call the most suitable LLM base from the base library to perform its exclusive task based on its task characteristics and performance requirements.

[0054] The parallel retrieval and result aggregation module is used to aggregate the retrieval results of the specialized retrieval agent cluster to obtain multi-source information;

[0055] The LLM synthesis engine is used to integrate information, perform logical reasoning and natural language generation on the multi-source information, and output a comprehensive answer that is clear in structure, semantically coherent and context-related.

[0056] The text-to-speech (TTS) module is used to convert the comprehensive answer into natural speech using text-to-speech (TTS) technology.

[0057] The secure voice intelligent customer service system of this invention is a voice intelligent customer service system based on a multi-agent retrieval augmented generation (Multi-Agent RAG) architecture and incorporating a security-first design. The system adopts a modular design, coordinating multiple specialized agents to process complex queries in parallel through a main retrieval agent. Combined with multi-layered security safeguards (including input / output security, inter-agent security, and data privacy protection), it ensures end-to-end secure intelligent interaction from voice input to streaming voice output.

[0058] Specifically, this embodiment employs a multi-layered security barrier, deploying layered security protection mechanisms at multiple key stages, including user input, collaboration within the intelligent agent, and output to the user, to mitigate security risks such as jailbreak attacks, prompt injection attacks, generation of harmful content, and information leakage. The protection mechanisms include, but are not limited to, input layer protection, intermediate layer protection, and output layer protection.

[0059] The input layer protection is used to provide security protection at the system input level;

[0060] The intermediate layer protection is used to provide security protection at the intermediate level of the system;

[0061] The output layer protection is used to provide security protection at the system output level.

[0062] In this embodiment, the specialized retrieval agent cluster includes, but is not limited to, three types of agents: structured data agents, semantic retrieval agents, and real-time information agents. These agents work in parallel to improve retrieval efficiency. Each agent, based on its task characteristics and performance requirements, calls the most suitable LLM framework from the framework library to execute its specific task (such as SQL generation, semantic search, and API call parameter generation).

[0063] The structured data agent is used to process queries involving structured data. It uses Text-to-SQL technology to convert natural language queries into SQL statements, access relational databases (such as PostgreSQL and MySQL), and obtain accurate results.

[0064] The semantic retrieval agent is used to process queries involving structured data. It uses Text-to-SQL technology to convert natural language queries into SQL statements, access relational databases (such as PostgreSQL and MySQL), and obtain accurate results.

[0065] The real-time information agent is used to obtain real-time public information (such as weather, stock prices, and policy updates) by calling external APIs or performing web searches (such as Google Search API and news API).

[0066] In this embodiment, the various agents in the specialized retrieval agent cluster work in parallel and are logically isolated from the main retrieval agent. The specialized retrieval agent, acting as an isolated AI agent, is responsible for processing data surface content (such as tool response data) and is logically isolated from the main retrieval agent. Agents only transmit necessary structured data (such as JSON-formatted search results) to each other, avoiding direct exposure of raw data surface content to the main agent, thereby controlling the risk of indirect injection attacks. All agents access their corresponding data sources or service interfaces through a tool calling mechanism.

[0067] Based on the above system architecture of the present invention, the steps of the secure voice intelligent customer service interaction method based on a multi-agent architecture of the present invention are as follows:

[0068] S1, user voice input, is converted to text via the ASR module, providing security protection at the input level. In practical applications, this is implemented through the ASR speech recognition module, using the open-source model Paraformer. Paraformer is an end-to-end speech recognition model based on non-autoregressive (NAR) modeling. Its core is to significantly improve inference speed (more than 10 times) by generating output tokens in parallel, while maintaining recognition accuracy comparable to autoregressive (AR) models. This model uses a CIF-based predictor to dynamically estimate the number of output tokens and generate acoustic embeddings. It introduces real label information through a GLM sampler to enhance context-dependent modeling and combines it with MWER loss for optimization. It achieves near-state-of-the-art AR model performance on AISHELL and large-scale industrial corpora, significantly reducing substitution errors and making it suitable for latency-sensitive high-concurrency speech recognition scenarios.

[0069] S2, the main retrieval agent receives user queries, performs intent recognition and task decomposition, obtains several types of sub-tasks, and distributes the sub-tasks to the corresponding specialized retrieval agents. The main retrieval agent, as the brain of this invention system, has the core responsibility of routing and coordination. An integrated input protection layer performs real-time, low-cost content security detection on user input. The main retrieval agent strictly adheres to the principle of control plane and data plane isolation, performing intent recognition and task decomposition only based on system prompts and tool descriptions, avoiding direct processing of data plane content and reducing the risk of indirect injection attacks from the source.

[0070] S3, the specialized retrieval agent performs parallel retrieval on the corresponding sub-tasks to obtain retrieval results, aggregates the retrieval results to obtain multi-source information, and performs security protection at the intermediate layer. In practical applications, various types of specialized retrieval agents constitute the multi-agent retrieval cluster of this invention, for example:

[0071] Structured data broker: Utilizes Text-to-SQL technology (such as LLM+SQL parser) to convert natural language queries into precise SQL query statements, and connects to databases such as PostgreSQL and MySQL;

[0072] Unstructured data proxy: Based on vector databases (such as Chroma, Milvus), it uses the embedding model to vectorize queries and documents, and retrieves information from documents such as PDF and Word through semantic similarity retrieval;

[0073] Real-time information broker: Obtain real-time news, weather, stock prices, and other information by calling third-party APIs (such as SerperDev, Google Search API) or web crawlers.

[0074] All specialized retrieval agents operate as isolated AI agents, and are monitored by multiple layers of security barriers (intermediate isolation layers) when processing data surface content to ensure that their output is only structured data (such as JSON, text fragments) and is transmitted to the main agent through a secure channel.

[0075] S4, the multi-source information is integrated, logically reasoned, and natural language generated to output a comprehensive answer that is clearly structured, semantically coherent, and context-sensitive, with security protection implemented at the output level. In practical applications, the central synthesis module constructs a context-rich Prompt from the heterogeneous data (JSON, text fragments, lists, etc.) returned by multiple agents, inputs it to the LLM, and instructs it to summarize, generalize, and naturalize the data to generate the final answer. The information fusion strategy of the LLM synthesis engine in this embodiment is as follows:

[0076] Input format: [Search result 1] + [Search result 2] + ... + User's original query;

[0077] See appendix Figure 2 As shown, the design employs Prompt Engineering.

[0078] S5, the comprehensive answer is converted into voice and played to the user to realize a secure voice-based intelligent customer service dialogue.

[0079] In this embodiment, Guardrails is deployed across multiple stages, including input, LLM planning, memory data, tool calls, and response output, to protect against the following threats: harmful content generation (violence, pornography, hate speech, etc.); jailbreak attacks (bypassing model security mechanisms); hallucinations (factual inconsistencies, logical inconsistencies); and information leakage (privacy or confidentiality breaches). Through layered protection, contextual constraints, content moderation, and input / output filtering mechanisms, the robustness and controllability of the AI ​​system are achieved.

[0080] Specifically, the security protection at the input level involves deploying a lightweight model to perform content security detection before the user queries and enters the main system. This model identifies and blocks content containing sensitive words, offensive language, unauthorized requests, or content that violates public order and good morals. If a risk is detected, the process is immediately terminated and a compliance prompt is returned to avoid the execution of subsequent high-cost modules.

[0081] The intermediate layer security protection involves setting up a secure isolation proxy mechanism in the interaction between the main retrieval agent and the specialized retrieval agent. This mechanism logically isolates control plane information (system commands, prompts) from data plane information (tool response content). Data plane information is only used for planning and reasoning through the isolated proxy in necessary scenarios to prevent indirect injection attacks. At the same time, Guardrails detection is embedded in each necessary link (such as tool description and response content, the agent's final response to the user, and message passing between agents).

[0082] The security protection at the output level involves a security review of the response content generated by the LLM synthesis engine to detect risks such as privacy leaks, factual inconsistencies, violence, pornography, and hate speech. If such risks are found, content correction or blocking is triggered to ensure the final output is compliant and secure.

[0083] In this embodiment, the synthesis of the response speech is implemented by the Text-to-Speech (TTS) module, using CosyVoice 2. CosyVoice 2 is a new generation multilingual speech synthesis system launched by Alibaba. Its core principle is based on a Large Language Model (LLM) to achieve decoupling and progressive decoding of semantic and acoustic information, thereby uniformly supporting high-quality non-streaming and low-latency streaming synthesis. Specifically, the system first converts the input text into discrete semantic speech tags through a text-speech language model, effectively separating the semantic content of the text. Subsequently, an innovative block-aware streaming matching model generates Mel spectra causally and block-by-block based on these semantic tags, and finally outputs the waveform through a vocoder. This separation modeling approach avoids the interference of acoustic details on the language model, while the block-aware mechanism and flexible attention masking strategy ensure low latency in streaming generation. At the same time, the unified model architecture eliminates the need to train streaming and non-streaming models separately, simplifying deployment and enabling it to have powerful multilingual capabilities and rich natural language command control functions.

[0084] Based on the design of the above system and method, a complete process of secure voice intelligent customer service interaction based on a multi-agent architecture in this embodiment is as follows:

[0085] Users initiate voice questions (e.g., "Could you check the status of order #12345 from last month, and recommend similar products as well");

[0086] The ASR module converts speech to text;

[0087] The input protection layer of the multi-layer safety barrier undergoes a content security check, and passage is only allowed after confirming there is no risk.

[0088] The main search agent analyzed the query and identified two sub-tasks: "order status query" and "product recommendation".

[0089] The master agent distributes the "order status" task to the structured data agent and the "recommendation" task to the unstructured data agent.

[0090] The two agents perform the retrieval and interact through the middle layer guardrail:

[0091] The structured data broker queries the order database via Text-to-SQL;

[0092] Unstructured data brokers use order data to find similar products;

[0093] The search results are returned to the aggregation layer;

[0094] The LLM synthesis engine integrates order information and recommendation lists to generate answers;

[0095] Verify the compliance and authenticity of the output protective layer test results;

[0096] The TTS module converts the answer stream into speech and broadcasts it to the user in real time.

[0097] Compared with existing single-agent RAG systems, the present invention has the following technical advantages:

[0098] 1. Significantly Improved Processing Efficiency and Concurrency: Single-agent systems employ serial task scheduling, making it difficult to process multi-source heterogeneous queries in parallel, which can easily become a system bottleneck. This invention utilizes a multi-agent parallel retrieval architecture to simultaneously process subtasks within complex queries, significantly reducing response latency and increasing system throughput.

[0099] 2. Enhanced complex query processing and multi-step reasoning capabilities: This invention uses a main retrieval agent for semantic routing and task decomposition, which can flexibly schedule multiple specialized agents to collaboratively process queries that require multi-step reasoning (such as simultaneously querying databases, document libraries, and real-time APIs), overcoming the problem of insufficient information integration and fragmented answers in complex scenarios by a single agent.

[0100] 3. High system scalability and modularity: Existing systems generally suffer from tight coupling between data sources and functions, resulting in poor scalability. The proxy module and security barrier mechanism of this invention both support plug-and-play functionality, enabling rapid integration with new data sources, new business logic, or security policies through standardized interfaces, facilitating continuous upgrades and maintenance for enterprises.

[0101] 4. Multi-layered security barriers achieve end-to-end protection: Existing solutions typically only set up a single security detection link at the input or output end, which cannot effectively defend against prompt word injection, unauthorized access, or intermediate communication attacks. This invention adopts triple security protection—input layer, intermediate isolation layer, and output layer—forming a closed-loop mechanism, fundamentally reducing the attack surface and blocking the flow of high-risk content in real time. The introduction of a control plane-data plane isolation design isolates potential security risks from core scheduling logic, significantly enhancing the system's anti-attack capability and controllability.

[0102] 5. Supports end-to-end voice interaction and streaming output for a superior user experience: This invention integrates ASR and streaming TTS to achieve a full voice interaction process and supports speech synthesis while LLM generates text, significantly reducing perceived latency for users. Most existing systems still rely primarily on text interaction, with weak voice support or high latency.

[0103] The above description only illustrates preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. In short, all variations made within the scope of the independent claims of the present invention are within the scope of protection of the present invention.

Claims

1. A secure voice-based intelligent customer service system based on a multi-agent architecture, characterized in that: This includes an ASR (Automatic Speech Recognition) module, a multi-layered security barrier, a main retrieval agent, a specialized retrieval agent cluster, a parallel retrieval and result aggregation module, an LLM (Limited Language Modeling) synthesis engine, and a TTS (Text-to-Speech) module. The speech recognition module (ASR) converts user voice input into text-based user queries. The multi-layered safety railing is used to provide layered security protection for key aspects of the secure voice intelligent customer service system. The main retrieval agent is used to receive user queries after they have been filtered by the input security barrier layer, and to perform intent recognition and task decomposition. The specialized retrieval agent cluster is used to call the most suitable LLM base from the base library to perform its exclusive task based on its task characteristics and performance requirements. The parallel retrieval and result aggregation module is used to aggregate the retrieval results of the specialized retrieval agent cluster to obtain multi-source information; The LLM synthesis engine is used to integrate information, perform logical reasoning and natural language generation on the multi-source information, and output a comprehensive answer that is clear in structure, semantically coherent and context-related. The text-to-speech (TTS) module is used to convert the comprehensive answer into natural speech using text-to-speech (TTS) technology.

2. The secure voice intelligent customer service system based on a multi-agent architecture as described in claim 1, characterized in that: The multi-layer safety barrier includes, but is not limited to, input layer protection, intermediate layer protection, and output layer protection, wherein... The input layer protection is used to provide security protection at the system input level; The intermediate layer protection is used to provide security protection at the intermediate level of the system; The output layer protection is used to provide security protection at the system output level.

3. The secure voice intelligent customer service system based on a multi-agent architecture as described in claim 1, characterized in that: The specialized retrieval agent cluster includes structured data agents, semantic retrieval agents, and real-time information agents, among which, The structured data agent is used to process queries involving structured data. It uses Text-to-SQL technology to convert natural language queries into SQL statements, access relational databases, and obtain accurate results. The semantic retrieval agent is used to process queries involving structured data. It uses Text-to-SQL technology to convert natural language queries into SQL statements, accesses relational databases, and obtains accurate results. The real-time information agent is used to obtain real-time public information by calling external APIs or performing web searches.

4. The secure voice intelligent customer service system based on a multi-agent architecture as described in claim 3, characterized in that: The specialized retrieval agent cluster has various agents working in parallel and is logically isolated from the main retrieval agent.

5. A secure voice-based intelligent customer service interaction method based on a multi-agent architecture, characterized in that, Includes the following steps: S1, user voice input, the ASR module converts voice into text, and security protection is provided at the input level; S2, the main retrieval agent receives the user query, performs intent recognition and task decomposition, obtains several types of sub-tasks, and distributes the sub-tasks to the corresponding specialized retrieval agents; S3, the specialized retrieval agent performs parallel retrieval of the corresponding sub-tasks to obtain retrieval results, aggregates the retrieval results to obtain multi-source information, and performs security protection at the intermediate level; S4, perform information integration, logical reasoning and natural language generation on the multi-source information, and output a comprehensive answer that is clear in structure, coherent in semantics and relevant in context, and perform security protection at the output level; S5, the comprehensive answer is converted into voice and played to the user to realize a secure voice-based intelligent customer service dialogue.

6. The secure voice intelligent customer service interaction method based on a multi-agent architecture according to claim 5, characterized in that: The security protection at the input level involves deploying a lightweight model to perform content security detection before the user queries and enters the main system. This model identifies and blocks content containing sensitive words, offensive language, unauthorized requests, or content that violates public order and good morals. If a risk is detected, the process is immediately terminated and a compliance prompt is returned.

7. The secure voice intelligent customer service interaction method based on a multi-agent architecture according to claim 5, characterized in that: The intermediate layer is protected by setting up a secure isolation proxy mechanism in the interaction between the main retrieval proxy and the specialized retrieval proxy. This mechanism logically isolates the control plane information from the data plane information. The isolated proxy uses the data plane information for planning and reasoning to prevent indirect injection attacks. At the same time, Guardrails detection is embedded in several stages.

8. The secure voice intelligent customer service interaction method based on a multi-agent architecture according to claim 5, characterized in that: The security protection at the output level involves performing a security review on the response content generated by the LLM synthesis engine to detect any security risks; if any are found, content correction or blocking is triggered.

9. The secure voice-based intelligent customer service interaction method based on a multi-agent architecture according to claim 5, characterized in that: The input format for the multi-source information synthesis is: [Search Result 1] + [Search Result 2] + ... + Original User Query; The search results are from various specialized search agents.

10. The secure voice intelligent customer service interaction method based on a multi-agent architecture according to claim 5, characterized in that: In S5, the TTS module converts the answer stream into speech and gradually plays the speech during the LLM generation process.

Images