A multi-agent question-answering system and method based on large language model self-driving routing

Abstract

The application discloses a kind of multi-agent question and answer system and question and answer method based on large language model self-driving routing, the system includes input receiving module, candidate Agent set construction module, LLM routing decision module, three-level decision module, Agent execution module and result output and log module.Candidate Agent set construction module collects the name and natural language function description of all registered Agent and generates candidate set;LLM routing decision module will candidate Agent information and user question be assembled as routing prompt word and be submitted to large language model, and the returned result is normalized after output analysis unit format and is output;Three-level decision module sequentially executes complete equality matching, containing matching and score bottom-up three progressive processing to the routing result after analysis, determines target Agent;Agent execution module calls target Agent to execute question and answer task;Result output and log module output legal reply and record routing decision log.The application effectively improves the response reliability and user experience of question and answer system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-agent question-answering system, question-answering method, terminal, and computer-readable storage medium based on a large language model-driven self-driving routing. Background Technology

[0002] When faced with various types of questions, such as legal retrieval, contract consultation, case analysis, and multimodal document recognition, existing intelligent question-answering systems typically employ two methods for request distribution (routing): one is hard-coded rule routing, which assigns user questions to fixed processing modules through keyword matching, regular expressions, or pre-trained intent classification models; the other is direct answering by a single large model without distribution, where a general-purpose model answers all questions.

[0003] However, rule-based or keyword routing requires manual maintenance of the rule base, making it impossible to truly understand semantic intent. It is prone to misassignment when encountering diverse expressions or cross-domain issues. Furthermore, a single large model cannot dynamically select the best option across multiple vertical subdomains, and it lacks fallback error correction to ensure the correctness of the routing. The routing process is uncontrollable and uninterpretable. Once the routing process fails (intent not recognized or service abnormality), it often results in a direct error or failure, lacking deterministic degradation measures.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a multi-agent question-answering system and method based on self-driven routing using a large language model, aiming to solve the problems of uncontrollable LLM routing results in existing question-answering systems, which easily lead to inaccurate agent matching, inability to obtain accurate answers to user questions, and low availability of question-answering services.

[0006] To achieve the above objectives, the present invention provides a multi-agent question answering system based on self-driven routing using a large language model. The multi-agent question answering system includes: an input receiving module, a candidate agent set construction module, an LLM routing decision module, a three-level decision module, an agent execution module, and a result output and logging module. The input receiving module, the candidate agent set construction module, the LLM routing decision module, the three-level decision module, the agent execution module, and the result output and logging module are connected sequentially. The input receiving module is used to receive the question text input by the user, perform semantic standardization processing on the question text, and output standard question text; The candidate agent set construction module is used to receive the standard question text and collect the registration information of all registered agents in the system to generate a candidate agent set; The LLM routing decision module is used to obtain the candidate agent set and the standard question text, assemble the candidate agent set and the standard question text into routing prompt words, output the routing prompt words to the large language model, and receive the routing results returned by the large language model; The three-level decision module is used to receive the routing result, perform first-level matching verification, second-level matching verification and score fallback processing on the routing result in sequence, and output the target Agent identifier; The Agent execution module is used to receive the target Agent identifier, invoke the target Agent corresponding to the target Agent identifier to perform the question-and-answer task, and receive the task execution result returned by the target Agent; The result output and log module is used to receive the task execution result, output the task execution result as a legal response to the user, and record the routing decision log.

[0007] Optionally, in the multi-agent question-answering system based on self-driven routing of a large language model, the candidate agent set construction module includes: an agent information storage unit, a set generation unit, and a single candidate routing unit; The Agent information storage unit is used to store the registration information of each Agent, which includes the Agent name and natural language function description; The set generation unit is used to receive the standard question text output by the input receiving module, and traverse the Agent information storage unit to read the registration information of all registered Agents and generate a candidate Agent set. The single candidate routing unit is used to determine the number of agents in the candidate agent set. When there is only one agent in the candidate agent set, the single candidate routing unit determines the agent as the target agent, and skips the LLM routing decision module and the third-level decision module to directly input the target agent identifier into the agent execution module.

[0008] Optionally, in the multi-agent question-answering system based on large language model self-driven routing, the LLM routing decision module includes: a prompt word assembly unit and a model invocation unit; The prompt word assembly unit is used to obtain the candidate Agent set and the standard question text, extract the Agent name and natural language function description of each Agent from the candidate Agent set, and assemble all the Agent names and all the natural language function descriptions with the standard question text into routing prompt words according to a preset template; The model invocation unit is used to submit the routing prompt words to the general large language model and receive the routing results returned by the general large language model.

[0009] Optionally, in the multi-agent question-answering system based on a large language model for self-driven routing, the routing prompt explicitly lists the names of all candidate agents as an output whitelist and includes constraint instructions. The constraint instruction is used to force the large language model to select only one Agent name from the output whitelist as output, and to prohibit the output of other text, punctuation or explanation.

[0010] Optionally, in the multi-agent question-answering system based on large language model self-driven routing, the LLM routing decision module further includes: an output parsing unit; The output parsing unit is connected to the model calling unit and is used to perform format normalization processing on the routing result after receiving the routing result returned by the large language model, generate parsed routing result, and output the parsed routing result to the third-level decision module. The format normalization process includes removing spaces, removing punctuation marks, and normalizing case.

[0011] Optionally, in the multi-agent question-answering system based on self-driven routing of a large language model, the three-level decision module includes: a first-level matching unit, a second-level matching unit, and a scoring fallback unit; The first-level matching unit is used to receive the routing result, and perform a complete equality matching verification between the routing result and the agent name of each agent in the candidate agent set. If the routing result is completely equal to the agent name of any agent, then the agent is determined as the target agent and the target agent identifier is output. The secondary matching unit is used to perform an inclusion-matching check on the routing result and the agent name of each agent in the candidate agent set in sequence when the primary matching unit fails to match any agent. If the routing result contains the agent name of any agent, or the agent name of any agent contains the routing result, then the agent is determined as the target agent and the target agent identifier is output. The scoring fallback unit is used to calculate a matching score for each Agent in the candidate Agent set when neither the first-level matching unit nor the second-level matching unit matches any Agent, and to take the Agent with the highest score as the target Agent and output the target Agent identifier.

[0012] Optionally, in the multi-agent question-answering system based on a large language model self-driven routing, the scoring fallback unit includes: a keyword extraction subunit, a score calculation subunit, and an output selection subunit; The keyword extraction subunit is connected to the input receiving module and is used to extract keywords from the standard question text to generate a keyword set. The scoring calculation subunit is connected to the keyword extraction subunit and the candidate agent set construction module, respectively, and is used to perform weighted scoring calculation on each agent in the candidate agent set based on the keyword; The selection output subunit is connected to the score calculation subunit and is used to compare the scores of each Agent, select the Agent with the highest score as the target Agent, and output the target Agent identifier.

[0013] Optionally, in the multi-agent question-answering system based on a large language model self-driven routing, the result output and log module includes: a response output unit and a log recording unit; The response output unit is used to receive the task execution result and output the task execution result to the user in natural language form; The log recording unit is used to record the routing results, the target Agent identifier, the matching level information executed by the three-level decision module, and the task execution status of the target Agent, so as to generate a routing decision log.

[0014] Furthermore, to achieve the above objectives, this invention also provides a question-answering method for a multi-agent question-answering system based on a large language model self-driven routing, wherein the question-answering method includes: The input receiving module receives the question text input by the user, performs semantic standardization processing on the question text, and outputs standard question text; The candidate agent set construction module receives the standard question text and collects the registration information of all registered agents in the system to generate a candidate agent set; The LLM routing decision module obtains the candidate agent set and the standard question text, assembles the candidate agent set and the standard question text into routing prompt words, outputs the routing prompt words to the large language model, and receives the routing results returned by the large language model; The third-level decision module receives the routing result, performs first-level matching verification, second-level matching verification, and score fallback processing on the routing result in sequence, and outputs the target Agent identifier; The Agent execution module receives the target Agent identifier, invokes the target Agent corresponding to the target Agent identifier to perform the question-and-answer task, and receives the task execution result returned by the target Agent; The result output and log module is used to receive the task execution result, output the task execution result as a legal response to the user, and record the routing decision log.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a question-answering program of a multi-agent question-answering system based on a large language model self-driven routing, which is stored in the memory and can run on the processor. When the question-answering program of the multi-agent question-answering system based on a large language model self-driven routing is executed by the processor, it implements the steps of the question-answering method of the multi-agent question-answering system based on a large language model self-driven routing as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a question-answering program of a multi-agent question-answering system based on a large language model self-driven routing, and when the question-answering program of the multi-agent question-answering system based on a large language model self-driven routing is executed by a processor, it implements the steps of the question-answering method of the multi-agent question-answering system based on a large language model self-driven routing as described above.

[0017] In this invention, the system includes an input receiving module, a candidate agent set construction module, an LLM routing decision module, a three-level decision module, an agent execution module, and a result output and logging module. These modules are connected sequentially. The candidate agent set construction module collects the names and natural language function descriptions of all registered agents to generate a candidate set. The LLM routing decision module assembles the candidate agent information and the user's question into routing prompts, submits them to the large language model, and outputs the returned results after format normalization by the output parsing unit. The three-level decision module performs three progressive processing steps on the parsed routing results: complete equality matching, inclusion matching, and a fallback scoring, to determine the target agent. The agent execution module calls the target agent to perform the question-and-answer task. The result output and logging module outputs legal responses and records the routing decision log. This invention uses LLM semantic routing and a fallback scoring system to form a double-insurance mechanism, improving system robustness while ensuring routing accuracy. It effectively avoids irrelevant answers or service unavailability caused by routing errors, significantly improving the response reliability and user experience of the question-and-answer system. Attached Figure Description

[0018] Figure 1 This is the overall architecture diagram of the multi-agent question-answering system based on self-driven routing of a large language model according to the present invention; Figure 2 This is a schematic diagram of the dual-insurance decision-making system of the multi-agent question-answering system based on self-driven routing of a large language model according to the present invention; Figure 3 This is a decision sequence diagram of the multi-agent question answering system based on large language model self-driven routing in this invention, which degrades to score fallback when LLM routing fails. Figure 4 This is a flowchart of a preferred embodiment of the question-answering method of the multi-agent question-answering system based on self-driven routing of a large language model according to the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] To address the problems in the existing technology, this embodiment provides a multi-agent question-answering system based on self-driven routing using a large language model, such as... Figure 1 As shown, the multi-agent question answering system based on large language model self-driven routing includes: an input receiving module, a candidate agent set construction module, an LLM routing decision module, a three-level decision module, an agent execution module, and a result output and log module. The input receiving module, the candidate agent set construction module, the LLM routing decision module, the three-level decision module, the agent execution module, and the result output and log module are connected in sequence. The input receiving module is used to receive the question text input by the user, perform semantic standardization processing on the question text, and output standard question text; The candidate agent set construction module is used to receive the standard question text and collect the registration information of all registered agents in the system to generate a candidate agent set; The LLM routing decision module is used to obtain the candidate agent set and the standard question text, assemble the candidate agent set and the standard question text into routing prompt words, output the routing prompt words to the large language model, and receive the routing results returned by the large language model; The three-level decision module is used to receive the routing result, perform first-level matching verification, second-level matching verification and score fallback processing on the routing result in sequence, and output the target Agent identifier; The Agent execution module is used to receive the target Agent identifier, invoke the target Agent corresponding to the target Agent identifier to perform the question-and-answer task, and receive the task execution result returned by the target Agent; The result output and log module is used to receive the task execution result, output the task execution result as a legal response to the user, and record the routing decision log.

[0021] It is understood that this invention provides a multi-agent question-answering system based on a large language model-driven self-driving routing system, comprising, in sequence, an input receiving module, a candidate agent (artificial intelligence agent) set construction module, an LLM (Large Language Model) routing decision module, a three-level decision module, an agent execution module, and a result output and log module; the input receiving module receives and standardizes user legal questions; the candidate agent set construction module collects the names and natural language function descriptions of all registered agents to generate a candidate set; the LLM routing decision module assembles candidate agent information and user questions into routing prompts and submits them to the large language model, and performs format normalization processing on the returned results before outputting the parsed routing results; the three-level decision module performs three progressive processing steps on the parsed routing results: complete equality matching, inclusion matching, and score fallback, to determine the target agent; the agent execution module calls the target agent to perform the question-answering task; and the result output and log module outputs legal answers and records routing decision logs. This invention employs a dual-insurance mechanism of LLM semantic routing and scoring fallback, which provides reliable fault tolerance and degradation protection when the output of a large language model is uncontrollable, effectively improving the accuracy of routing decisions and the robustness of the system in a multi-agent question answering system.

[0022] Furthermore, the candidate agent set construction module includes: an agent information storage unit, a set generation unit, and a single candidate route unit; The Agent information storage unit is used to store the registration information of each Agent, which includes the Agent name and natural language function description; The set generation unit is used to receive the standard question text output by the input receiving module, and traverse the Agent information storage unit to read the registration information of all registered Agents and generate a candidate Agent set. The single candidate routing unit is used to determine the number of agents in the candidate agent set. When there is only one agent in the candidate agent set, the single candidate routing unit determines the agent as the target agent, and skips the LLM routing decision module and the third-level decision module to directly input the target agent identifier into the agent execution module.

[0023] In this embodiment, the Agent registration information storage unit is used to store the registration information of each Agent. The registration information of each Agent includes the Agent's name and a natural language function description. After any Agent completes registration, its registration information is stored in the Agent registration information storage unit, and adding a new Agent does not require modifying any registration information of existing Agents, thus realizing flexible expansion of the system's routing capabilities.

[0024] The set generation unit is connected to the Agent registration information storage unit, and is used to traverse the Agent registration information storage unit, read the registration information of all registered Agents, generate a candidate Agent set, and provide the candidate Agent set as a candidate space for routing decisions to the subsequent routing module.

[0025] The single-candidate routing unit, connected to the set generation unit, is used to determine the number of Agents in the candidate Agent set. When the candidate Agent set contains only one Agent, the single-candidate direct routing unit directly identifies that Agent as the target Agent, passes the target Agent identifier to the Agent execution module, and skips the LLM routing decision module and the third-level decision module. This optimized branch avoids unnecessary large language model calls when the number of candidates is unique, thereby reducing system response latency and computational resource consumption.

[0026] Furthermore, the LLM routing decision module includes: a prompt word assembly unit and a model invocation unit; The prompt word assembly unit is used to obtain the candidate Agent set and the standard question text, extract the Agent name and natural language function description of each Agent from the candidate Agent set, and assemble all the Agent names and all the natural language function descriptions with the standard question text into routing prompt words according to a preset template; The model invocation unit is used to submit the routing prompt words to the general large language model and receive the routing results returned by the general large language model.

[0027] like Figure 2As shown, in this embodiment, the prompt word assembly unit is connected to the candidate Agent set construction module and is used to obtain the candidate Agent set and the standard question text. The prompt word assembly unit extracts the Agent name and natural language function description of each Agent from the candidate Agent set and assembles them according to a preset template to generate routing prompt words. The routing prompt words contain three components: the first part instructs the large language model to play the role of routing decision-maker and explains the goal and rules of routing decision; the second part lists the Agent name and natural language function description of each candidate Agent to form a selectable candidate list; the third part includes the standard question text and instructs the large language model to determine which Agent is most suitable for handling the current user's question based on the function description of each Agent. The preset template is a fixed-format prompt word framework, and only the candidate list and standard question text content are replaced each time it is called to ensure that the prompt word structure input to the large language model remains consistent.

[0028] The model invocation unit is connected to the prompt word assembly unit and is used to receive the routing prompt words generated by the prompt word assembly unit. The model invocation unit encapsulates the application programming interface (API) call logic of the large language model, submits the routing prompt words as input parameters to the large language model, and waits for the large language model to return the inference result. The large language model performs semantic understanding and decision reasoning based on the instructions and candidate information in the routing prompt words, and outputs a result. The model invocation unit receives this output result and outputs it as the routing result to the third-level decision module.

[0029] The output parsing unit, connected to the model invocation unit, performs format normalization processing on the routing result returned by the large language model after receiving it, generating a parsed routing result, and outputs the parsed routing result to the third-level decision module. The format normalization processing includes removing spaces, removing punctuation marks, and normalizing case. After format normalization, the parsed routing result is presented in a uniform plain text format, facilitating accurate string matching and verification by the subsequent third-level decision module. The output parsing unit, acting as a data adaptation layer between the LLM routing decision module and the third-level decision module, isolates the uncertainty of the large language model's output format, enabling subsequent modules to perform matching operations based on a unified input format.

[0030] Furthermore, the routing prompt explicitly lists the names of all candidate agents as an output whitelist and includes constraint instructions; The constraint instruction is used to force the large language model to select only one Agent name from the output whitelist as output, and to prohibit the output of other text, punctuation or explanation.

[0031] It is understood that the output whitelist in the routing prompt is a list of all candidate agents in the candidate agent set, presented one by one. The output whitelist explicitly limits the output range of the large language model to the agent names in this list; that is, the large language model can only select one agent from the candidate agent set and cannot generate any other names outside of it. The construction of the output whitelist is synchronized with the dynamic changes in the candidate agent set. Whenever the candidate agent set construction module generates a new candidate agent set, the prompt assembly unit regenerates the corresponding output whitelist based on that set.

[0032] The constraints in the routing prompts include output format constraints and output content constraints. The output format constraints force the large language model to output only the Agent name itself, without any other text, punctuation, or explanatory statements. The output content constraints force the large language model to select exactly one Agent name from the output whitelist as output; it cannot select multiple Agents or any name outside the output whitelist. These constraints are written in natural language into the routing prompts, and the large language model follows these instructions to complete the formatted output during inference.

[0033] The output whitelist and the constraint instructions work together to form a limiting mechanism for the output space of the large language model. The constraint instructions, at the semantic level, tell the large language model how it should output, while the output whitelist, at the content level, restricts what the large language model can output. When used together, the large language model, during reasoning, is clearly aware of both the output format requirements (outputting only names) and the range of options (limited to names in the whitelist). This simultaneously constrains the large language model's generation behavior in both semantic and content dimensions, effectively reducing the risk of output illusions or format drift, and allowing the routing results to be directly parsed and used without complex natural language understanding post-processing.

[0034] Furthermore, the LLM routing decision module also includes: an output parsing unit; The output parsing unit is connected to the model calling unit and is used to perform format normalization processing on the routing result after receiving the routing result returned by the large language model, generate parsed routing result, and output the parsed routing result to the third-level decision module. The format normalization process includes removing spaces, removing punctuation marks, and normalizing case.

[0035] In this embodiment, the output parsing unit is connected to the model invocation unit and is used to receive the routing results obtained by the model invocation unit from the large language model. Although the routing results returned by the large language model are required to output only the Agent name under the constraint instructions of the routing prompt words, in actual invocation, the output may still contain extra spaces, punctuation marks, inconsistent capitalization, or explanatory text. The output parsing unit, as an intermediate processing layer between the model invocation unit and the three-level decision module, is used to perform format normalization processing on the routing results, eliminate the inconsistency in format in the output of the large language model, and generate parsed routing results.

[0036] The format normalization process includes removing spaces, removing punctuation, and case normalization. Removing spaces means removing leading and trailing whitespace characters and extra spaces within the string in the routing result, making the output a continuous string. Removing punctuation means removing commas, periods, quotation marks, and other punctuation characters that may exist in the routing result to eliminate interference from punctuation in subsequent string matching. Case normalization means converting the routing result to lowercase to ensure that subsequent matching is not affected by case differences. After performing the above normalization process, the routing result is converted into a plain text format without spaces or punctuation, and entirely in lowercase. The three-level decision module uses the parsed routing result as the object of matching verification, rather than directly using the original routing result returned by the large language model. Because the parsed routing result has undergone format normalization, eliminating various format drift factors in the output of the large language model, the three-level decision module can achieve a higher matching success rate when performing exact equality matching and containment matching. This reduces the situation where matching failures due to format issues trigger fallback scoring, thereby improving the efficiency and stability of the overall routing decision.

[0037] Furthermore, the three-level decision-making module includes: a first-level matching unit, a second-level matching unit, and a scoring fallback unit; The first-level matching unit is used to receive the routing result, and perform a complete equality matching verification between the routing result and the agent name of each agent in the candidate agent set. If the routing result is completely equal to the agent name of any agent, then the agent is determined as the target agent and the target agent identifier is output. The secondary matching unit is used to perform an inclusion-matching check on the routing result and the agent name of each agent in the candidate agent set in sequence when the primary matching unit fails to match any agent. If the routing result contains the agent name of any agent, or the agent name of any agent contains the routing result, then the agent is determined as the target agent and the target agent identifier is output. The scoring fallback unit is used to calculate a matching score for each Agent in the candidate Agent set when neither the first-level matching unit nor the second-level matching unit matches any Agent, and to take the Agent with the highest score as the target Agent and output the target Agent identifier.

[0038] In this embodiment, the first-level matching unit is connected to the LLM routing decision module and is used to receive the parsed routing result output by the LLM routing decision module. The first-level matching unit obtains the candidate Agent set generated by the candidate Agent set construction module and performs a complete equality match check between the parsed routing result and the Agent name of each Agent in the candidate Agent set. The complete equality match means determining whether the parsed routing result is exactly the same as the Agent name of a certain Agent, including a character sequence comparison. If the two are completely identical, the match is considered successful. If the parsed routing result is completely equal to the Agent name of any Agent, the first-level matching unit identifies that Agent as the target Agent, outputs the target Agent identifier, the routing decision process ends, and the second-level matching unit and the scoring fallback unit are no longer triggered.

[0039] The secondary matching unit is connected to the primary matching unit and is triggered when the primary matching unit fails to match any Agent. The secondary matching unit obtains the candidate Agent set and performs an inclusion-matching check on the parsed routing result and the Agent name of each Agent in the candidate Agent set. This inclusion-matching check is a bidirectional inclusion judgment, meaning it checks whether the parsed routing result contains the Agent name of a certain Agent, or whether the Agent name of that Agent contains the parsed routing result; a match is considered successful if either condition is met. This bidirectional inclusion judgment is used to handle situations where the large language model output still has some format drift after processing by the output parsing unit, such as when the output result is a partial name fragment or contains extra characters, it can still match correctly. If the parsed routing result and the Agent name of any Agent satisfy the inclusion-matching condition, the secondary matching unit identifies that Agent as the target Agent, outputs the target Agent identifier, the routing decision process ends, and the scoring fallback unit is no longer triggered.

[0040] The scoring fallback unit is connected to both the primary matching unit and the secondary matching unit, and is triggered when neither the primary nor the secondary matching unit matches any Agent. Figure 3 As shown, if neither the first-level matching unit nor the second-level matching unit finds a match, it means that the parsed routing result cannot establish a correspondence with any candidate Agent through string matching. In this case, the third-level decision module degrades to a fallback scoring mode. The fallback scoring unit obtains the candidate Agent set and the standard question text, calculates a matching score for each Agent in the candidate Agent set, reflecting the suitability of each Agent with the current user question, selects the Agent with the highest score as the target Agent, and outputs the target Agent identifier. As the last level of the third-level decision module, the fallback scoring unit ensures that even when the LLM routing result is completely unavailable, the system can still select an Agent to respond to the user question through the scoring mechanism, avoiding service interruption.

[0041] The primary matching unit, the secondary matching unit, and the fallback scoring unit constitute a progressively yielding decision chain: the primary matching unit strives for precise matching, the secondary matching unit tolerates format drift, and the fallback scoring unit compensates through semantic scoring when matching completely fails. The process terminates upon success at the previous level and automatically flows to the next level if the previous level fails, until the target Agent is determined.

[0042] The scoring catch-all unit includes: a keyword extraction subunit, a score calculation subunit, and an output selection subunit; The keyword extraction subunit is connected to the input receiving module and is used to extract keywords from the standard question text to generate a keyword set. The scoring calculation subunit is connected to the keyword extraction subunit and the candidate agent set construction module, respectively, and is used to perform weighted scoring calculation on each agent in the candidate agent set based on the keyword; The selection output subunit is connected to the score calculation subunit and is used to compare the scores of each Agent, select the Agent with the highest score as the target Agent, and output the target Agent identifier.

[0043] The weighted scoring calculation includes: if the Agent's natural language function description indicates that the Agent can handle the question type involved in the standard question text, then the Agent is given a first preset score (preferably 50 points); for each keyword in the keyword set, if the keyword matches the Agent's natural language function description, then each matching keyword is given a second preset score (preferably 10 points); if the Agent name matches the standard question text, then an additional third preset score (preferably 20 points) is given; the maximum score for a single Agent is 100 points.

[0044] Understandably, the keyword extraction subunit is connected to the input receiving module and is used to acquire the standard question text output by the input receiving module, and extract keywords from the standard question text to generate a keyword set. The keyword extraction method is as follows: first, the standard question text is segmented into independent word units; then, stop word filtering is performed on the segmentation results to remove stop words that do not contribute to the semantic meaning; finally, terminology recognition for the corresponding question is performed on the filtered words, and the recognized terms are included as keywords in the keyword set.

[0045] Taking legal issues as an example, the filtered words are subjected to legal terminology recognition. This recognition is based on a pre-stored legal domain dictionary, which includes names of legal entities, legal acts, and legal documents. The keyword set is stored in list form, with each item representing a keyword.

[0046] The scoring calculation subunit is connected to both the keyword extraction subunit and the candidate agent set construction module. The scoring calculation subunit receives the keyword set generated by the keyword extraction subunit, obtains the candidate agent set, and performs a weighted scoring calculation on each agent in the candidate agent set. The weighted scoring calculation is performed independently for each agent, ultimately generating a matching score for each agent.

[0047] The weighted scoring calculation includes three bonus items and an upper limit constraint. The first bonus item is functional matching bonus. For example, if the Agent's natural language functional description indicates that the Agent can handle the type of question involved in the standard question text, then 50 points are added. This bonus item has the highest weight among the three bonus items, and its purpose is to prioritize semantic functional matching over lexical keyword matching, ensuring that Agents capable of handling this type of question have a decisive advantage in scoring. The second bonus item is keyword matching bonus: for each keyword in the keyword set, if the keyword matches the Agent's natural language functional description, then each matching keyword adds 10 points. If multiple keywords in the keyword set match the same Agent's functional description, the points are accumulated. The third bonus item is name matching bonus: if the Agent's name matches the standard question text, then an additional 20 points are added. Name matching means that at least one word in the Agent's name appears in the standard question text, used to give extra rewards to Agents with a direct name association with the user's question. The maximum score for a single Agent is 100 points. This means that if the total score from the three bonus categories exceeds 100 points, it will be counted as 100 points. This maximum score prevents a single Agent from receiving inflated scores due to longer natural language function description texts hitting more keywords, ensuring that all Agent scores are compared on a uniform scale.

[0048] The selection output subunit is connected to the score calculation subunit and is used to receive the matching scores of each Agent output by the score calculation subunit, compare the scores of each Agent, select the Agent with the highest score as the target Agent, and output the identifier of the target Agent. If two or more Agents have the same score and are all the highest scores, the selection output subunit selects one as the target Agent according to preset rules. The preset rules include, but are not limited to, random selection, selecting the Agent whose name has the highest similarity to the standard question text, or selection according to the order of Agent registration. The preset rules are configured during system initialization to ensure that the selection result is deterministic in the case of the same score.

[0049] Furthermore, the Agent execution module includes a task invocation unit and a result aggregation unit.

[0050] The task invocation unit is connected to the third-level decision module and is used to receive the target agent identifier output by the third-level decision module. The target agent identifier is a unique identifier determined by the third-level decision module after first-level matching, second-level matching, or a fallback scoring process, and is used to uniquely point to a registered agent in the system. Based on the target agent identifier, the task invocation unit searches for and retrieves the corresponding execution interface of the target agent from the system. The execution interface is a programmable call entry point exposed by the agent. The task invocation unit takes the standard question text output by the input receiving module as input parameters and calls the target agent through the execution interface, triggering the target agent to perform a question-and-answer task. After receiving the standard question text, the target agent analyzes and processes the legal issues involved in the standard question text based on its internal knowledge base, reasoning logic, or external tool invocation capabilities, generating the original execution result.

[0051] The result aggregation unit is connected to the task invocation unit and is used to receive the raw execution results obtained by the task invocation unit from the target Agent. The raw execution results returned by different Agents may differ in data structure, field naming, and format specifications. The result aggregation unit performs format standardization processing on the raw execution results, uniformly converting raw execution results from different sources and with different structures into a preset standard output format. The task execution results have a unified data format, independent of the specific Agent type executing the task, enabling subsequent modules to process and display the task execution results in a consistent manner without needing to write differentiated parsing logic for different Agents.

[0052] Furthermore, the result output and log module includes: a response output unit and a log recording unit; The response output unit is used to receive the task execution result and output the task execution result to the user in natural language form; The log recording unit is used to record the routing results, the target Agent identifier, the matching level information executed by the three-level decision module, and the task execution status of the target Agent, so as to generate a routing decision log.

[0053] The response output unit is connected to the Agent execution module and is used to receive the task execution result output by the Agent execution module, and output the task execution result to the user in natural language form. The task execution result, after being formatted by the result aggregation unit, has a unified data format. The response output unit extracts the response content field from the task execution result, converts it into user-readable natural language text, and returns it to the user through the user interface, completing the question-and-answer service.

[0054] The logging unit is connected to the LLM routing decision module, the third-level decision module, and the Agent execution module. During system operation, the logging unit obtains the routing results and the parsed routing results from the LLM routing decision module, the target Agent identifier and the matching level information executed by the third-level decision module from the third-level decision module, and the task execution status of the target Agent from the Agent execution module. The matching level information refers to which level of the three-level decision module ultimately determines the target Agent in this routing decision, including three scenarios: first-level matching hit, second-level matching hit, or score fallback trigger. The task execution status refers to the process status of the target Agent executing the question-and-answer task, including task in progress, task successfully completed, or task execution failed.

[0055] The logging unit organizes the above information in a structured manner according to a preset log format, generates a routing decision log, and writes the routing decision log to persistent storage. The routing decision log completely records the entire route of this routing decision. The routing decision log is used by system maintenance personnel to trace and audit the routing decision process. When routing errors or Agent execution anomalies occur, maintenance personnel can locate the link and cause of the problem by querying the routing decision log. Simultaneously, the historical routing data accumulated in the routing decision log can serve as the data basis for subsequent optimization of the weighted scoring calculation coefficients in the scoring fallback unit. Through statistical analysis of a large number of historical routing decision logs, the correlation between each scoring factor and the actual routing results can be obtained, guiding the adjustment and optimization of the weight coefficients.

[0056] like Figure 4 As shown, based on the multi-agent question-answering system based on large language model self-driven routing described in the above embodiments, the present invention also provides a question-answering method for the multi-agent question-answering system based on large language model self-driven routing. The question-answering method for the multi-agent question-answering system based on large language model self-driven routing includes the following steps: Step S10: The input receiving module receives the question text input by the user, performs semantic standardization processing on the question text, and outputs standard question text.

[0057] Specifically, the semantic standardization process includes removing irrelevant characters from the user input text, standardizing the expression format, and identifying and completing omitted subject-verb-object structures, so that legal issues with different expressions are converted into a unified internal representation, which facilitates subsequent modules to accurately understand and process the semantics of user questions.

[0058] Step S20: The candidate agent set construction module receives the standard question text and collects the registration information of all registered agents in the system to generate a candidate agent set.

[0059] Specifically, each Agent is registered in the form of "name + natural language function description" during registration. The candidate Agent set construction module iterates through all registered Agents, reads the name and natural language function description of each Agent, and generates a candidate Agent set.

[0060] When the candidate agent set contains only one agent, the single-candidate direct routing unit directly identifies that agent as the target agent, skips subsequent routing steps, and is directly invoked and executed by the agent execution module. When the candidate agent set contains multiple agents, the candidate agent set is passed to the LLM routing decision module.

[0061] Step S30: The LLM routing decision module obtains the candidate agent set and the standard question text, assembles the candidate agent set and the standard question text into a routing prompt word, outputs the routing prompt word to the large language model, and receives the routing result returned by the large language model.

[0062] Specifically, the LLM routing decision module obtains the candidate Agent set and the standard question text, and assembles the Agent name and natural language function description of each Agent together with the standard question text into routing prompt words according to a preset template.

[0063] The routing prompt explicitly lists the names of all candidate agents as an output whitelist and includes a constraint instruction that forces the large language model to select exactly one agent name from the output whitelist as output, without outputting any other text, punctuation, or explanation. The model invocation unit submits the routing prompt to the large language model and receives the returned routing result. The output parsing unit performs format normalization processing on the routing result, including removing spaces, removing punctuation marks, and normalizing case, generating a parsed routing result, which is then output to the third-level decision module.

[0064] Step S40: The three-level decision module receives the routing result, performs first-level matching verification, second-level matching verification and score fallback processing on the routing result in sequence, and outputs the target Agent identifier.

[0065] The three-level decision module receives the parsed routing result and performs two levels of matching checks on the agent names of each agent in the candidate agent set. The first-level matching unit performs an exact equality matching check, determining whether the parsed routing result is exactly equal to any agent name. If no match is found, the second-level matching unit performs an inclusion matching check, determining whether the parsed routing result contains any agent name or whether any agent name contains the parsed routing result. If still no match is found, the fallback scoring unit is triggered, performing a weighted score calculation for each agent, selecting the agent with the highest score, and outputting the target agent identifier. This three-level decision module employs a progressively yielding decision-making strategy, pursuing accurate matching while also ensuring fault tolerance.

[0066] Step S50: The Agent execution module receives the target Agent identifier, calls the target Agent corresponding to the target Agent identifier to execute the question-and-answer task, and receives the task execution result returned by the target Agent.

[0067] Specifically, the Agent execution module receives the target Agent identifier, obtains the corresponding target Agent execution interface based on the target Agent identifier, and calls the target Agent with the standard question text as input parameter. After receiving the standard question text, the target Agent analyzes and processes the user's legal question based on its internal knowledge base and reasoning logic, generates a raw execution result, and returns it to the task invocation unit. The result aggregation unit receives the raw execution result, performs format standardization processing on the raw execution result, and uniformly converts the raw execution results with different structures and formats returned by different Agents into a preset standard output format to generate the task execution result.

[0068] Step S60: The result output and log module is used to receive the task execution result, output the task execution result as a legal response to the user, and record the routing decision log.

[0069] Specifically, the response output unit outputs the task execution result to the user in natural language form, thus completing the interaction of the question-and-answer service.

[0070] The logging unit obtains the routing result and the parsed routing result from the LLM routing decision module, the target Agent identifier and the matching level information executed from the three-level progressive adjudication module, and the task execution status of the target Agent from the Agent execution module. It then generates a routing decision log and writes it to persistent storage. The routing decision log fully records the complete routing decision process, allowing system maintenance personnel to trace and audit the routing decision process. It also provides a data foundation for subsequent optimization of the scoring weight coefficients in the scoring fallback unit.

[0071] The beneficial effects of this invention are as follows: (1) Zero-training routing: No labeled corpus or classifier training is required; intent routing can be completed solely based on the natural language description of the Agent. (2) Zero-code extension: To add a new professional agent, you only need to register its "name + description" without modifying the routing logic or rule base; (3) Double insurance high availability: When the LLM route fails, it automatically degrades to the deterministic scoring mechanism to ensure that an available Agent can be returned under any circumstances, and the route is never lost; (4) Explainable and auditable: The entire process records the basis for routing decisions, which facilitates traceability and optimization.

[0072] Furthermore, such as Figure 5 As shown, based on the above-mentioned multi-agent question-answering system and question-answering method based on the self-driven routing of a large language model, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0073] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, Smart Media Array (SMC), Secure Digital (SD) card, or Flash Array card. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a question-answering program 40 of a multi-agent question-answering system based on a large language model self-driven routing. This question-answering program 40 can be executed by the processor 10 to implement the question-answering method of the multi-agent question-answering system based on a large language model self-driven routing in this application.

[0074] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the question-answering method of the multi-agent question-answering system based on the large language model self-driven routing.

[0075] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visualized patient interface. The components of the terminal communicate with each other via a system bus.

[0076] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a question-answering program of a multi-agent question-answering system based on a large language model self-driven routing, and the question-answering program of the multi-agent question-answering system based on a large language model self-driven routing implements the steps of the question-answering method of the multi-agent question-answering system based on a large language model self-driven routing as described above when executed by a processor.

[0077] In summary, this invention provides a multi-agent question-answering system and method based on self-driven routing using a large language model. The system includes an input receiving module, a candidate agent set construction module, an LLM routing decision module, a three-level decision module, an agent execution module, and a result output and logging module. These modules are connected sequentially. The candidate agent set construction module collects the names and natural language function descriptions of all registered agents to generate a candidate set. The LLM routing decision module assembles the candidate agent information and the user question into routing prompts and submits them to the large language model. The returned results are then normalized by the output parsing unit and output. The three-level decision module performs three progressive processing steps on the parsed routing results: complete equality matching, inclusion matching, and a fallback scoring, to determine the target agent. The agent execution module calls the target agent to perform the question-answering task. The result output and logging module outputs legal responses and records the routing decision log. This invention uses LLM semantic routing and a fallback scoring mechanism to create a double insurance mechanism, improving system robustness while ensuring routing accuracy. It effectively avoids irrelevant answers or service unavailability caused by routing errors, significantly improving the response reliability and user experience of the question-answering system.

[0078] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0079] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0080] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multi-agent question-answering system based on self-driven routing using a large language model, characterized in that, The multi-agent question answering system based on large language model self-driven routing includes: an input receiving module, a candidate agent set construction module, an LLM routing decision module, a three-level decision module, an agent execution module, and a result output and log module, wherein the input receiving module, the candidate agent set construction module, the LLM routing decision module, the three-level decision module, the agent execution module, and the result output and log module are connected in sequence; The input receiving module is used to receive the question text input by the user, perform semantic standardization processing on the question text, and output standard question text; The candidate agent set construction module is used to receive the standard question text and collect the registration information of all registered agents in the system to generate a candidate agent set; The LLM routing decision module is used to obtain the candidate agent set and the standard question text, assemble the candidate agent set and the standard question text into routing prompt words, output the routing prompt words to the large language model, and receive the routing results returned by the large language model; The three-level decision module is used to receive the routing result, perform first-level matching verification, second-level matching verification and score fallback processing on the routing result in sequence, and output the target Agent identifier; The Agent execution module is used to receive the target Agent identifier, invoke the target Agent corresponding to the target Agent identifier to perform the question-and-answer task, and receive the task execution result returned by the target Agent; The result output and log module is used to receive the task execution result, output the task execution result as a legal response to the user, and record the routing decision log.

2. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 1, characterized in that, The candidate agent set construction module includes: an agent information storage unit, a set generation unit, and a single candidate routing unit; The Agent information storage unit is used to store the registration information of each Agent, which includes the Agent name and natural language function description; The set generation unit is used to receive the standard question text output by the input receiving module, and traverse the Agent information storage unit to read the registration information of all registered Agents and generate a candidate Agent set. The single candidate routing unit is used to determine the number of agents in the candidate agent set. When there is only one agent in the candidate agent set, the single candidate routing unit determines the agent as the target agent, and skips the LLM routing decision module and the third-level decision module to directly input the target agent identifier into the agent execution module.

3. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 2, characterized in that, The LLM routing decision module includes: a prompt word assembly unit and a model invocation unit; The prompt word assembly unit is used to obtain the candidate Agent set and the standard question text, extract the Agent name and natural language function description of each Agent from the candidate Agent set, and assemble all the Agent names and all the natural language function descriptions with the standard question text into routing prompt words according to a preset template; The model invocation unit is used to submit the routing prompt words to the general large language model and receive the routing results returned by the general large language model.

4. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 3, characterized in that, The routing prompt explicitly lists the names of all candidate agents as an output whitelist and includes constraint instructions; The constraint instruction is used to force the large language model to select only one Agent name from the output whitelist as output, and to prohibit the output of other text, punctuation or explanation.

5. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 3, characterized in that, The LLM routing decision module further includes: an output parsing unit; The output parsing unit is connected to the model calling unit and is used to perform format normalization processing on the routing result after receiving the routing result returned by the large language model, generate parsed routing result, and output the parsed routing result to the third-level decision module. The format normalization process includes removing spaces, removing punctuation marks, and normalizing case.

6. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 1, characterized in that, The three-level decision-making module includes: a first-level matching unit, a second-level matching unit, and a scoring fallback unit; The first-level matching unit is used to receive the routing result, and perform a complete equality matching verification between the routing result and the agent name of each agent in the candidate agent set. If the routing result is completely equal to the agent name of any agent, then the agent is determined as the target agent and the target agent identifier is output. The secondary matching unit is used to perform an inclusion-matching check on the routing result and the agent name of each agent in the candidate agent set in sequence when the primary matching unit fails to match any agent. If the routing result contains the agent name of any agent, or the agent name of any agent contains the routing result, then the agent is determined as the target agent and the target agent identifier is output. The scoring fallback unit is used to calculate a matching score for each Agent in the candidate Agent set when neither the first-level matching unit nor the second-level matching unit matches any Agent, and to take the Agent with the highest score as the target Agent and output the target Agent identifier.

7. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 6, characterized in that, The scoring catch-all unit includes: a keyword extraction subunit, a score calculation subunit, and an output selection subunit; The keyword extraction subunit is connected to the input receiving module and is used to extract keywords from the standard question text to generate a keyword set. The scoring calculation subunit is connected to the keyword extraction subunit and the candidate agent set construction module, respectively, and is used to perform weighted scoring calculation on each agent in the candidate agent set based on the keyword; The selection output subunit is connected to the score calculation subunit and is used to compare the scores of each Agent, select the Agent with the highest score as the target Agent, and output the target Agent identifier.

8. The multi-agent question-answering system based on a large language model and self-driven routing according to claim 1, characterized in that, The result output and log module includes: a response output unit and a log recording unit; The response output unit is used to receive the task execution result and output the task execution result to the user in natural language form; The log recording unit is used to record the routing results, the target Agent identifier, the matching level information executed by the three-level decision module, and the task execution status of the target Agent, so as to generate a routing decision log.

9. A question-answering method based on the multi-agent question-answering system with self-driven routing based on a large language model as described in any one of claims 1-8, characterized in that, The question-and-answer method includes: The input receiving module receives the question text input by the user, performs semantic standardization processing on the question text, and outputs standard question text; The candidate agent set construction module receives the standard question text and collects the registration information of all registered agents in the system to generate a candidate agent set; The LLM routing decision module obtains the candidate agent set and the standard question text, assembles the candidate agent set and the standard question text into routing prompt words, outputs the routing prompt words to the large language model, and receives the routing results returned by the large language model; The third-level decision module receives the routing result, performs first-level matching verification, second-level matching verification, and score fallback processing on the routing result in sequence, and outputs the target Agent identifier; The Agent execution module receives the target Agent identifier, invokes the target Agent corresponding to the target Agent identifier to perform the question-and-answer task, and receives the task execution result returned by the target Agent; The result output and log module is used to receive the task execution result, output the task execution result as a legal response to the user, and record the routing decision log.

10. A terminal, characterized in that, The terminal includes: a memory, a processor, and a question-answering program of a multi-agent question-answering system based on a large language model self-driven routing, which is stored in the memory and can run on the processor. When the question-answering program of the multi-agent question-answering system based on a large language model self-driven routing is executed by the processor, it implements the steps of the question-answering method of the multi-agent question-answering system based on a large language model self-driven routing as described in any one of claims 7-8.

Images