Financial question-answering method, system, device and medium based on multi-agent interaction

Abstract

The present application relates to a financial question-and-answer method, system, device and medium based on multi-agent interaction. The method includes: understanding the query of the user input through the recall of related entity words and similar question-and-answer pairs, and then performing knowledge recall and tool recall, and filling the recall result into the prompt template of the planning agent; answering the user input according to the prompt template by the planning agent, extracting the answer content to obtain the API execution link and generating a directed acyclic graph; calling the calling tool encapsulated by the relevant API by the execution agent to execute the tasks corresponding to the directed acyclic graph in parallel; generating the answer according to the output result of the execution agent by the summary agent, re-planning and judging according to the input and output information of the planning agent and the answer of the summary agent by the reflection agent, and outputting the current answer. The use of this method can provide users with more professional, accurate and efficient financial question-and-answer services.

Description

Technical Field

[0001] The present application relates to the field of intelligent question-answering technology, and in particular to a financial question-answering method, system, device and medium based on multi-agent interaction. Background Art

[0002] Traditional question-answering systems usually parse and answer users' questions through intent recognition and retrieval recall, respectively. These systems parse the natural language input by users, identify their intent, and recall relevant information from pre-built knowledge bases or databases to generate answers. However, this approach has obvious limitations in complex question-answering scenarios, especially when faced with complex multi-step tasks and dynamic information. In the long-term iteration process, question-answering systems have accumulated many types of API interfaces, most of which can be used as independent atomic services. We can split user questions into multiple sub-questions according to the problem-solving process. Each sub-question may require calling one or more atomic APIs to solve. This problem definition method allows us to abstract structured and unstructured QA into API combination call problems, while making it possible to answer user questions directly. So how to make full use of these rich API interfaces, convert users' natural language questions into executable API call chains in a dynamic and flexible way, and solve them end-to-end is becoming an increasingly important issue for us.

[0003] AI Agent is a system driven by a large language model. It has the ability to understand, perceive, plan, remember and use tools autonomously, and can automatically perform complex tasks. The multi-agent system adopts multiple agents with specialized functions, each of which has its own independent knowledge reserves. The user's instructions will be processed one by one by these agents in a predetermined order, and each agent will contribute its expertise.

[0004] Large language models have powerful summarization, extraction and reasoning capabilities. Question-answering systems based on large language models are gradually replacing many capabilities of traditional question-answering systems. For example, when the question-answering system needs to call various API interfaces, it can be solved with the help of the large language model tool call capability. However, the current open source large language model tool call is relatively weak, especially when there are many tools and complex tool parameters. For example, in the financial question-answering system, we involve about 300 API interfaces in the financial product screening and financial product diagnosis scenarios. When the user's question involves multiple API interfaces, it is difficult for the open source LLM to fully identify them. For example, the user's question involves the API interfaces such as "query recent yield", "query redemption rate", "query subscription rate" for financial product queries, and "multiplication" and "addition" in numerical calculations. This is still a big challenge for the large language model.

[0005] Technologies such as Chain of Thought (COT), Tree of Thought (TOT), and Reasoning Action Trajectory (REACT) combine the capabilities of large language models with reasoning and action to improve their performance in understanding and interactive decision-making tasks. However, there is still the problem of unstable output content, and LLM's task decomposition and number of cycles are uncontrollable, resulting in problems such as uncontrollable system response time, which will seriously affect the user experience in the actual production process.

[0006] Therefore, the open source large language model tool has low recognition accuracy, is prone to hallucinations, and has poor generalization. Especially when faced with complex and changeable financial question-and-answer scenarios, the system often falls into the dilemma of inefficiency and inaccuracy. Summary of the invention

[0007] Based on this, it is necessary to provide a financial question-and-answer method, system, device and medium based on multi-agent interaction to address the above technical problems.

[0008] A financial question-answering method based on multi-agent interaction, the method comprising:

[0009] Obtain user input, understand the query of the user input through related entity word recall and similar question-answer pair recall, perform knowledge recall and tool recall based on query understanding, and fill the recall results into the prompt template of the planning agent;

[0010] The planning agent answers the user input according to the prompt template, and the parser extracts the answer content to obtain the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are dependencies between API interfaces;

[0011] The tasks corresponding to the directed acyclic graph are executed in parallel by executing a calling tool encapsulated by a related API by an executing agent;

[0012] The summarizing agent generates an answer based on the output results of the executing agent, and the reflecting agent makes a re-planning judgment based on the input and output information of the planning agent and the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the condition for stopping the iteration is met, and the current answer is output.

[0013] A financial question-answering system based on multi-agent interaction, the system comprising:

[0014] Planning agent, execution agent, summary agent and reflection agent;

[0015] The planning agent answers the user input according to the prompt template, extracts the answer content through the parser, obtains the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are the dependencies between API interfaces; the prompt template is obtained through the recall results of related entity word recall, similar question and answer pair recall, knowledge recall and tool recall;

[0016] The execution agent calls the calling tool encapsulated by the relevant API to execute the tasks corresponding to the directed acyclic graph in parallel;

[0017] The summarizing agent generates an answer based on the output of the executing agent, and the reflecting agent makes a re-planning judgment based on the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the conditions for stopping the iteration are met, and the current answer is output.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0019] Obtain user input, understand the query of the user input through related entity word recall and similar question-answer pair recall, perform knowledge recall and tool recall based on query understanding, and fill the recall results into the prompt template of the planning agent;

[0020] The planning agent answers the user input according to the prompt template, and the parser extracts the answer content to obtain the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are dependencies between API interfaces;

[0021] The tasks corresponding to the directed acyclic graph are executed in parallel by executing a calling tool encapsulated by a related API by an executing agent;

[0022] The summarizing agent generates an answer based on the output results of the executing agent, and the reflecting agent makes a re-planning judgment based on the input and output information of the planning agent and the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the condition for stopping the iteration is met, and the current answer is output.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Obtain user input, understand the query of the user input through related entity word recall and similar question-answer pair recall, perform knowledge recall and tool recall based on query understanding, and fill the recall results into the prompt template of the planning agent;

[0025] The planning agent answers the user input according to the prompt template, and the parser extracts the answer content to obtain the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are dependencies between API interfaces;

[0026] The tasks corresponding to the directed acyclic graph are executed in parallel by executing a calling tool encapsulated by a related API by an executing agent;

[0027] The summarizing agent generates an answer based on the output results of the executing agent, and the reflecting agent makes a re-planning judgment based on the input and output information of the planning agent and the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the condition for stopping the iteration is met, and the current answer is output.

[0028] The above-mentioned financial question-and-answer method, system, device and medium based on multi-agent interaction, first of all, understand the query through entity words and similar questions and answers, which can accurately grasp the user's intention and facilitate the subsequent accurate response. The knowledge and tool recall based on this provides rich references for the planning agent, thereby optimizing the answer generation. Secondly, the parser constructs a directed acyclic graph to clarify the API interface dependency, and guides the execution agent to perform tasks in parallel through this graph, which can improve the task processing efficiency and avoid process confusion. When facing complex and multi-step financial problems, it ensures that task dependencies are effectively managed and improves the system response speed. Furthermore, the summary agent integrates the execution results to generate answers, and the reflection agent makes re-planning judgments. If necessary, the planning agent prompt template is updated and iterated, which is conducive to continuously improving the answer quality and enabling the system to dynamically adapt to complex and changeable financial question-and-answer situations. The collaborative operation of multiple agents can give full play to their respective advantages. From understanding problems, planning answers to executing tasks and optimizing answers, each link is closely connected, which is conducive to providing users with more professional, accurate and efficient financial question-and-answer services, thereby improving the system's performance and adaptability in information interaction scenarios in the financial field and meeting the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1is a flowchart of a financial question-answering method based on multi-agent interaction in one embodiment;

[0030] Figure 2 A schematic diagram of a process for recalling a standard name of a financial entity word in an embodiment;

[0031] Figure 3 A schematic diagram of a process of recalling expert knowledge in one embodiment;

[0032] Figure 4 A schematic diagram of the structure of a tool retrieval multi-label classification model in one embodiment;

[0033] Figure 5 A schematic diagram of an API execution link in an embodiment;

[0034] Figure 6 is a structural block diagram of a financial question-answering system based on multi-agent interaction in one embodiment;

[0035] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] In one embodiment, Figure 1 As shown, a financial question-answering method based on multi-agent interaction is provided, comprising the following steps:

[0038] Step 102, obtain user input, perform query understanding on the user input through relevant entity word recall and similar question and answer pair recall, perform knowledge recall and tool recall based on query understanding, and fill the recall result into the prompt template of the planning agent.

[0039] This stage mainly solves the recall of candidate entity words such as financial products in user queries. For example, user queries often only contain the abbreviation of financial product entity words, which corresponds to the complete financial product entity word name. In addition, few shot prompting as a prompting strategy provides multiple examples to LLM, which allows LLM to generate more accurate and correctly formatted output. Therefore, the method of the present invention will recall a certain number of similar question and answer pairs from the accumulated question and answer library. A hybrid recall strategy is used to recall several related entity words from the entity word library, and the recalled related entity words and similar question and answer pair lists will be used as part of the prompt input of the planning agent.

[0040] Considering that LLM lacks domain expertise, such as the calculation formula and annotations of a company's price-to-book ratio, by recalling relevant domain knowledge, LLM is assisted in text generation, thereby improving the quality and accuracy of predictions. Efficient tool selection based on user input. Not all user inputs require all available tools, and it is critical to select the correct toolset to minimize the prompt size and improve performance. Therefore, the present invention trains a ToolRAG recall model to filter out a suitable API list from the API set based on the user query. Using RAG technology to recall API interfaces can alleviate the contextual input limitations of large models. The tool list and expert knowledge recalled at this stage are also part of the prompt input for the planning agent.

[0041] Step 104, the planning agent answers the user input according to the prompt template, the parser extracts the answer content, obtains the API execution link, and generates a directed acyclic graph according to the API execution link.

[0042] In the planning stage, it is crucial to fully analyze user intent. This stage involves decomposing user problems into multiple sub-problems to achieve specified goals. In addition, given the inherent dependencies between certain tasks, the planning stage also requires accurate depiction of the dependencies and execution order between the decomposed tasks, thereby facilitating the interconnection between sub-problems. Therefore, we construct a planning agent to generate a dependency-based directed acyclic graph of API interface call links based on user queries. The directed acyclic graph includes nodes and edges, where nodes are API interfaces and edges are dependencies between API interfaces. In the planning stage, the planning agent is only responsible for generating a directed acyclic graph, and will not specifically execute related tools.

[0043] In real business scenarios, there are a large number of function interfaces. The open source LLM cannot accurately recognize most of the function interfaces that have not been seen (not involved in training). At the same time, when the function name changes or the parameters are added, deleted, or modified, the recognition accuracy is also very low. To address the above problems, we fine-tuned a Tool LLM model based on the open source model using the tool learning corpus in the financial question-answering scenario (see Chinese invention patent CN118690853A for the fine-tuning process). The fine-tuning process mainly uses the manually constructed ReAct format tool call data as a small sample prompt and inputs it to the large language model, and fine-tunes the large language model in combination with the constructed tool call data to improve the accuracy and generalization of the large model tool call. Use the planning agent to generate a directed acyclic graph of the API interface call link. In order to improve the planning ability of LLM, a Tool LLM model is specially trained to improve the low accuracy of the existing open source large model API recognition, and the problem of hallucinations and the generation of non-existent function names or parameters.

[0044] Step 106, executing the tasks corresponding to the directed acyclic graph in parallel by executing the calling tool encapsulated by the related API by the executing agent.

[0045] After planning is complete, the execution unit is called in parallel to execute the relevant APIs. Once the dependencies of the tasks are met, the unit will schedule the tasks. Since many tools involve other calls to the search engine or LLM, additional parallelism can significantly improve speed.

[0046] Step 108, the summarizing agent generates an answer based on the output results of the executing agent, and the reflecting agent makes a re-planning judgment based on the input and output information of the planning agent and the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the condition for stopping the iteration is met, and the current answer is output.

[0047] The summarizing agent summarizes the task execution results, and the reflecting agent decides whether to respond to the final answer or pass the progress back to the planning agent based on the entire historical record (including the task execution results).

[0048] In the above-mentioned financial question-answering method based on multi-agent interaction, firstly, the query understanding of recall through entity words and similar questions and answers can accurately grasp the user's intention, which is conducive to subsequent accurate response. Based on this knowledge and tool recall, it provides rich references for the planning agent, thereby optimizing the generation of answers. Secondly, the parser constructs a directed acyclic graph to clarify the API interface dependency, and guides the execution agent to execute tasks in parallel through this graph, which can improve the efficiency of task processing and avoid process confusion. When facing complex and multi-step financial problems, it ensures that task dependencies are effectively managed and improves the system response speed. Moreover, the summary agent integrates the execution results to generate answers, and the reflection agent makes re-planning judgments. If necessary, the planning agent prompt template is updated and iterated, which is conducive to continuously improving the answer quality and enabling the system to dynamically adapt to complex and changeable financial question-answering situations. The collaborative operation between multiple agents can give full play to their respective advantages. From understanding questions, planning answers to executing tasks and optimizing answers, each link is closely connected, which is conducive to providing users with more professional, accurate and efficient financial question-answering services, thereby improving the performance and adaptability of the system in information interaction scenarios in the financial field and meeting the diverse needs of users.

[0049] In one embodiment, obtaining user input, performing query understanding on the user input through recalling relevant entity words and similar question and answer pairs, performing knowledge recall and tool recall based on the query understanding, and filling the recall result into the prompt template of the planning agent includes: obtaining user input, and recalling relevant entity words and a similar question and answer pair list in a financial entity vocabulary and a historical question and answer library respectively according to the user input: the similar question and answer pair list includes multiple similar question and answer pairs, and each similar question pair includes a similar question and a thought chain; performing query understanding based on the relevant entity words and the similar question and answer pair list, and recalling knowledge fragment text and a candidate API list in an expert knowledge base and a tool library respectively based on the user query; and filling the prompt template of the planning agent according to the relevant entity words, the similar question and answer pair list, the knowledge fragment text and the candidate API list.

[0050] In one embodiment, recalling related entity words in the financial entity word library according to user input includes: performing mixed multi-way recall in the financial entity word library according to the financial entity abbreviation in the user input to obtain related entity words, and matching them by fuzzy matching method during recall. In this embodiment, the traditional question-answering system needs to perform pre-processing such as error correction and completion on the entity word abbreviation of the user's query statement (mainly the abbreviation of financial targets such as financial products in the financial question-answering system) during the user query understanding stage. LLM has a strong semantic understanding ability and can more accurately find the link of the standard name corresponding to the abbreviation of the query target from the candidate financial entity standard name. Therefore, we use fuzzy matching to recall the relevant standard entity word names from the predefined financial entity word library (which stores the standard name set of financial targets in the entire market) according to the user query.

[0051] Specifically, Figure 2 As shown in the figure, the recall process diagram of the standard names of financial entity words is used. We use the longest common subsequence algorithm (LSC), the longest common substring algorithm (LCS) and string matching algorithms for hybrid multi-way recall. Each type of algorithm recalls 5 entity words to ensure the recall rate.

[0052] In one embodiment, recalling a list of similar question-answer pairs in a historical question-answer database based on user input includes: using a fine-tuned similar question discrimination model to predict the similarity between the user input and each historical input, and taking the historical input with a similarity greater than a threshold as a similar question; constructing a corresponding thought chain based on the similar questions, and obtaining a list of similar question-answer pairs based on each similar question and the corresponding thought chain. In this embodiment, in order to improve the LLM's ability to understand user queries, similar historical user queries are recalled from the historical dialogue corpus.

[0053] Specifically, train a similar question discrimination model to calculate whether the current user query is similar to the historical user query. The user query similar question discrimination model includes an input layer, a BERT layer, a fully connected (Dense) layer, a softmax layer, and an output layer. The input part contains query1 [SEP] query2, which means there are two query sentences query1 and query2, which are separated by [SEP]. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that performs deep semantic understanding and encoding of the input text and converts the input text into a vector representation with rich semantic information. The fully connected layer receives the output from the BERT layer and performs further feature extraction and conversion. The softmax layer converts the output of the Dense layer into a probability distribution. In the inference process, similar questions with a probability greater than 0.9 after softmax are selected.

[0054] After obtaining similar questions, the output process with thought chain is automatically constructed, that is, the answers to the obtained similar questions are converted into the format of the output part of the constructed question-answer pair example. Finally, the similar questions and thought chain outputs are combined into a few shot single prompt example for the planner agent to use.

[0055] In one embodiment, recalling the knowledge fragment text in the expert knowledge base based on the user query includes: storing the content in the expert knowledge base in blocks, creating an inverted index for the block data, vectorizing the block data and storing it in the vector search engine, searching for relevant knowledge fragments in the vector search engine according to the vector representation of the user query, and querying the corresponding knowledge fragment text according to the inverted index. In this embodiment, considering that LLM lacks professional knowledge in the field, such as the calculation formula and annotations of the company's price-to-book ratio, by recalling relevant field knowledge, LLM is provided with retrieved field information to assist LLM in text generation, thereby improving the quality and accuracy of prediction.

[0056] Specifically, Figure 3The flowchart of expert knowledge recall shown in the figure shows that the knowledge base stores expert knowledge in the financial field, including financial encyclopedia, investment education, investment consulting knowledge, etc. First, the expert knowledge is sliced ​​and divided into blocks and stored in the database. Specifically, the blocks are divided based on the specified delimiters such as segmentation and line breaks. At the same time, the size of the blocks and the number of repeated texts between blocks are specified. The size of the block depends on the longest input of the embedding vector model. The specific length used is 512. The database is redis, and an inverted index is created. Then, the block data is vectorized through the vector model, the vector model is bge, and stored in the vector retrieval engine, the vector retrieval engine is faiss. Finally, after the user query is vectorized through the vector model, it passes through the faiss vector retrieval engine to all the top-k related fragment related indexes, and the stored knowledge fragment text is queried according to the inverted index. The query and the retrieved knowledge fragment text will be used as part of the prompt of the planner intelligent agent LLM.

[0057] In one embodiment, recalling a candidate API list in a tool library based on a user query includes: retrieving a tool multi-label classification model after fine-tuning the user query input to obtain a predicted probability of the user query input in several categories; the category is a tool for processing the user query; and selecting a category greater than a probability threshold to obtain a candidate API list. In this embodiment, in real-world scenarios such as financial scenario question-and-answer systems, there are many API interfaces. Due to the maximum input limit of the LLM context, all API information cannot be fully transmitted to the LLM. In addition, if all APIs are used as LLM context inputs, most of the irrelevant API information will also cause information overload problems, reducing the accuracy of LLM answers, and too long contexts will also lead to excessive token consumption and increase response time. The present invention uses RAG technology to recall API interfaces, which significantly reduces the number of candidate APIs, alleviates the LLM context input restrictions, and also improves the accuracy of tool matching.

[0058] Specifically, Figure 4 The schematic diagram of the tool retrieval multi-label classification model shown in the figure constructs tool retrieval as a multi-label classification problem. A BERT model is fine-tuned on the training data. The user query is used as the input of the model during the training phase, and then the final CLS token is passed through a simple fully connected layer of size 768xN to convert it into an N-dimensional vector (where N is the number of tools). The output of this layer passes through a sigmoid layer to produce the probability of selecting each tool. During the inference process, tools with a probability greater than 50% are selected and their descriptions are included in the prompt.

[0059] Based on the above recall results, a prompt template for the planning agent is constructed. During the reasoning process of the planning agent, the prompt word template of the planning agent is as follows: "You are an expert in the financial field. You need to combine the provided background knowledge and use the tools provided by the orchestration to resolve the user's question.

[0060] Here is the relevant background knowledge:

[0061] ${BACKGROUND}

[0062] You have the following custom tools:

[0063] ${TOOLIST}

[0064] The standard product names that may be involved in user questions are:

[0065] ${PRODUCT}

[0066] Please use the following format to return results:

[0067] Question: User questions that need to be answered

[0068] Thought: You should think about what action you need to take next

[0069] Action: The name of the Action, including api_id for tool id, tool_name for toolset name, api_name for tool name, etc.

[0070] Action Input: Action input, including rely_apis: indicates whether it depends on the return of the previous step. If it does, rely_apis saves the dependent api_id; required_parameters: indicates the input parameters of api_name in the Action.

[0071] Observation: The output of an Action.

[0072] ... (Thought / Action / Action Input / Observation can be looped multiple times)

[0073] Thought: I know the final answer.

[0074] Final Answer: The final answer to the original input question, output in json format.

[0075] You can refer to the following user questions and answers:

[0076] ${QARECALL}

[0077] The current date is: ${DATE}

[0078] start!

[0079] Question:

[0080] ${QUERY}".

[0081] in:

[0082] ${BACKGROUND} is the knowledge fragment in the knowledge recall phase;

[0083] ${TOOLIST} is the candidate API list for the tool recall phase;

[0084] ${PRODUCT} is the standard name of the candidate financial entity recalled in the query understanding phase;

[0085] ${QARECALL} is the similar question-answer pairs recalled in the query understanding phase;

[0086] ${DATE} is the current date, which is fixed at xx / xx / xx / xx / xx:xx / xx.

[0087] ${QUERY} is the current user query input.

[0088] ${BACKGROUND}, ${TOOLIST}, ${PRODUCT}, and ${QARECALL} can be understood as the memory of the planning agent. After concatenating the above content, the intermediate output results of the planning agent include the output results after the tool is called for the user query.

[0089] Finally, the system will extract the Action, Action Input, Observation, and Final Answer parts of the output content of the planning agent through the parser. The extracted API execution link is as follows: Figure 5 As shown. The parser will combine the parsed content into a text in JSON format.

[0090] The scheduler builds task dependencies based on the output of the planning agent and triggers the execution of different tasks. Through multi-threading, tasks without dependencies on the DAG can be executed in parallel.

[0091] The executor is an encapsulation of the API interface, unified into a tool that can be called by LLM. The execution process of the executor is the process of calling the tool.

[0092] After the task is executed, the return results of the called tool are reformatted into answers to structured questions.

[0093] In the reasoning process of the summary agent, the prompt word template of the summary agent is as follows: "You are an expert in information analysis in the financial field, good at extracting key information from complex information. Your task is to generate a complete structured answer to the specific question raised by the user based on the relevant background knowledge and the execution results of the provided calling tool combined with your own knowledge.

[0094] You need to follow the following rules:

[0095] 1. Understand and analyze the background information and the execution results of the calling tools, and combine your own knowledge to ensure that you provide complete and structured answers that are accurate, clear and easy to understand.

[0096] 2. Answer questions in detail, avoid repeating sentences, and pay close attention to the timeliness of information.

[0097] 3. Do not give answers that are not helpful to the question, and do not use unclear words such as XXX, ABC, etc.

[0098] 4. The overall answer should not contain repeated information. If necessary, line breaks can be used to improve the reading experience.

[0099] The output must be a Json snippet formatted as shown below, with the answer field being the complete structured answer to the question.

[0100] {

[0101] "answer": string,

[0102] }

[0103] Related background knowledge:

[0104] ${BACKGROUND}

[0105] You have the following custom tools:

[0106] ${TOOLIST}

[0107] The standard product names that may be involved in user questions are:

[0108] ${PRODUCT}

[0109] The execution result of calling the tool:

[0110] ${TOOLCALL}

[0111] The current date is: ${DATE}

[0112] start!

[0113] Questions asked by users must be answered in Chinese.

[0114] The user's question is: ${QUERY}".

[0115] in:

[0116] ${BACKGROUND} is the knowledge fragment in the knowledge recall phase;

[0117] ${TOOLIST} is the candidate API list for the tool recall phase;

[0118] ${PRODUCT} is the standard name of the candidate financial entity recalled in the query understanding phase;

[0119] ${TOOLCALL} is the reformatted result of the task scheduling and execution phases;

[0120] ${DATE} is the current date, which is fixed at xx / xx / xx / xx / xx:xx / xx.

[0121] ${QUERY} is the current user query input.

[0122] During the reasoning process of the reflective agent, the prompt word template of the reflective agent is as follows: "You are an expert in information analysis in the financial field. Your goal is to judge whether the answer to the question is logical and provides valuable information, and to make suggestions and comments on the answer to the question.

[0123] You need to follow the following rules:

[0124] 1.Answer must completely answer the specified question to be useful.

[0125] 2. If the answer is to provide a way to query information, it is useless.

[0126] The output must be a Json code snippet formatted as follows. The suggestion field determines whether the answer is correct for the question and gives modification suggestions. The is_replan field determines whether the answer needs to be replanned.

[0127] {

[0128] "suggestion": string,

[0129] "is_replan": true / false

[0130] }

[0131] Previous conversation: ${HISTORY}

[0132] The questions asked by the user must be answered in Chinese. The specified questions are:

[0133] ${QUERY}

[0134] The answer given is: ${ANSWER}

[0135] start!".

[0136] in:

[0137] ${HISTORY} contains all the records from query understanding to tool execution.

[0138] ${QUERY} The current user's query.

[0139] ${ANSWER} summarizes the output of the agent.

[0140] In one embodiment, updating the prompt template of the planning agent according to the answer and the suggestion output by the reflection agent includes: adding the answer output by the summary agent and the suggestion output by the reflection agent to the prompt template of the planning agent to obtain an updated prompt template. In this implementation, compared with the initial prompt word template of the planning agent, ${TOOLCALL} and ${SUGGESTION} are added to the prompt word template of the planning agent in the re-planning stage to store the answer of the previous round of calling the summary agent and the evaluation suggestion of the reflection agent. The process of planning-execution-summarization-reflection can be executed repeatedly.

[0141] In the re-planning phase, the prompt template of the planning agent is as follows: “You are an expert in the financial field. You need to combine the background knowledge provided to parse the user’s question by arranging the tools provided.

[0142] Here is the relevant background knowledge:

[0143] ${BACKGROUND}

[0144] You have the following custom tools:

[0145] ${TOOLIST}

[0146] The standard product names that may be involved in user questions are:

[0147] ${PRODUCT}

[0148] Please use the following format to return results:

[0149] Question: User questions that need to be answered

[0150] Thought: You should think about what action you need to take next

[0151] Action: The name of the Action, including api_id for tool id, tool_name for toolset name, api_name for tool name, etc.

[0152] Action Input: Action input, including rely_apis: indicates whether it depends on the return of the previous step. If it does, rely_apis saves the dependent api_id; required_parameters: indicates the input parameters of api_name in the Action.

[0153] Observation: The output of an Action.

[0154] ... (Thought / Action / Action Input / Observation can be looped multiple times)

[0155] Thought: I know the final answer.

[0156] Final Answer: The final answer to the original input question, output in json format.

[0157] You can refer to the following user questions and answers:

[0158] ${QARECALL}

[0159] The execution result of calling the tool:

[0160] ${TOOLCALL}

[0161] Modification comments:

[0162] ${SUGGESTION}

[0163] The current date is: ${DATE}

[0164] start!

[0165] Question:

[0166] ${QUERY}".

[0167] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0168] In one embodiment, Figure 6 As shown, a financial question-answering system based on multi-agent interaction is provided, including: a planning agent, an execution agent, a summary agent and a reflection agent;

[0169] The planning agent answers the user input according to the prompt template, extracts the answer content through the parser, obtains the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are the dependencies between API interfaces; the prompt template is obtained through the recall results of related entity word recall, similar question and answer pair recall, knowledge recall and tool recall;

[0170] The execution agent calls the calling tool encapsulated by the relevant API to execute the tasks corresponding to the directed acyclic graph in parallel;

[0171] The summarizing agent generates an answer based on the output of the executing agent, and the reflecting agent makes a re-planning judgment based on the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the conditions for stopping the iteration are met, and the current answer is output.

[0172] exist Figure 6 In , when the agent executes tasks in parallel, the call record will be stored. The tool is a package of the API interface. In a complete conversation, the user will interact with the system many times. Each time the executor module is triggered, the tool will be called, and the call history will be stored, that is, Figure 6 The memory function in.

[0173] For the specific definition of the financial question-and-answer system based on multi-agent interaction, please refer to the definition of the financial question-and-answer method based on multi-agent interaction above, which will not be repeated here. Each module in the above-mentioned financial question-and-answer system based on multi-agent interaction can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0174] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input system connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a financial question-and-answer method based on multi-agent interaction is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input system of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0175] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0178] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A financial question-answering method based on multi-agent interaction, characterized in that: The method comprises: Obtain user input, understand the query of the user input through related entity word recall and similar question-answer pair recall, perform knowledge recall and tool recall based on query understanding, and fill the recall results into the prompt template of the planning agent; The planning agent answers the user input according to the prompt template, and the parser extracts the answer content to obtain the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are dependencies between API interfaces; The tasks corresponding to the directed acyclic graph are executed in parallel by executing a calling tool encapsulated by a related API by an executing agent; The summarizing agent generates an answer based on the output results of the executing agent, and the reflecting agent makes a re-planning judgment based on the input and output information of the planning agent and the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the condition for stopping the iteration is met, and the current answer is output.

2. The financial question-answering method based on multi-agent interaction according to claim 1, characterized in that: The obtaining of user input, performing query understanding on the user input by recalling related entity words and similar question-answer pairs, performing knowledge recall and tool recall based on query understanding, and filling the recall result into the prompt template of the planning agent includes: Obtain user input, and recall relevant entity words and a list of similar question-answer pairs in a financial entity word library and a historical question-answer library according to the user input: the list of similar question-answer pairs includes multiple similar question-answer pairs, and each similar question pair includes a similar question and a thought chain; Performing query understanding on the list according to the related entity words and similar questions and answers, and recalling knowledge fragment texts and candidate API lists in the expert knowledge base and tool library respectively based on the user query; The prompt template of the planning agent is filled in according to the relevant entity words, the list of similar question-answer pairs, the knowledge fragment text and the candidate API list.

3. The financial question-answering method based on multi-agent interaction according to claim 2 is characterized in that: Recall relevant entity words in the financial entity word library based on user input, including: According to the abbreviation of the financial entity in the user input, a mixed multi-way recall is performed in the financial entity vocabulary to obtain relevant entity words, and a fuzzy matching method is used for matching during the recall.

4. The financial question-answering method based on multi-agent interaction according to claim 2, characterized in that: The list of similar question-answer pairs recalled in the historical question-answer database based on user input includes: The fine-tuned similarity question discrimination model is used to predict the similarity between the user input and each historical input, and the historical input with a similarity greater than the threshold is regarded as a similar question; Corresponding thought chains are constructed according to similar questions, and a list of similar question-answer pairs is obtained according to each similar question and the corresponding thought chain.

5. The financial question-answering method based on multi-agent interaction according to claim 2, characterized in that: Recalling knowledge fragment text in the expert knowledge base based on user query includes: The content in the expert knowledge base is stored in blocks, an inverted index is created for the block data, and the block data is vectorized and stored in the vector retrieval engine. The relevant knowledge fragments are searched in the vector retrieval engine according to the vector representation of the user query, and the corresponding knowledge fragment text is queried according to the inverted index.

6. The financial question-answering method based on multi-agent interaction according to claim 2, characterized in that: The candidate API list recalled in the tool library based on the user query includes: The tool retrieves the multi-label classification model after fine-tuning the user query input to obtain the predicted probability of the user query input in several categories; the categories are tools for processing the user query; The categories with probability greater than the threshold are selected to obtain the candidate API list.

7. The financial question-answering method based on multi-agent interaction according to claim 1, characterized in that: Updates to the Planning Agent’s prompt template based on answers and suggestions output by the Reflection Agent include: The answers output by the summary agent and the suggestions output by the reflection agent are added to the prompt template of the planning agent to obtain an updated prompt template.

8. A financial question-answering system based on multi-agent interaction, characterized in that: The system includes a planning agent, an execution agent, a summary agent and a reflection agent; The planning agent answers the user input according to the prompt template, extracts the answer content through the parser, obtains the API execution link, and generates a directed acyclic graph according to the API execution link; the directed acyclic graph includes nodes and edges, the nodes are API interfaces, and the edges are the dependencies between API interfaces; the prompt template is obtained through the recall results of related entity word recall, similar question and answer pair recall, knowledge recall and tool recall; The execution agent calls the calling tool encapsulated by the relevant API to execute the tasks corresponding to the directed acyclic graph in parallel; The summarizing agent generates an answer based on the output of the executing agent, and the reflecting agent makes a re-planning judgment based on the answer of the summarizing agent. If re-planning is required, the prompt template of the planning agent is updated according to the answer and the suggestions output by the reflecting agent, and re-planning is performed according to the updated prompt template. The above process is iterated until the conditions for stopping the iteration are met, and the current answer is output.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images