Knowledge graph question and answer method based on Monte Carlo tree search and multi-agent cooperation

Abstract

The invention provides a Monte Carlo tree search and multi-agent cooperation-based knowledge graph question and answer method, which comprises the following steps that: a plurality of agents are initialized, and each agent has an initial process, a state and a target; a graph query statement is generated through multi-agent cooperation, and a complete track with scores is obtained; and utilizing a Monte Carlo tree search method, optimizing a query track through the steps of selection, expansion, simulation and back propagation according to the query track generated through multi-agent cooperation, generating an executable graph query statement, and finally outputting an answer set of the question. According to the method, the advantages of an end-to-end method and a step-by-step reasoning method are combined, meanwhile, the defects that the knowledge graph environment perception ability is insufficient and the search space is too large are overcome, dynamic generation, evaluation and optimization of the query path are achieved by designing a division and cooperation mechanism of the expansion agent, the simulation agent and the valuing agent, and the search efficiency is improved. Therefore, the accuracy and robustness of the question answering system are improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a knowledge graph question-answering method based on Monte Carlo tree search and multi-agent collaboration. Background Art

[0002] Large language models have made significant progress in the field of natural language processing due to their advanced text generation and semantic understanding capabilities. However, large language models often operate in a "black box" manner and are unable to explain their internal knowledge system, and often have problems with factual errors and hallucinations. To solve such problems, knowledge graph question answering technology uses knowledge graphs to answer questions, providing large language models with reliable and up-to-date knowledge and improving the interpretability of the system. Traditional knowledge graph question answering technology usually retrieves triples or subgraphs related to the question from the KB and converts the question into a graph query language to obtain an answer. With the emergence of large language models, the use of large language models to assist knowledge graph question answering technology has become mainstream.

[0003] The existing mainstream knowledge graph question answering technologies are divided into two categories: one is an end-to-end method that directly takes questions in natural language form as input and uses the relevant knowledge in the knowledge graph to assist the large language model to answer the questions. The whole process is completed in one step; the other is a step-by-step reasoning method that answers questions by reasoning on the knowledge graph step by step. However, both methods face challenges.

[0004] Among them, end-to-end methods can be divided into two categories: one is to first retrieve the triples and subgraphs related to the question from the knowledge graph based on the input, and use this knowledge as the context input of the large language model to assist it in answering; the other is to first convert the question in natural language into a logical graph query language, apply it to the knowledge graph, and directly get the answer to the question. However, the end-to-end knowledge graph question-answering method lacks a full understanding of the knowledge graph environment, and the generated graph queries often have similar and fixed structures.

[0005] In addition, knowledge graph question answering methods with step-by-step reasoning are prone to encountering problems with large-scale search spaces or local optimality. Such methods usually perform retrieval and generation alternately step by step around the subject entity, which can easily lead to extremely large search spaces. If errors occur in the intermediate steps of the search process, it may lead to local optimality. Summary of the invention

[0006] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0007] To this end, the first objective of this application is to propose a knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration.

[0008] The second objective of this application is to propose a knowledge graph question-answering device based on Monte Carlo tree search and multi-agent collaboration.

[0009] The third objective of the present application is to provide an electronic device.

[0010] A fourth objective of the present application is to provide a computer-readable storage medium.

[0011] A fifth object of the present application is to provide a computer program product.

[0012] To achieve the above objectives, the first embodiment of the present application proposes a knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration, including: Process natural language problems through prompt word engineering to initialize multiple agents, where each agent has an initial process, state and goal; Through multi-agent collaboration, graph query statements are generated to obtain a complete trajectory with a score. The extended agent provides multiple options for query tool selection based on the agent state and task objectives, and interacts with the knowledge graph through an unsupervised retrieval model to determine the best tool parameters. The simulation agent scores the candidate parameters generated by the extended agent and selects the option with the highest score as the optimal option. The value agent scores the generated complete trajectory and selects the trajectory with the highest score as the final answer. Using the Monte Carlo tree search method, based on the query trajectory generated by multi-agent collaboration, the query trajectory is optimized through selection, expansion, simulation, and backpropagation steps to generate executable graph query statements and finally output the answer set to the question.

[0013] Optionally, the processing of natural language problems through prompt word engineering to initialize multiple agents includes: Initialize the agent process, which includes: generating an initial prompt based on a given natural language question, and clarifying the task goal through prompt word engineering. The initial prompt includes a description of the knowledge graph question-answering task, a description of the query tools that the agent can call, and the given natural language question, forming a historical record of the agent ; Initialize the agent state, which consists of the agent history and the exploration sequence of the current task, and the exploration sequence is constructed by the think-action-observation triple Describe, among which, For the agent in Step by step thinking, For actions generated based on thinking, is the observed result after the action, The status of the step is ; Initialize the agent goal, which is to determine the operations to be performed in each step, including: identifying the top-level entities in the problem as the starting part and deciding the query tool to call; selecting the query tool and determining the required tool parameters based on the actions generated by thinking; generating new parameters through observation and updating the graph query statement until a complete graph query statement is formed.

[0014] Optionally, the extended agent includes: In the training phase, the extended agent is based on the training set Build real data based on predefined agent processes and transform the tool parameters in the state operation of all agents The choice is regarded as the output, and the corresponding previous state and current steps As instruction input, a instruction dataset is formed for fine-tuning, and fine-tuning is performed using the following loss function:

[0015] in, is the policy model of the extended agent, is the training data set, For the The tool parameters selected in step 1, is the previous state, For the The thinking part of the step; In the reasoning phase, after a tool is selected in a certain step, the trained extended agent is used to generate the parameters needed to call the tool. ; Use an unsupervised phrase-level retrieval model to match the parameters generated by the extended agent with the potential options in the knowledge graph, and select the top candidate with the highest matching degree. As the selected tool parameter, the matching formula is:

[0016] in, The highest matching tool parameters, For matching results, It is a knowledge graph; According to the matching results, the extended agent selects the tool parameters with the highest matching parameters as the selected parameters for the current step, and continues reasoning until the complete graph query statement is completed and the final answer set is derived.

[0017] Optionally, the simulated intelligent agent includes: During the training phase, the simulated agent uses the training set Initial prompt in and the final state of each step As input, the steps from the current state to the final conclusion are taken as output, a fully supervised fine-tuning dataset is constructed, and the following loss function is used for training:

[0018] in, is a strategy model of the simulated intelligent agent; In the reasoning phase, the simulated agent is used as a scoring model to provide a score for each candidate generated by the extended agent. The scoring formula is:

[0019] in, is the predefined full score value, The temperature factor to control the score difference; Based on the scoring results, the simulated agent selects the candidate with the highest score as the new option and continues the reasoning process until a final conclusion is reached.

[0020] Optionally, the value-taking agent includes: During the training phase, the value-taking agent uses the initial prompt As input, the complete agent process is taken as output and trained using the following loss function:

[0021] In the inference stage, the value-taking agent is used as a scoring model to score the generated complete trajectory. The scoring formula is:

[0022] According to the scoring results, the evaluation agent selects the trajectory with the highest score as the final selected complete reasoning trajectory, and generates an executable graph query statement through the trajectory to derive the final answer.

[0023] Optionally, the Monte Carlo tree search method is used to optimize the query trajectory through selection, expansion, simulation, and back-propagation steps based on the query trajectory generated by multi-agent collaboration, generate an executable graph query statement, and finally output the answer set of the question, including: The initial input is the natural language format question given by the user and the initialized agent, which is used to generate the state of each step; when a new search round begins, start from the root node and gradually search down the explored child nodes until the leaf node is reached; in each layer, starting from the current state, the next child node is selected according to the tree confidence bound algorithm. The child node represents a state in the agent process. The selection criterion is to maximize the balance between the Q value of the current state and the number of explorations. The selection formula is:

[0024] in, is the number of visits to the agent state during the Turkalo tree search, The candidate exploration set for triplets generated by the extended agent and the simulated agent, is the Q value of the agent state, To control the variables of exploration tendency; When the selection step reaches a non-terminal leaf node and does not exceed the maximum depth, the expanded agent generates multiple possible next states based on the current state and the simulated agent reorders these states, and the expanded node is added to the search tree as a new leaf node; After the expansion, the newly added child nodes are scored by the simulated agent, the node with the highest score is selected by the following formula, and the simulation is continued until the final conclusion is reached to generate a complete query generation trajectory, the formula is:

[0025] When the reasoning process reaches the final state, the agent calculates the final Q value based on the complete query trajectory evaluated, and back-propagates this Q value along the path to update the Q values ​​of all nodes on the path. The update formula is:

[0026] The Q value of the node is updated through back propagation, and finally the query trajectory with the highest weighted score is selected as the final generated graph query statement, and the answer set of the question is retrieved from the knowledge graph through this statement.

[0027] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a knowledge graph question-answering device based on Monte Carlo tree search and multi-agent collaboration, comprising: An initialization module, for processing natural language problems through prompt word engineering, and initializing multiple agents, wherein each agent has an initial process, a state, and a goal; The multi-agent collaboration module is used to generate graph query statements through multi-agent collaboration to obtain a complete trajectory with a score, wherein the extended agent provides multiple options for query tool selection according to the agent state and task objectives, and interacts with the knowledge graph through an unsupervised retrieval model to determine the best tool parameters; the simulation agent scores the candidate parameters generated by the extended agent and selects the option with the highest score as the optimal option; the value agent scores the generated complete trajectory and selects the trajectory with the highest score as the final answer; The Monte Carlo tree search module is used to use the Monte Carlo tree search method to optimize the query trajectory through selection, expansion, simulation, and backpropagation steps based on the query trajectory generated by multi-agent collaboration, generate executable graph query statements, and finally output the answer set of the question.

[0028] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects above.

[0029] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the above-mentioned first aspects.

[0030] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the method as described in any one of the above-mentioned first aspects.

[0031] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: By combining the Monte Carlo tree search and the knowledge graph question-answering method of multi-agent collaboration, the accuracy and efficiency of knowledge graph question-answering can be effectively improved. Through the collaboration between agents, agents of different roles can perform their respective duties and jointly optimize the query process, reducing the occurrence of errors, and dynamically selecting the optimal path during the reasoning process, thereby ensuring more accurate answers. In addition, the introduction of Monte Carlo tree search makes the entire question-answering process more exploratory and optimizeable, and can find the optimal solution in a larger search space. In addition, through the combination of fine-tuning and unsupervised retrieval models, the generalization ability and accuracy of the model can be continuously improved, ensuring that it can still effectively respond to diverse problems in a complex and dynamic knowledge graph environment.

[0032] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration provided in an embodiment of the present application; Figure 2 A flowchart of a knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a knowledge graph question-answering device based on Monte Carlo tree search and multi-agent collaboration provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0035] In response to the problems existing in the prior art, an embodiment of the present application provides a knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration.

[0036] Before explaining the technical solution of the present application, the premise definition of the present application is first explained.

[0037] In this application, the knowledge graph is defined as a large-scale graph structure, denoted as , by the entity set , Relationship Set , and triple sets Among them, the entity set Includes instance entity collection , ontology entity set , and literal entity collections The literal entity set can be further divided into the text description set , a collection of values , and time collection . Relations in a relation set Used to connect two entities, each triple in the triple set It is represented by the (entity, relationship, entity) structure, i.e. .

[0038] In order to interact with the knowledge graph and extract effective information, it is usually necessary to use graph queries Come to the knowledge graph Each graph query Can be represented as a list of functions , the list contains l steps, each of which performs a specific operation to promote the query process. This application designs query tools with specific names and parameters for the agent to generate functions. The role of each query tool is to extract or infer the required entities or relationships from the knowledge graph according to the task requirements.

[0039] The goal of this application is to answer a natural language question in the knowledge graph question answering task. and a knowledge graph , the problem Converted to a knowledge graph Graph queries executed on Execute the graph query After that, the result set is a set of answers that can be obtained from the knowledge graph, where the elements are the entity sets corresponding to the questions. Through this conversion and execution mechanism, accurate answers can be obtained from the knowledge graph, thereby achieving efficient solutions to natural language questions.

[0040] Figure 1 and Figure 2 A flow chart of a knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration provided in an embodiment of the present application. Figure 1 and Figure 2 As shown, the method comprises the following steps: Step 101, processing natural language problems through prompt word engineering, initializing multiple agents, wherein each agent has an initial process, a state and a goal.

[0041] In the embodiment of the present application, step 101 processes natural language problems through prompt word engineering to initialize multiple agents. Each agent includes initialization of the initial process, state and goal, and the specific steps are as follows: (1) Agent process initialization.

[0042] In the embodiment of the present application, the input for initializing the agent process is a given natural language question, such as "Which movie with a television running time of less than 60 minutes features Taylor Lautner?" For the agent process, it starts with an initial prompt, which includes an introduction to the knowledge graph question answering task, a description of the query tools that the agent can call, and the given natural language question. These contents constitute the history record of the agent. , and clarified the initial prompts for specific tasks through cue word engineering.

[0043] Specifically, the initial prompt contains the following parts: a description of the knowledge graph question-answering task: including the goal of the task, which is to derive the answer to the question from the knowledge graph; a description of the query tool: including the tools and functions that can be called by the agent, such as querying entities, relations, attributes, etc.; a given natural language question: for example, "Which movie with an atelevision running time of less than 60 minutes features Taylor Lautner?".

[0044] This information is used as the initial input, and the agent generates its initial prompt based on this content, clarifies the task goal, and initializes the state of the agent through the prompt. The output of this part is the agent generated by the given problem with a clear task goal and query tool description.

[0045] (2) Initialization of agent state.

[0046] In the embodiment of the present application, the initialization of the agent state includes generating an exploration sequence, and the agent uses the "thinking-action-observation" triple to describe the state of each step. Each triplet consists of the following parts: 1) The agent in the Steps of thinking , that is, the agent analyzes the task based on the current state and decides what action to take next. For example, thinking may include identifying which entities or relations from the question are relevant to the query. 2) Actions generated based on thinking , which indicates the specific operation that the agent needs to perform, such as calling a query tool or selecting query parameters. 3) The results observed after the action , usually the information returned by the query tool or the intermediate results produced during the reasoning process.

[0047] It can be understood that the exploration sequence is described by these triplets of thinking, acting, and observing, and the whole process continues until the task is completed. Step status From the initial state and from step 1 to step In addition, the observations of all steps can be combined to form the current function list , the list corresponds to the graph query statement.

[0048] This process will continue until the agent actively calls the "Finish" tool or the current expression exceeds the maximum length. , marking the end of the agent process. Finally, the agent evaluates the candidate queries and selects the optimal trajectory, and finally derives the graph query statement and the corresponding answer set Therefore, the final output of this section is an agent that clearly defines the format of the output for each subsequent step, and the agent will continue to output the thinking, action, and observation triples during reasoning.

[0049] (3) Initialization of the agent’s goals.

[0050] In the embodiment of the present application, the target initialization of the agent is mainly to determine the operations to be performed in each step. The operations of each step include the following parts: Thought: In the thought phase, the LLM (Large Language Model) thinks about the current operation that needs to be performed. Based on the question, the agent needs to decide how to act and choose the graph query tool to call. For example, in "Which movie with a television running time of less than 60 minutes features Taylor Lautner?", it first needs to identify the top entities in the question and decide which query tool to use to extract the relevant entities.

[0051] Action: In the action phase, the agent selects a query tool and determines its required parameters based on the operations generated by thinking. For example, in the above problem, the agent will select the Extract_entity tool to extract the top entity "Taylor Lautner" in the question and pass it as a parameter to the query tool.

[0052] Observation: After executing the query tool, the agent will generate new parameters and update the graph query statement based on the returned results. This process will continue until a complete graph query statement is formed. The ultimate goal of the agent goal initialization is to determine the actions to be performed in each step, specifically: by identifying the top-level entities in the problem as the starting part and deciding the query tool to call; selecting the query tool and determining the required tool parameters based on the actions generated by thinking; generating new parameters through observation and updating the graph query statement until a complete graph query statement is formed.

[0053] Through these steps, the agent is able to generate query statements that can be executed in the knowledge graph based on the given question and derive the final answer.

[0054] Through the initialization process of the above-mentioned intelligent agent, including the initialization of the intelligent agent process, state and goal, this application can ensure that the intelligent agent has clear steps and structures in the deduction from the initial natural language question to the final answer during the reasoning process. Each intelligent agent dynamically adjusts its thinking, actions and observations to efficiently interact with the knowledge graph, generate correct graph query statements, and finally derive the answer to the question.

[0055] Step 102, generate a graph query statement through multi-agent collaboration to obtain a complete trajectory with a score.

[0056] The embodiment of the present application designs and trains three agents: an extended agent, a simulation agent, and a value agent. The extended agent provides multiple options for query tool selection based on the agent state and task objectives, and interacts with the knowledge graph through an unsupervised retrieval model to determine the best tool parameters; the simulation agent scores the candidate parameters generated by the extended agent and selects the option with the highest score as the optimal option; the value agent scores the generated complete trajectory and selects the trajectory with the highest score as the final answer. More specifically: (1) Extending the agent.

[0057] The input of this part is the initialized agent. The extended agent is responsible for providing multiple alternative options for the selection of tool parameters after the agent selects the query tool. The work of the extended agent is divided into the training phase and the inference phase, which are described in detail below.

[0058] In the training phase, the extended agent is based on the training set Build real data based on predefined agent processes and transform the tool parameters in the state operation of all agents The choice is regarded as the output, and the corresponding previous state and current steps As instruction input, it constitutes the instruction dataset for fine-tuning. The extended agent is fine-tuned using the following loss function:

[0059] in, To expand the agent’s policy model, is the training data set, For the The tool parameters selected in step 1, is the previous state, For the The goal of this loss function is to maximize the probability of the extended agent selecting tool parameters at each step, thereby optimizing its reasoning effect.

[0060] In the inference phase, when the agent’s thinking selects the tool to be used, the extended agent will use the trained model to generate the parameters needed to call the tool. .

[0061] In addition, in order to enable the extended agent to interact effectively with the knowledge graph, the embodiment of the present application uses an unsupervised phrase-level retrieval model to match the parameters generated by the extended agent with the potential options in the knowledge graph, and selects the top candidate with the highest matching degree. As the selected tool parameter, the matching formula is:

[0062] in, The highest matching tool parameters, For matching results, For the knowledge graph.

[0063] Based on the matching results, the extended agent selects the tool parameters with the highest matching parameters as the selected parameters for the current step and continues reasoning until the complete graph query statement is completed and the final answer set is derived.

[0064] In one possible embodiment, assume that the task of the agent is to identify the top-level entities involved in the question. For example, for the question "Which movie with a television running time of less than 60 minutes features Taylor Lautner?", the extended agent will first determine that the Extract_entity tool needs to be called to extract the top-level entity. Based on the previous state and thinking, the extended agent generates the parameter "Taylor Lautner" as the query parameter, and then compares it with the relevant entities in the knowledge graph through the matching model, and finally confirms that the entity is the correct parameter for the query tool. The output of this part is the function parameter generated by the extended agent, which is used for subsequent graph query statement generation and answer derivation.

[0065] Through the above process, the extended agent can generate appropriate tool parameters according to the task and state in the reasoning stage, and interact efficiently with the knowledge graph through the unsupervised retrieval model, and finally generate the correct graph query statement and derive the final answer. Through the optimization of the training and reasoning stages, the extended agent can accurately and efficiently handle complex knowledge graph question-answering tasks.

[0066] (2) Simulating intelligent agents.

[0067] The input of this part is the options with high matching degree in the knowledge graph. After using the extended agent to determine the parameters for the tool selected in the current step, the simulation agent is responsible for providing scores for the candidates generated by the extended agent during the reasoning process, and selects the best option based on the score results to continue reasoning. The working process of the simulation agent includes the training phase and the reasoning phase, which are described in detail below.

[0068] During the training phase, the simulated agent uses the training set Initial prompt in and the final state of each step As input, the steps from the current state to the final conclusion are taken as output. This data is constructed as a fully supervised fine-tuning dataset for the simulated agent to learn how to generate scores for different candidate options. In this way, the simulated agent is able to judge which options are most likely to lead to the correct answer based on the state and thinking during the reasoning phase. The loss function of the training process is as follows:

[0069] in, is the policy model of the simulated agent. The goal of this loss function is to maximize the probability of each step of the simulated agent to ensure that the agent can make reasonable choices during the reasoning process. Through training, the simulated agent learns how to score each candidate according to the input state and thinking, and then can select the best option for the candidate generated by the extended agent.

[0070] In the inference phase, the simulated agent acts as a scoring model, and based on each candidate generated by the extended agent and current status Provide a score for each candidate. The simulated agent uses the score to determine which options are more likely to lead to the correct answer, and ultimately selects the highest-scoring option and advances the reasoning process.

[0071] During the inference process, the scoring formula is:

[0072] in, is the predefined full score value, is the temperature factor that controls the fractional difference.

[0073] After scoring, the simulated agent selects the candidate with the highest score as the optimal option for the current step based on the calculated score. For example, suppose the extended agent generates two candidate options: Option 1: Rating is 99.8; Option 2: Rating is 98.7.

[0074] In this case, the simulated agent selects the option with a score of 99.8 as the final query tool call parameter for the current step.

[0075] In the embodiment of the present application, after selecting the best candidate, the simulated agent will continue to reason until a complete query statement is generated and the final answer set is derived. Each step depends on the selection of the previous step, and the simulated agent ensures that the optimal path is selected at each step through a scoring mechanism.

[0076] In one possible implementation, assume that the agent is processing the question: "Which movie with an audience running time of less than 60 minutes features Taylor Lautner?" During the reasoning process, the extended agent first identifies the top entity "Taylor Lautner" in the question and selects the relevant query tool to search. Then, the simulated agent selects the entity based on the current state. Score the generated candidate options. Suppose the following two candidates are generated: Extract_entity[Taylor Lautner], rated 99.8; Find_relation[film.actor.film] with a score of 98.7.

[0077] The simulated agent will select the highest-scoring Extract_entity[TaylorLautner] as the final option based on the scoring results, continue reasoning, and eventually generate a complete query statement and deduce the answer to the question.

[0078] Through the above process, the simulated agent scores each candidate in the reasoning process, selects the best option, and gradually advances the reasoning process to finally obtain the optimal answer. Through the fully supervised training and scoring mechanism in the reasoning stage, the simulated agent can effectively optimize the selection in complex reasoning tasks and improve the accuracy and efficiency of knowledge graph question and answering.

[0079] (3) Value-taking agent.

[0080] The input of this part is the trajectory generated by the agent after determining the parameters called by the query tool. In addition to using the simulation agent to score each option, this application also designs a value-taking agent to score the generated complete trajectory and select the optimal reasoning trajectory based on the scoring results, thereby generating the final graph query statement and deriving the answer to the question. The work of the value-taking agent includes a training phase and a reasoning phase, which are described in detail below.

[0081] During the training phase, the agent uses the initial prompt As input, the complete agent process is taken as output, and the generated inference trajectory will be used as the training dataset. During the training process, the following loss function is used for fine-tuning to optimize the scoring ability of the value-taking agent:

[0082] The goal of the loss function is to minimize the prediction error of the value-taking agent when generating a complete reasoning trajectory and maximize the probability of generating the correct trajectory. Through fine-tuning, the value-taking agent can effectively assign scores to the generated complete trajectories, thereby making accurate choices during reasoning.

[0083] In the inference phase, the value-taking agent is used as a scoring model to evaluate the complete inference trajectory. Scoring is performed and the optimal reasoning trajectory is selected based on the scoring results. The scoring formula is:

[0084] Rating value Represents the score of the entire reasoning trajectory, and the agent determines whether the trajectory is optimal based on this score.

[0085] In the reasoning phase, the evaluation agent will select the highest-scoring trajectory as the final selected complete reasoning trajectory based on the score. This optimal trajectory will be used as input to generate the final graph query statement, and then deduce the answer to the question. For example, assuming there are multiple reasoning trajectories, the evaluation agent will select the trajectory with the highest score, and generate an execution query statement through this trajectory to retrieve the answer from the knowledge graph.

[0086] In the embodiment of the present application, by selecting the highest-scoring reasoning path, the agent generates the final graph query statement and submits it to the knowledge graph for execution, and finally derives the answer to the question. This process ensures that the selected reasoning path is optimal, thereby ensuring the accuracy of the answer to the question.

[0087] In one possible embodiment, assume that the agent is processing the question: "Which movie with an audience running time of less than 60 minutes features Taylor Lautner?" During the reasoning process, the extended agent generates multiple candidates, the simulated agent scores each candidate, and finally generates multiple reasoning trajectories, each trajectory representing a different query path. The value-taking agent selects the trajectory with the highest score as the final selected reasoning path based on the scores of these trajectories.

[0088] In general, the value-taking agent scores the generated complete reasoning trajectory to ensure that each step of the reasoning process can be carried out along the optimal path. Through fine-tuning in the training phase and the scoring mechanism in the reasoning phase, the value-taking agent effectively improves the accuracy of the question-answering system and ensures that the final query statement is generated based on the optimal reasoning trajectory, thereby accurately deriving the answer to the question.

[0089] Step 103, using the Monte Carlo tree search method, based on the query trajectory generated by multi-agent collaboration as mentioned above, optimize the query trajectory through selection, expansion, simulation, and backpropagation steps, generate an executable graph query statement, and finally output the answer set of the question.

[0090] In the reasoning phase, the embodiment of the present application uses the Monte Carlo Tree Search (MCTS) method, and the three agents (expansion agent, simulation agent, and value agent) trained above collaborate to execute the predefined query generation agent process. Monte Carlo Tree Search is a heuristic algorithm in the form of a tree structure, in which each node represents a state of the agent process. The algorithm starts from the root node representing the initial state, and iteratively explores and enriches the search tree through selection, expansion, simulation, and back-propagation steps. A total of N rounds of search are performed, and in the first The agent process can be expressed by the agent state trajectory as: .

[0091] Each step is described in detail below.

[0092] (1) Selection steps:

[0093] The input of this part is the natural language format question initially given by the user and the initialized agent, which is used to generate the state of each step.

[0094] When a new search round begins, the search starts from the root node and gradually searches down to the explored child nodes until the leaf node is reached. In each layer, starting from the current state, the next child node is selected according to the Upper Confidence Bound applied to Trees (UCT) algorithm. The child node represents a state in the agent process. The selection criteria are to maximize the balance between the Q value of the current state and the number of explorations. The selection formula is:

[0095] in, is the number of visits to the agent state during the Turkalo tree search, The candidate exploration set for triplets generated by the extended agent and the simulated agent, is the Q value of the agent state, To control the variables of exploration tendency, Larger values ​​encourage exploration of less visited nodes.

[0096] In each round of search, the algorithm selects a child node that is most likely to lead to correct reasoning. By balancing the Q value of the current state and the number of explorations, the child node that maximizes the balance between the Q value of the current state and the number of explorations is selected. For example, in the reasoning process, assuming that the current reasoning has reached the first step, it is necessary to select from the child nodes of the current step according to the scoring algorithm shown in the simulated agent. If the Q value of the current child node 1 is 99.89, the operation is Find_relation[film.actor.film], and the Q value of child node 2 is 96.97, the operation is Find_relation[tv.tv_actor.starting_roles], and the number of visits to the two is the same, then child node 1 will be selected next and integrated with the current existing nodes.

[0097] The output of this step is a Monte Carlo tree that selects the specified child node in MCTS, allowing the agent to continue exploring downward.

[0098] (2) Expansion step.

[0099] The expansion step occurs after the selection step and is only executed when the selected node is a non-terminal node and the maximum depth has not been reached. In this step, the expansion agent generates multiple possible next states based on the current state, and these states are reordered by the simulation agent. The expanded nodes are added to the search tree as new leaf nodes.

[0100] For the above problem, suppose that after selecting the above sub-node 1, it is found that the leaf node has been reached. Then the first step is to generate B new leaf nodes by expanding the agent, that is, B triplets consisting of Thought-Action-Observation. Next, use the SimCSE retrieval model to match these B triplets with the knowledge graph, and select the child nodes ranked in the top d in similarity in the knowledge graph as the final expanded child nodes. According to the above problem, this step expands two new leaf nodes on the operation Find_relation[film.actor.film] of sub-node 1, namely Extract_entity

[60] and Find_relation[film.performance.film]. The final output of this part is the Monte Carlo tree with expanded leaf nodes.

[0101] (3) Simulation steps:

[0102] The input of this part is the Monte Carlo tree with expanded leaf nodes. After the expansion step, the simulation step is used to score the newly added child nodes through the simulated agent, select the node with the highest score through the following formula, and continue the simulation until the final conclusion is reached to generate a complete query generation trajectory. The formula is:

[0103] After the expansion of the two subnodes in the previous step, the simulated agent selects the subnode Find_relation[film.performance.film] for execution because it has a high score. After execution, the agent infers that the current expression can solve the problem and enters the completed state. The output of this part is the exploration sequence of the policy model that has been updated by executing the subnode with the highest score.

[0104] (4) Propagation steps:

[0105] When the reasoning process reaches the final state, the evaluation agent scores the entire query trajectory generated, and combines the simulation agent's score for the last step to calculate the Q value of the final state. The calculation formula is as follows:

[0106] in, The final state The Q value, The simulated agent scores the nodes generated in the last step, is the score of the generated complete query trajectory by the evaluation agent, is a weight parameter used to balance the process score and the overall score. This process calculates the Q value of the final state by combining the scores of the simulation agent and the value agent, thus providing a basis for the subsequent backpropagation step.

[0107] After calculating the Q value of the final state, the algorithm will backpropagate the Q value along the reasoning path and update the Q values ​​of all nodes on the path. The specific update formula is as follows:

[0108] Through back propagation, the algorithm updates the Q values ​​of all nodes on the path so that the nodes in the path get the score information from the terminal state. This update ensures that each node in the tree can optimize its score based on the final reasoning result.

[0109] After completing the backpropagation update, the Monte Carlo Tree Search algorithm explores different query trajectories through multiple search rounds. After each round of search, all generated query trajectories are compared based on their score values. Finally, the algorithm selects the query trajectory with the highest weighted score as the final generated query trajectory. Specifically, after completing After a round of searching, Among the trajectories generated by the round search, the trajectory with the highest score is selected as the final query trajectory. By splicing the "observation" part of the selected trajectory into a function list, an executable SPARQL query statement is finally formed.

[0110] Finally, the selected query trace is converted into an executable SPARQL query statement, which retrieves the answer set of the question from the knowledge graph. SPARQL (SPARQL Protocol and RDF Query Language) is a standardized query language for querying knowledge graphs (RDF format).

[0111] In one possible embodiment, assume that the agent is processing the question: "Which movie with an audience running time of less than 60 minutes features Taylor Lautner?". Assume that through the MCTS process, the simulation agent and the evaluation agent calculate the final state as "Finish". At this time, the evaluation agent will calculate the final Q value and propagate it back along the reasoning path, first updating the Q value of the node Find_relation[film.performance.film], and then propagating the Q value up along the path to update the Q values ​​of other nodes on the path.

[0112] For example, in this process: suppose the Q value of the Find_relation[film.performance.film] node is updated to 99.8; then, the Q values ​​of other nodes on the path (such as Extract_entity

[60] ) will be updated in turn.

[0113] Finally, the query trajectory with the highest weighted score is selected (e.g., Find_relation[film.actor.film] to Extract_entity

[60] ), which is converted into an executable SPARQL query statement, and the answer set to the question is retrieved by querying the knowledge graph.

[0114] The propagation step is the last step in the Monte Carlo tree search, which ensures the optimal path selection for the entire search process. By calculating the Q value of the final state, back-propagating to update the Q value of the node, and selecting the query trajectory with the highest score, an executable graph query statement is finally generated. This process not only optimizes the query trajectory, but also ensures that the query statement can accurately retrieve the answer set of the question from the knowledge graph.

[0115] In order to implement the above embodiments, the present application also proposes a knowledge graph question-answering device based on Monte Carlo tree search and multi-agent collaboration. Figure 3 A schematic diagram of the structure of a knowledge graph question-answering device based on Monte Carlo tree search and multi-agent collaboration provided in an embodiment of the present application. Figure 3 As shown, the device comprises: An initialization module 100 is used to process natural language problems through prompt word engineering and initialize multiple agents, wherein each agent has an initial process, a state and a goal; The multi-agent collaboration module 200 is used to generate a graph query statement through multi-agent collaboration to obtain a complete trajectory with a score, wherein the extended agent provides multiple options for query tool selection according to the agent state and task objectives, and interacts with the knowledge graph through an unsupervised retrieval model to determine the best tool parameters; the simulation agent scores the candidate parameters generated by the extended agent and selects the option with the highest score as the optimal option; the value agent scores the generated complete trajectory and selects the trajectory with the highest score as the final answer; The Monte Carlo tree search module 300 is used to utilize the Monte Carlo tree search method, based on the query trajectory generated by multi-agent collaboration, optimize the query trajectory through selection, expansion, simulation, and back-propagation steps, generate executable graph query statements, and finally output a set of answers to the question.

[0116] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0117] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0118] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0119] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0120] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0121] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0122] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0123] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0124] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0126] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0129] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0131] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A knowledge graph question answering method based on Monte Carlo tree search and multi-agent collaboration, characterized in that: include: Process natural language problems through prompt word engineering to initialize multiple agents, where each agent has an initial process, state and goal; Through multi-agent collaboration, graph query statements are generated to obtain a complete trajectory with a score. The extended agent provides multiple options for query tool selection based on the agent state and task objectives, and interacts with the knowledge graph through an unsupervised retrieval model to determine the best tool parameters. The simulation agent scores the candidate parameters generated by the extended agent and selects the option with the highest score as the optimal option. The value agent scores the generated complete trajectory and selects the trajectory with the highest score as the final answer. Using the Monte Carlo tree search method, based on the query trajectory generated by multi-agent collaboration, the query trajectory is optimized through selection, expansion, simulation, and backpropagation steps to generate executable graph query statements and finally output the answer set to the question.

2. The method according to claim 1, characterized in that The process of processing natural language problems through prompt word engineering and initializing multiple agents includes: Initialize the agent process, which includes: generating an initial prompt based on a given natural language question, and clarifying the task goal through prompt word engineering. The initial prompt includes a description of the knowledge graph question-answering task, a description of the query tools that the agent can call, and the given natural language question, forming a historical record of the agent ; Initialize the agent state, which consists of the agent history and the exploration sequence of the current task, and the exploration sequence is constructed by the think-action-observation triple Describe, among which, For the agent in Step by step thinking, For actions generated based on thinking, is the observed result after the action, The status of the step is ; Initialize the agent goal, which is to determine the operations to be performed in each step, including: identifying the top-level entities in the problem as the starting part and deciding the query tool to call; selecting the query tool and determining the required tool parameters based on the actions generated by thinking; generating new parameters through observation and updating the graph query statement until a complete graph query statement is formed.

3. The method according to claim 2, characterized in that The extended agent includes: In the training phase, the extended agent is based on the training set Build real data based on predefined agent processes and transform the tool parameters in the state operation of all agents The choice is regarded as the output, and the corresponding previous state and current steps As instruction input, a instruction dataset is formed for fine-tuning, and fine-tuning is performed using the following loss function: in, is the policy model of the extended agent, is the training data set, For the The tool parameters selected in step 1, is the previous state, For the The thinking part of the step; In the reasoning phase, after a tool is selected in a certain step, the trained extended agent is used to generate the parameters needed to call the tool. ; Use an unsupervised phrase-level retrieval model to match the parameters generated by the extended agent with the potential options in the knowledge graph, and select the top candidate with the highest matching degree. As the selected tool parameter, the matching formula is: in, The highest matching tool parameters, For matching results, It is a knowledge graph; According to the matching results, the extended agent selects the tool parameters with the highest matching parameters as the selected parameters for the current step, and continues reasoning until the complete graph query statement is completed and the final answer set is derived.

4. The method according to claim 3, characterized in that The simulated intelligent agent includes: During the training phase, the simulated agent uses the training set Initial prompt in and the final state of each step As input, the steps from the current state to the final conclusion are taken as output, a fully supervised fine-tuning dataset is constructed, and the following loss function is used for training: in, is a strategy model of the simulated intelligent agent; In the reasoning phase, the simulated agent is used as a scoring model to provide a score for each candidate generated by the extended agent. The scoring formula is: in, is the predefined full score value, is the temperature factor controlling the score difference; Based on the scoring results, the simulated agent selects the candidate with the highest score as the new option and continues the reasoning process until a final conclusion is reached.

5. The method according to claim 4, characterized in that The value-taking agent includes: During the training phase, the value-taking agent uses the initial prompt As input, the complete agent process is taken as output and trained using the following loss function: In the inference stage, the value-taking agent is used as a scoring model to score the generated complete trajectory. The scoring formula is: According to the scoring results, the evaluation agent selects the trajectory with the highest score as the final selected complete reasoning trajectory, and generates an executable graph query statement through the trajectory to derive the final answer.

6. The method according to claim 5, characterized in that The Monte Carlo tree search method is used to optimize the query trajectory through selection, expansion, simulation, and back-propagation steps based on the query trajectory generated by multi-agent collaboration, generate an executable graph query statement, and finally output the answer set of the question, including: The initial input is the natural language format question given by the user and the initialized agent, which is used to generate the state of each step; when a new search round begins, start from the root node and gradually search down the explored child nodes until the leaf node is reached; in each layer, starting from the current state, the next child node is selected according to the tree confidence bound algorithm. The child node represents a state in the agent process. The selection criterion is to maximize the balance between the Q value of the current state and the number of explorations. The selection formula is: in, is the number of visits to the agent state during the Turkalo tree search, The candidate exploration set for triplets generated by the extended agent and the simulated agent, is the Q value of the agent state, To control the variables of exploration tendency; When the selection step reaches a non-terminal leaf node and does not exceed the maximum depth, the expanded agent generates multiple possible next states based on the current state and the simulated agent reorders these states, and the expanded node is added to the search tree as a new leaf node; After the expansion, the newly added child nodes are scored by the simulated agent, the node with the highest score is selected by the following formula, and the simulation is continued until the final conclusion is reached to generate a complete query generation trajectory, the formula is: When the reasoning process reaches the final state, the agent calculates the final Q value based on the complete query trajectory evaluated, and back-propagates this Q value along the path to update the Q values ​​of all nodes on the path. The update formula is: The Q value of the node is updated through back propagation, and finally the query trajectory with the highest weighted score is selected as the final generated graph query statement, and the answer set of the question is retrieved from the knowledge graph through this statement.

7. A knowledge graph question-answering device based on Monte Carlo tree search and multi-agent collaboration, characterized in that: include: An initialization module, for processing natural language problems through prompt word engineering, and initializing multiple agents, wherein each agent has an initial process, a state, and a goal; The multi-agent collaboration module is used to generate graph query statements through multi-agent collaboration to obtain a complete trajectory with a score, wherein the extended agent provides multiple options for query tool selection according to the agent state and task objectives, and interacts with the knowledge graph through an unsupervised retrieval model to determine the best tool parameters; the simulation agent scores the candidate parameters generated by the extended agent and selects the option with the highest score as the optimal option; the value agent scores the generated complete trajectory and selects the trajectory with the highest score as the final answer; The Monte Carlo tree search module is used to use the Monte Carlo tree search method to optimize the query trajectory through selection, expansion, simulation, and backpropagation steps based on the query trajectory generated by multi-agent collaboration, generate executable graph query statements, and finally output the answer set of the question.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images