Intelligent agent method for complex question and answer reasoning of figure knowledge graph based on large model

Abstract

The invention discloses an intelligent agent method for complex question and answer reasoning of a figure knowledge graph based on a large model. The method comprises the steps that on the basis of multiple recall modes, corresponding prompt examples are dynamically matched according to questions to be answered, and the multiple recall modes comprise any one of embedded vector similarity matching, character string fuzzy matching and fusion matching; key entity recognition, relation screening and multi-candidate entity recursion search of the question to be answered are carried out by using a large language model in a few-sample prompt mode, and a candidate entity set of each sub-question of the question to be answered is obtained; and on the basis of the to-be-selected entity set, thinking and actions are iteratively generated in an intelligent agent mode by utilizing a large language model, and through an observation result obtained through interaction with the knowledge graph, a final answer is obtained after multiple rounds of interaction. According to the method, the reasoning ability of the large language model is enhanced with high-quality prompts.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph. Background Art

[0002] In recent years, the emergence of pre-trained language models has led to significant improvements in the performance of large language models (LLMs) in various natural language processing tasks. Thanks to extensive pre-training on large-scale text corpora, LLMs are able to solve complex reasoning problems involving complex knowledge relations, as well as mathematical problems that require complex derivation processes.

[0003] However, despite their excellent performance in many tasks, LLMs still face challenges due to the lack of strong factual knowledge judgments, which can lead to inaccurate or misleading information - a phenomenon often referred to as "hallucination". To alleviate this problem, various methods that integrate knowledge graphs with large language models have been proposed to address the challenges of Knowledge-based Question Answering (KBQA). These methods take advantage of the factual knowledge provided by knowledge graphs while leveraging the powerful language understanding and reasoning capabilities of large language models.

[0004] Previous work on question-answering reasoning in knowledge bases can be divided into two categories: methods based on information retrieval (IR) and methods based on semantic parsing (SP). Methods based on semantic parsing typically involve converting natural language questions into symbolic logical form, which can be interpreted by machines as executable queries. This conversion allows the system to retrieve answers directly from the knowledge base by mapping the semantics of the question to structured data contained in the knowledge graph. In contrast, methods based on information retrieval start from entities and iteratively explore the knowledge graph until the final answer is found.

[0005] For character attribute knowledge graph question answering tasks, character-related questions exhibit two distinct characteristics: (1) they often involve numerical statistics and calculations, such as querying the number of albums released by a singer in 2024, and (2) they often involve complex reasoning about character relationships, such as querying who someone’s nephew is. However, this information is not explicitly stored and needs to be inferred based on brother and son relationships, which requires large amounts of computation and is time-consuming. Summary of the invention

[0006] To solve the above technical problems, the present invention proposes an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph. The method effectively interacts with the knowledge graph through an iterative reasoning framework and proposes a dynamic example retrieval method to enhance the reasoning ability of LLMs.

[0007] An intelligent agent method for complex question-answering reasoning based on a large-model character knowledge graph, the method comprising: Based on multiple recall methods, dynamically matching corresponding prompt examples according to the questions to be answered, the multiple recall methods include any one of embedding vector similarity matching, string fuzzy matching, and fusion matching; By adopting the method of few-sample prompts and using a large language model, key entity recognition, relationship screening, and recursive search of multiple candidate entities are performed on the question to be answered, so as to obtain the candidate entity set for each sub-question of the question to be answered; Based on the set of candidate entities, a large language model is used to iteratively generate thoughts and actions in an intelligent agent manner, and the observation results are obtained through interaction with the knowledge graph, and the final answer is obtained after multiple rounds of interaction.

[0008] Furthermore, the embedding vector similarity matching includes embedding all entries in the corpus into the vector space, encoding each given question and example question as a vector, calculating the similarity between the two vectors, selecting several most similar questions in the corpus, and constructing prompt examples using the selected several most similar questions.

[0009] Furthermore, the string fuzzy matching includes using Levenshtein Distance to measure the similarity between two strings: , in, and Represents two strings, Represents a string Length, Represents a string The first character of represents the substring from the second character to the end, express and Similarity of two strings.

[0010] Furthermore, the fusion matching includes combining retrieval methods based on embedded vector similarity matching and string fuzzy matching to generate a fusion retriever, and the fusion retriever uses the inverse sort fusion score to combine multiple sorted lists into a fusion sort.

[0011] Furthermore, the method of using a few sample prompts and using a large language model to perform key entity recognition, relationship screening, and recursive search of multiple candidate entities for the question to be answered, and obtaining a candidate entity set for each sub-question of the question to be answered includes: The large language model autonomously identifies key entities in the question based on context examples, matches and links the extracted key entities to entities in the knowledge graph, searches for all relationships linked to the entities, calculates the semantic similarity between all relationships and the sub-question currently being considered, selects specific relationships with the highest correlation, and obtains a set of candidate entities associated with the current sub-question.

[0012] Furthermore, based on the entity set to be selected, the large language model is used to iteratively generate thoughts and actions in an intelligent agent manner, and the observation results are obtained through interaction with the knowledge graph, and the final answer is obtained after multiple rounds of interaction. The method includes: based on the entity set to be selected, the large language model is used to iteratively generate thoughts and actions in a multi-round dialogue manner, and the next step of thoughts and actions are generated according to the observation results generated by each round of actions, and the answer to the question to be answered is finally obtained after multiple rounds of interaction.

[0013] The beneficial effects of the present invention are: This paper proposes an agent iterative reasoning framework using a large language model, which can interact with the knowledge graph to faithfully and reliably perform the reasoning process. In addition, a method for dynamic prompt example recall is proposed to enhance the reasoning ability of the large language model with high-quality prompts. Experiments show that the method of the present invention achieves good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a principle block diagram of an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph in the present invention; Figure 2 Schematic diagram of multiple rounds of interaction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0016] Figure 1 The principle diagram of the intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph is given. It can be seen that the method mainly consists of the following three steps: dynamic recall of prompt examples, knowledge graph interaction, and multi-round iterative reasoning. These three steps will be introduced in detail below.

[0017] Step 1: Dynamically match the corresponding prompt examples according to the question to be answered based on multiple recall methods, including any one of embedding vector similarity matching, string fuzzy matching, and fusion matching: The goal of this step is to use string matching and sentence embedding vector similarity matching to dynamically recall the corresponding examples according to the question to be answered. In addition, in order to meet the challenges of generalization, this method also adds some more complex question templates when constructing the example set to improve the reasoning ability of the model. The following introduces the construction process of the example set: Since the training set only contains question-answer pairs and the detailed reasoning steps are omitted, it is necessary to construct examples that include solution steps to obtain high-quality prompts. In order to save costs, the training set is used to extract the corresponding examples. 500 questions are randomly selected from Respectively represent the elements of the training set for questions and answers. Problem Sets Each question in , using a large language model to generate trajectories , and from Get the final answer in If the result Equal to the marked answer ,Will In addition, 60 samples adapted from the training set problems were manually annotated to meet the challenges of more complex problems in the final test set. Finally, the example set To perform subsequent retrieval and recall, three recall methods are used, which are introduced one by one below: Embedding vector similarity matching: Semantic search aims to improve search accuracy by fully understanding the semantics of the search query and the corpus. The basic concept of semantic search involves embedding all entries in the corpus (whether it is a sentence, paragraph, or document) into a vector space. Given a question , applying a dense text encoder will retrieve the given question in the corpus and each example question Then, the similarity between these question vectors is calculated and selected in the search corpus. The selected question and its reasoning steps are then used to construct a prompt.

[0018] String fuzzy matching: Compared with vector-based retrieval, the main advantage of fuzzy matching is its robustness to word changes and grammatical errors. In addition, due to the use of keyword matching, fuzzy matching can improve the recall rate of similar questions. Specifically, Levenshtein Distance is used to measure the similarity between two strings, which is defined as follows:

[0019] in, and Represents two strings, Represents a string Length, Represents a string The first character of represents the substring from the second character to the end, express and The Levenshtein Distance of two strings, that is, the similarity between the two.

[0020] Fusion matching: Combine vector-based and string-based fuzzy matching retrieval methods to generate a fusion searcher. The fusion searcher uses the Reciprocal Rank Fusion (RRF) score to combine multiple ranked lists into a more comprehensive ranking. Given a ranked list based on vectors and a ranked list based on string fuzzy matching, the element The RRF score is calculated as follows: , in, and Representation elements The ranking of the vector matching results and the string fuzzy matching results in the sorted list. If the element If it is not in a list, Should be positive infinity. represents a constant to suppress the influence of lower rankings, Indicates the rank of element d in a sorted list of a certain kind of matching results, i.e., vector matching results or string fuzzy matching results.

[0021] Step 2: Use a few-sample prompt method to use a large language model to identify key entities, filter relationships, and recursively search multiple candidate entities to obtain the candidate entity set for each sub-question. The following is an introduction to these three processes: (1) Key Entity Recognition: Use a large language model to extract key entities from the current thinking process. The large language model follows examples in context and uses The parameters in the operation are returned. Figure 2 As shown (Observation 1), when the sub-question is "I need to know what musical works Wang Jianjie has" (Thinking 1), the large language model will generate the operation "Search[Wang Jianjie]", in which the key entity "Wang Jianjie" will be extracted for subsequent use.

[0022] (2) Entity linking (i.e., recursive search of multiple candidate entities): The extracted key entities will be matched and linked to the entities in the knowledge graph. However, not all extracted entities can be located in the knowledge graph, which may be due to incomplete extracted entities or other aliases. If the corresponding entity cannot be matched, semantic similarity matching will be performed on all entities in the knowledge graph, and the longest string match will be used to obtain the final extracted entity.

[0023] (3) Relationship exploration (i.e., relationship screening): After determining the entity most relevant to the current thinking, perform relationship exploration on the entity. Initially, search for all relationships linked to the entity. Then, calculate the semantic similarity between these relationships and the sub-problem currently being thought about, and determine the top relationships ranked by similarity. Then these previous The relationships and their directions are presented to the large language model, so as to finally select the specific relationship with the highest relevance. In particular, for some bidirectional relationships, such as "partner" and "friend", the extracted entities are used as the head entity and the tail entity to search and obtain the final target entity set.

[0024] After executing the above three processes, the target entity set most relevant to the current thinking, that is, the triples associated with the current sub-question, will be obtained. In order to balance precision and recall, and avoid the problem of too long context length in large language models, the part of the target entity set that exceeds m elements is replaced with "...", such as Figure 2 All target entity sets are retained for use in subsequent stages and can be referenced as parameters in operations, such as "observation_1".

[0025] Step 3: Use the large language model to iteratively generate thoughts and actions in an intelligent agent manner, and obtain observations through interaction with the knowledge graph. After multiple rounds of interaction, the final answer is obtained: Given a question , first the initial thinking is generated by the large language model To decompose the original problem into the first sub-problem. In each round except the first round, the large language model should generate a thinking , which summarizes the observations obtained from the previous round and plans the sub-problems for the next round. Then, the large language model needs to generate an action from the action space , the action space provides tools to help large language models explore knowledge graphs, perform logical operations, or perform statistical counting. After the action, the returned result is used as the observation of the current round When in action Encountered , the result is parsed into the final answer. Therefore, each trajectory generated from the large language model The definition is as follows: , The following are the tools defined by the action space that can be called by the large language model: Basic Operations: Tools in this category mainly help large language models interact with knowledge graphs and filter existing data. It includes retrieving candidate triples by interacting with knowledge graphs ( ), select the intermediate entity based on the second hop result ( ) and get the final answer ( ) operation.

[0026] Logical combination class: This class is designed to help large language models perform logical operations on extracted entity sets, consisting of entity set intersections ( ), Union ( ) and difference ( ) composition.

[0027] Statistics and Counting Classes: This category focuses on enabling models to perform statistical analysis and counting operations on extracted data. It includes functions for counting the number of elements in a collection ( ), determine the largest and smallest elements in a set ( ), compare the sizes of two sets ( ) and checking whether a specific element exists in a collection ( ) function.

[0028] In order to illustrate the effectiveness of the system, the present invention conducted experiments to verify the performance of the method.

[0029] About the test corpus: CCKS2024 Person Knowledge Graph Complex Reasoning Evaluation Dataset: The knowledge graph of this task provides 256,378 entities, 240 relations, and 785,553 triples related to personal relations. The training set contains 1238 question-answer pairs. The performance is evaluated on two different test sets, the preliminary test set has 1,000 samples and the final test set has 2,000 samples. The experimental results are shown in Table 1.

[0030] Table 1

[0031]

[0032] The experimental results show that the method of the present invention achieved 0.8177 in the preliminary test set, ranked first on the leaderboard, and won the first prize in the final with an F1 score of 0.8213. It can also be seen from the results that by integrating the dynamic example retriever and self-consistency technology, the iterative knowledge graph reasoning framework of the present invention can achieve excellent performance and effectively cope with the challenges of complex reasoning in character knowledge graph question answering.

[0033] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph, characterized in that: The method comprises: Based on multiple recall methods, dynamically matching corresponding prompt examples according to the questions to be answered, the multiple recall methods include any one of embedding vector similarity matching, string fuzzy matching, and fusion matching; By adopting the method of few-sample prompts and using a large language model, key entity recognition, relationship screening, and recursive search of multiple candidate entities are performed on the question to be answered, so as to obtain the candidate entity set for each sub-question of the question to be answered; Based on the set of candidate entities, a large language model is used to iteratively generate thoughts and actions in an intelligent agent manner, and the observation results are obtained through interaction with the knowledge graph, and the final answer is obtained after multiple rounds of interaction.

2. According to claim 1, an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph, characterized in that: The embedding vector similarity matching includes embedding all entries in the corpus into the vector space, encoding each given question and example question as a vector, calculating the similarity between the two vectors, selecting several most similar questions in the corpus, and constructing prompt examples using the selected several most similar questions.

3. According to claim 1, an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph, characterized in that: The string fuzzy matching involves using Levenshtein Distance to measure the similarity between two strings: ， in, and Represents two strings, Represents a string Length, Represents a string The first character of represents the substring from the second character to the end, express and Similarity of two strings.

4. According to claim 1, an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph, characterized in that: The fusion matching includes combining retrieval methods based on embedded vector similarity matching and string fuzzy matching to generate a fusion retriever, and the fusion retriever uses the inverse sort fusion score to combine multiple sort lists into a fusion sort.

5. According to claim 1, the intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph is characterized in that: The method of using a few sample prompts and using a large language model to perform key entity recognition, relationship screening, and recursive search of multiple candidate entities for the question to be answered, and the candidate entity sets for each sub-question of the question to be answered include: The large language model autonomously identifies key entities in the question based on context examples, matches and links the extracted key entities to entities in the knowledge graph, searches for all relationships linked to the entities, calculates the semantic similarity between all relationships and the sub-question currently being considered, selects specific relationships with the highest correlation, and obtains a set of candidate entities associated with the current sub-question.

6. According to claim 1, an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph, characterized in that: The method comprises: based on the entity set to be selected, iteratively generating thoughts and actions by the large language model in an intelligent agent manner, and obtaining observation results through interaction with the knowledge graph, and obtaining the final answer after multiple rounds of interaction, including: based on the entity set to be selected, iteratively generating thoughts and actions by the large language model in a multi-round dialogue manner, and generating the next step of thoughts and actions according to the observation results generated by each round of actions, and finally obtaining the answer to the question to be answered after multiple rounds of interaction.

Images