An intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph

Abstract

This invention discloses an intelligent agent method for complex question-answering reasoning based on a large-scale model and a person knowledge graph. The method includes: dynamically matching prompt examples to the question to be answered based on multiple recall methods, including any of embedding vector similarity matching, string fuzzy matching, and fusion matching; using a small number of prompts, utilizing a large language model to identify key entities in the question to be answered, screen relationships, and recursively search multiple candidate entities to obtain a set of candidate entities for each sub-question of the question to be answered; based on the set of candidate entities, utilizing the large language model to iteratively generate thoughts and actions in an intelligent manner, and through observations obtained through interaction with the knowledge graph, obtain the final answer after multiple rounds of interaction. This invention enhances the reasoning ability of the large language model with high-quality prompts.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing technology, and specifically relates to an intelligent agent method for complex question-answering reasoning based on a large model of character knowledge graph. Background Art

[0002] In recent years, the emergence of pre-trained language models has led to significant improvements in the performance of large language models (LLMs) across various natural language processing tasks. LLMs, thanks to extensive pre-training on large text corpora, are capable of solving complex reasoning problems involving complex knowledge relationships and mathematical problems requiring intricate derivation.

[0003] However, despite their impressive performance in many tasks, LLMs still face challenges due to their lack of robust factual knowledge, which can lead to inaccurate or misleading information—a phenomenon often referred to as "hallucination." To alleviate this problem, various approaches have been proposed to integrate knowledge graphs with large language models to address the challenges of knowledge-based question answering (KBQA). These approaches leverage the factual knowledge provided by knowledge graphs while leveraging the powerful language understanding and reasoning capabilities of large language models.

[0004] Previous work on question-answering reasoning in knowledge bases can be divided into two categories: methods based on information retrieval (IR) and methods based on semantic parsing (SP). Semantic parsing methods typically involve converting natural language questions into a symbolic logical form that a machine can interpret as an executable query. This conversion allows the system to retrieve answers directly from the knowledge base by mapping the semantics of the question to the structured data contained in the knowledge graph. In contrast, IR-based methods start with entities and iteratively explore the knowledge graph until the final answer is found.

[0005] For character attribute knowledge graph question answering tasks, character-related questions exhibit two obvious characteristics: (1) they often involve numerical statistics and calculations, such as querying the number of albums released by a singer in 2024, and (2) they often involve complex reasoning about character relationships, such as querying who someone's nephew is, but this information is not explicitly stored and needs to be inferred based on brother and son relationships, which is computationally intensive and time-consuming. Summary of the Invention

[0006] To address these technical issues, this paper proposes an intelligent agent-based approach for complex question-answering reasoning based on a large-scale person knowledge graph. This approach effectively interacts with the knowledge graph through an iterative reasoning framework and proposes a dynamic example retrieval method to enhance the reasoning capabilities of LLMs.

[0007] An intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph, the method comprising:

[0008] Dynamically matching prompt examples to the question to be answered based on multiple recall methods, including embedding vector similarity matching, string fuzzy matching, and fusion matching;

[0009] Using a few-sample prompt method, a large language model is used to identify key entities in the question to be answered, filter relationships, and recursively search multiple candidate entities to obtain the candidate entity set for each sub-question of the question to be answered;

[0010] Based on the set of candidate entities, a large language model is used to iteratively generate thoughts and actions in an intelligent agent manner, and the observation results are obtained through interaction with the knowledge graph, and the final answer is obtained after multiple rounds of interaction.

[0011] Furthermore, the embedding vector similarity matching includes embedding all entries in the corpus into a vector space, encoding each given question and example question as a vector, calculating the similarity between the two vectors, selecting several most similar questions in the corpus, and constructing prompt examples using the selected several most similar questions.

[0012] Furthermore, the string fuzzy matching includes using Levenshtein Distance to measure the similarity between two strings:

[0013] ,

[0014] in, and Represents two strings, Represents a string length, Represents a string The first character of represents the substring from the second character to the end, express and The similarity of two strings.

[0015] Furthermore, the fusion matching includes combining retrieval methods based on embedding vector similarity matching and string fuzzy matching to generate a fusion retriever, and the fusion retriever uses the inverse sort fusion score to combine multiple sort lists into a fusion sort.

[0016] Furthermore, the method of using a few sample prompts and utilizing a large language model to perform key entity recognition, relationship screening, and recursive search of multiple candidate entities for the question to be answered, and obtaining a candidate entity set for each sub-question of the question to be answered includes:

[0017] The large language model autonomously identifies key entities in the question based on context examples, matches and links the extracted key entities to entities in the knowledge graph, searches for all relationships linked to the entities, calculates the semantic similarity between all relationships and the sub-question currently being considered, selects the specific relationship with the highest correlation, and obtains the set of candidate entities associated with the current sub-question.

[0018] Furthermore, based on the set of entities to be selected, the large language model is used to iteratively generate thoughts and actions in an intelligent agent manner, and the observation results are obtained through interaction with the knowledge graph, and the final answer is obtained after multiple rounds of interaction. The method includes: based on the set of entities to be selected, the large language model is used to iteratively generate thoughts and actions in a multi-round dialogue manner, and the next step of thoughts and actions is generated according to the observation results generated by each round of actions, and the answer to the question to be answered is finally obtained after multiple rounds of interaction.

[0019] The beneficial effects of the present invention are:

[0020] This paper proposes an agent-based iterative reasoning framework that leverages a large language model. It interacts with a knowledge graph to faithfully and reliably execute reasoning. Furthermore, a method for dynamically prompted example recall is proposed to enhance the reasoning capabilities of the large language model with high-quality prompts. Experiments demonstrate that this method achieves promising results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a principle block diagram of an intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph in the present invention;

[0022] Figure 2 Schematic diagram of multiple rounds of interaction in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0024] Figure 1This paper presents a schematic diagram of the principles of an intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph. As can be seen, this method primarily consists of three steps: dynamic recall of prompt examples, knowledge graph interaction, and multi-round iterative reasoning. These three steps are described in detail below.

[0025] Step 1: Dynamically match the corresponding prompt examples according to the question to be answered based on multiple recall methods, including any one of embedding vector similarity matching, string fuzzy matching, and fusion matching: The goal of this step is to use string matching and sentence embedding vector similarity matching to dynamically recall the corresponding examples according to the question to be answered. In addition, in order to meet the challenge of generalization, this method also adds some more complex question templates when constructing the example set to improve the reasoning ability of the model. The following introduces the construction process of the example set: Since the training set only contains question-answer pairs and the detailed reasoning steps are omitted, it is necessary to construct examples that include solution steps to obtain high-quality prompts. In order to save costs, the training set is used to extract the corresponding examples from the training set. 500 questions are randomly selected from Represent the elements of the training set for questions and answers respectively. Problem Sets in Each question in , using a large language model to generate trajectories and from Get the final answer If the result Equal to the marked answer ,Will In addition, 60 samples adapted from the training set problems were manually annotated to meet the challenges of more complex problems in the final test set. Finally, the example set For subsequent retrieval and recall, three recall methods are used, which are introduced one by one below:

[0026] Embedding vector similarity matching: Semantic search aims to improve search accuracy by fully understanding the semantics of the search query and the corpus. The basic concept of semantic search involves embedding all entries in the corpus (whether it is a sentence, paragraph or document) into a vector space. Given a question , applying a dense text encoder will retrieve the corpus for the given question and each example question Then, the similarity between these question vectors is calculated and selected in the search corpus. The selected questions and their reasoning steps are then used to construct prompts.

[0027] String fuzzy matching: Compared to vector-based retrieval, the main advantage of fuzzy matching is its robustness to word variations and grammatical errors. Furthermore, by leveraging keyword matching, fuzzy matching can improve the recall rate of similar queries. Specifically, the Levenshtein distance is used to measure the similarity between two strings, defined as follows:

[0028]

[0029] in, and Represents two strings, Represents a string length, Represents a string The first character of represents the substring from the second character to the end, express and The Levenshtein distance between two strings, that is, the similarity between the two.

[0030] Fusion matching: Combines vector-based and string-based fuzzy matching retrieval methods to generate a fusion searcher. The fusion searcher uses the Reciprocal Rank Fusion (RRF) score to combine multiple ranked lists into a more comprehensive ranking. Given a ranked list based on vectors and a ranked list based on string fuzzy matching, the element The RRF score is calculated as follows:

[0031] ,

[0032] in, and Presentation Elements The ranking of the vector matching results and the string fuzzy matching results in the sorted list. If the element If it is not in a list, Should be positive infinity. represents a constant to suppress the influence of lower rankings, Indicates the rank of element d in a sorted list of a certain type of matching results, i.e., vector matching results or string fuzzy matching results.

[0033] Step 2: Using a few-shot prompt approach, we leverage a large language model to identify key entities, filter relationships, and recursively search multiple candidate entities to obtain candidate entity sets for each sub-question. The following is an introduction to these three steps:

[0034] (1) Key entity recognition: Use a large language model to extract key entities from the current thinking in the process. The large language model follows the examples in the context and returns through the parameters in the operation. As Figure 2 shown (Observation 1), when the sub-question is "need to know what Wang Jianjie's music works are" (Thought 1), the large language model will generate the operation "Search[Wang Jianjie]", where the key entity "Wang Jianjie" will be extracted for subsequent use.

[0035] (2) Entity linking (i.e., recursive search for multiple candidate entities): The extracted key entities will be matched and linked to the entities in the knowledge graph. However, not all extracted entities can be located in the knowledge graph, which may be because the extracted entities are incomplete or have other aliases. If no corresponding entity can be matched, semantic similarity matching will be performed on all entities in the knowledge graph, and the longest string matching will be used to obtain the finally extracted entity.

[0036] (3) Relationship exploration (i.e., relationship screening): After determining the entity most relevant to the current thinking, relationship exploration is performed on this entity. Initially, all relationships linked to the entity are searched. Subsequently, the semantic similarity between these relationships and the sub-question of the current thinking is calculated to determine the top relationships ranked by similarity. Then these top relationships and their relationship directions are presented to the large language model to finally select the specific relationship with the highest relevance. In particular, for some bidirectional relationships, such as "partner" and "friend", the extracted entity will be used as the head entity and the tail entity to search and obtain the final set of target entities.

[0037] After performing the above three processes, the set of target entities most relevant to the current thinking, that is, the triples associated with the current sub-question, will be obtained. To balance precision and recall and avoid problems related to the excessive context length of the large language model, the part of the target entity set with more than m elements will be replaced with "...", as Figure 2 shown (Observation 1). All target entity sets will be retained for subsequent stages and can be referenced as parameters in the operation, such as "observation_1".

[0038] Step 3, use the large language model to iteratively generate thoughts, actions in the way of an agent, and the observation results obtained by interacting with the knowledge graph, and get the final answer after multiple rounds of interaction:

[0039] Given a question , first, the large language model generates an initial thought To decompose the original problem into the first sub-problem. In each round except the first round, the large language model should generate a thinking , which summarizes the observations obtained from the previous round and plans the sub-problems for the next round. Then, the large language model needs to generate an action from the action space , the action space provides tools to help large language models explore knowledge graphs, perform logical operations, or perform statistical counting. After the action, the returned result is used as the observation of the current round When in action Encountered , the result is parsed into the final answer. Therefore, each trajectory generated from the large language model The definition is as follows:

[0040] ,

[0041] The following are the tools defined by the action space that can be called by the large language model:

[0042] Basic Operations: This category of tools mainly helps large language models interact with knowledge graphs and filter existing data. It includes retrieving candidate triples by interacting with knowledge graphs ( ), select the intermediate entity based on the second hop result ( ) and get the final answer ( ) operation.

[0043] Logical combination class: This class is designed to help large language models perform logical operations on extracted entity sets, which are determined by entity set intersection ( ), Union ( ) and the difference ( )composition.

[0044] Statistics and Counting Classes: This category focuses on enabling models to perform statistical analysis and counting operations on extracted data. It includes functions for counting the number of elements in a collection ( ), determine the maximum and minimum elements in the set ( ), compare the sizes of two sets ( ) and checking whether a specific element exists in a collection ( ) function.

[0045] In order to illustrate the effectiveness of the system, the present invention conducted experiments to verify the performance of the method.

[0046] About the test corpus:

[0047] CCKS2024 Person Knowledge Graph Complex Reasoning Evaluation Dataset: The knowledge graph for this task provides 256,378 entities, 240 relations, and 785,553 triples related to person relationships. The training set contains 1,238 question-answer pairs. Performance is evaluated on two different test sets: a preliminary test set with 1,000 samples and a final test set with 2,000 samples. The experimental results are shown in Table 1.

[0048] Table 1

[0049]

[0050] Experimental results show that our method achieved an F1 score of 0.8177 in the preliminary test set, ranking first on the leaderboard, and won first prize in the finals with an F1 score of 0.8213. The results also demonstrate that by integrating a dynamic example retriever and self-consistency technology, our iterative knowledge graph reasoning framework achieves outstanding performance, effectively addressing the complex reasoning challenges of character knowledge graph question answering.

[0051] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph, characterized by: The method comprises: Based on multiple recall methods, the corresponding prompt examples are dynamically matched according to the question to be answered. The multiple recall methods include any one of embedding vector similarity matching, string fuzzy matching, and fusion matching. The fusion matching includes combining retrieval methods based on embedding vector similarity matching and string fuzzy matching to generate a fusion retriever. The fusion retriever uses the reciprocal sort fusion RRF score to combine multiple ranked lists into a fusion ranking: ， in, and Represents an element in a fused sort Ranking in the sorted list of embedding vector similarity matching results and string fuzzy matching results, represents a constant used to suppress the impact of lower rankings; Using a few-sample prompt method, a large language model is used to identify key entities in the question to be answered, filter relationships, and recursively search multiple candidate entities to obtain the candidate entity set for each sub-question of the question to be answered; Based on the set of candidate entities, a large language model is used to iteratively generate thoughts and actions in an intelligent agent manner, and the observation results are obtained through interaction with the knowledge graph, and the final answer is obtained after multiple rounds of interaction.

2. The intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph according to claim 1 is characterized in that: The embedding vector similarity matching includes embedding all entries in the corpus into a vector space, encoding each given question and example question into a vector, calculating the similarity between the two vectors, selecting several most similar questions in the corpus, and constructing prompt examples using the selected several most similar questions.

3. The intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph according to claim 1 is characterized in that: The string fuzzy matching involves measuring the similarity between two strings using Levenshtein Distance: ， in, and Represents two strings, Represents a string length, Represents a string The first character of represents the substring from the second character to the end, express and The similarity of two strings.

4. The intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph according to claim 1 is characterized in that: The method of using a few sample prompts and utilizing a large language model to perform key entity recognition, relationship screening, and recursive search of multiple candidate entities for the question to be answered, and the candidate entity sets for each sub-question of the question to be answered include: The large language model autonomously identifies key entities in the question based on context examples, matches and links the extracted key entities to entities in the knowledge graph, searches for all relationships linked to the entities, calculates the semantic similarity between all relationships and the sub-question currently being considered, selects the specific relationship with the highest correlation, and obtains the set of candidate entities associated with the current sub-question.

5. The intelligent agent method for complex question-answering reasoning based on a large-scale character knowledge graph according to claim 1 is characterized in that: The method comprises: based on the entity set to be selected, iteratively generating thoughts and actions by the large language model in an intelligent agent manner, and obtaining observation results through interaction with the knowledge graph, and obtaining the final answer after multiple rounds of interaction, including: based on the entity set to be selected, iteratively generating thoughts and actions by the large language model in a multi-round dialogue manner, and generating the next step of thoughts and actions based on the observation results generated by each round of actions, and finally obtaining the answer to the question to be answered after multiple rounds of interaction.

Images