Question and answer method and system combining large language model and knowledge graph

Abstract

The invention discloses a question and answer method and system combining a large language model and a knowledge graph, and relates to the field of question and answer, and the method comprises the steps: determining corresponding meta-knowledge based on a meta-knowledge base according to a retrieval question, and the meta-knowledge comprises knowledge description, query actions and question entities; the meta-knowledge base comprises a knowledge graph; the meta-knowledge and short-term memory are combined to obtain cue word information, then a large language model is adopted to execute a query action and complete current reasoning, and an obtained response result is evaluated to obtain a reward score; if the reward score does not meet the preset condition, generating reflection correction information according to the response result, respectively storing the reflection correction information and the cue word information into a long-term memory and a short-term memory to realize data updating, updating the meta-knowledge base based on the updated long-term memory, and then executing the next round of reasoning; and if the reward score meets a preset condition, marking the response result as a question and answer result. The knowledge graph-based question and answer accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of question answering, and in particular to a question answering method and system combining a large language model with a knowledge graph. Background Art

[0002] Existing KGQA (Knowledge Graph Question Answering) methods have certain limitations when dealing with complex problems, especially in multi-round reasoning tasks, where existing methods are easily affected by the increase in data volume, resulting in decreased reasoning accuracy. In addition, existing methods mostly rely on static knowledge graphs or simple retrieval techniques, and fail to effectively utilize the reasoning capabilities of large language models for complex graph reasoning tasks. Summary of the invention

[0003] The purpose of this application is to provide a question-answering method and system that combines a large language model with a knowledge graph, which can improve the accuracy of question-answering based on the knowledge graph.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a question-answering method that combines a large language model with a knowledge graph, comprising: determining corresponding meta-knowledge based on a retrieval question based on a meta-knowledge base; the meta-knowledge includes knowledge description, query action and question entity; the meta-knowledge base includes a knowledge graph; combining the meta-knowledge with short-term memory to obtain prompt word information; based on the prompt word information, using a large language model to execute the query action, and completing the current reasoning to obtain a corresponding response result; evaluating the response result to obtain a reward score; if the reward score does not meet a preset condition, generating reflection correction information based on the response result; storing the prompt word information and the corresponding reflection correction information in long-term memory and short-term memory respectively to implement data update, and updating the meta-knowledge base based on the updated long-term memory, and then returning to the step of determining the corresponding meta-knowledge based on the meta-knowledge base according to the retrieval question to perform the next round of reasoning; if the reward score meets the preset condition, marking the response result as a question-answering result.

[0006] In the second aspect, the present application provides a question-answering system that combines a large language model with a knowledge graph, including a knowledge retrieval module, a self-reflection mechanism module and a multi-storage memory module; the multi-storage memory module includes a short-term memory and a long-term memory; the knowledge retrieval module is used to: determine the corresponding meta-knowledge based on the meta-knowledge base according to the retrieval question; the meta-knowledge includes knowledge description, query action and question entity; the meta-knowledge base includes a knowledge graph; the short-term memory is used to: combine with the meta-knowledge to obtain prompt word information; the self-reflection mechanism module is used to: based on the prompt word information, use a large language model to execute the query action, and complete the current reasoning to obtain the corresponding response result; evaluate the response result to obtain a reward score; if the reward score does not meet the preset conditions, generate reflection correction information according to the response result; store the prompt word information and the corresponding reflection correction information in the long-term memory and the short-term memory respectively to realize data update; if the reward score meets the preset conditions, mark the response result as a question-answering result; the long-term memory is used to: update the meta-knowledge base.

[0007] According to the specific embodiments provided by the present application, the present application has the following technical effects: the present application sets up a meta-knowledge base to provide corresponding meta-knowledge for retrieval questions, which can assist in understanding the questions and provide a data basis for the accuracy of subsequent answers. Combining meta-knowledge with short-term memory can guide the subsequent large language model to effectively extract and absorb information from the knowledge graph when performing query actions. After executing the query action and completing the current reasoning and obtaining the response result, the response result is evaluated to obtain a reward score. When it does not meet the preset conditions, reflective correction information is generated, and then stored in short-term memory and long-term memory, and the meta-knowledge base is updated, and then the next round of reasoning is performed until the reward score meets the preset conditions. So far, the present application introduces a self-reflection mechanism, which, as a dynamic and iterative process, can continuously optimize and refine the interaction between the large language model and the knowledge graph, and through the generation and update of reflective correction information, continuously analyze previous results and adjust future actions accordingly, thereby improving the reliability and accuracy of question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0009] Figure 1 A flowchart of the question-answering method combining a large language model and a knowledge graph provided in this application.

[0010] Figure 2 Schematic diagram of the question-answering system that combines a large language model and a knowledge graph provided for this application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] This application proposes a knowledge graph question-answering method with self-reflection ability, which solves existing problems by combining with knowledge graphs and provides a more flexible and efficient KGQA method.

[0013] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0014] In an exemplary embodiment, Figure 1 and Figure 2 As shown, in an embodiment of the present application, the question-answering method combining a large language model with a knowledge graph includes the following steps 101 to 107.

[0015] Step 101, based on the meta-knowledge base, determine the corresponding meta-knowledge according to the search question; the meta-knowledge includes knowledge description, query action and question entity; the meta-knowledge base includes a knowledge graph.

[0016] In one application example, the meta-knowledge base is used to help the language model learn how to use the knowledge graph. The meta-knowledge base retrieves knowledge from a specialized database to obtain strategies for the large language model to learn about the knowledge graph, including understanding its structure, relationships, and retrieval methods. In this process, the large language model acquires the ability to efficiently use the structured knowledge in the knowledge graph, and constantly refers to this meta-knowledge to guide retrieval, significantly enhancing the model's ability to formulate correct strategies and understand complex knowledge graph structures and relationships.

[0017] Specifically, the meta-knowledge base includes an experience database and a priori database; the experience database includes procedural memory, semantic memory and episodic memory; the priori database contains procedural memory and semantic memory; wherein the procedural memory includes operational knowledge of the knowledge graph; the semantic memory includes conceptual understanding of entities, attributes and relationships in the knowledge graph; the episodic memory includes any historical retrieval question and the corresponding knowledge graph retrieval process. After receiving the retrieval question vector q input by the user, a preliminary search is performed in the meta-knowledge base KB (prior database) and the knowledge graph KG to obtain relevant knowledge, namely meta-knowledge, as the basis for subsequent reasoning. In a specific application, the extraction of procedural memory from the priori knowledge base is mainly composed of the following three key components: knowledge base retrieval, knowledge graph retrieval and sorting.

[0018] Among them, knowledge base retrieval: In order to retrieve the knowledge graph documents most relevant to the retrieval question vector q from the knowledge base, the most advanced embedding model is used to represent the query and document (or document fragment) as vectors. By calculating the cosine similarity between these vectors, the top k most relevant documents are selected. As a supplement, BM25 can also be used for text-based retrieval, using word frequency and inverse document frequency to improve the retrieval effect. Specifically, the following formula is used to match the relevant documents of the retrieval question:

[0019] D q =KR(q,KB).

[0020] Among them, Dq is the set of relevant documents retrieved from the meta-knowledge base according to the retrieval question vector, KB is the meta-knowledge base, and KR is the knowledge retrieval module.

[0021] The cosine similarity is calculated using the following formula:

[0022]

[0023] Among them, cosine_similarity(q,d) is the cosine similarity between the retrieval question vector q and the relevant document d.

[0024] The ranking of related documents is calculated using the following ranking function:

[0025]

[0026] Among them, BM25(q,d) is the ranking of the search question vector q and the related document d, n is the number of search terms in the search question; IDF(q i ) is the i-th search term vector q i The inverse document frequency is used to measure the importance of the query term; tf(q i ,d) is the i-th search term vector qi The number of times it appears in the relevant document d, i.e., the word frequency; k1 and b are both adjustment parameters, and avgdl is the average document length.

[0027] k1 is used to control the saturation of word frequency tf. Its value is usually between [1.2, 2.0], and the default value is 1.5. If k1 is large, the weight of word frequency increases; if k1 is small, the influence of word frequency weakens. b is a parameter that controls the influence of document length. Its value range is [0,1], and the default value is 0.75. The closer b is to 1, the greater the influence of document length. If b = 0, the influence of document length is ignored. The influence of document length: 1) Short documents: The keyword density is high, and the score may be high; BM25 will balance its influence through length normalization. 2) Long documents: The vocabulary is rich, but it may contain more irrelevant content, and the score may be amplified; BM25 will penalize the influence of long documents through parameters b and document length. In summary, BM25 adjusts the influence of document length on the score through b to ensure that the scores of short and long documents are more fair and avoid score imbalances caused by document length deviations.

[0028] Knowledge Graph Retrieval: To retrieve the structural information of the nodes and edges that are most relevant to the query question vector q from the knowledge graph, metadata embedding is introduced in the query architecture, thereby facilitating the application of sophisticated vector-based similarity retrieval algorithms that are critical in accurately locating relevant meta-knowledge. Specifically, this approach makes it possible to retrieve knowledge related to knowledge graph personalization, query strategies, and related entities, attributes, and relationships from a wide range of database repositories. By embedding these metadata vectors, not only is the ability to identify complex entities and their interrelationships enhanced, but the multi-dimensional attributes embedded in the data are also understood. The sophisticated vector retrieval process is carefully designed to locate entity and relationship patterns to retrieve targeted expertise from the knowledge graph. This retrieval process is based on a sophisticated query method that intelligently maps queries to corresponding metadata vectors, ensuring that the search results are both relevant and accurate. The operations associated with this method involve the steps taken by the model to further adjust, contextualize, and convey information. These operations are tailored to the details of the query and are dynamically adjusted as the complexity of the database evolves, ensuring that the model's response is not only accurate, but also contextually rich and supported by the latest metadata insights.

[0029] In the final result of graph search, LLM is used to convert the retrieved local structured information consisting of nodes and edges into text descriptions. The formula is as follows:

[0030] D m =LLM(G m ).

[0031] Among them, Dm is the meta-knowledge document and Gm is the retrieval question.

[0032] Sorting: refers to re-ranking the results retrieved from the knowledge base and knowledge graph using the inverted ranking fusion algorithm. The RRF formula is as follows:

[0033]

[0034] Among them, r i (d) is the rank of the relevant document d in the ith ranked list, n is the number of ranked lists, and k is a constant (usually set to 60 to avoid division by small values); this algorithm is only applicable to documents retrieved from the knowledge base, while the text description obtained from the structured knowledge graph is used as supplementary information. The combined output forms the meta-knowledge document, which is calculated as follows: D M =D q +D m .

[0035] The algorithm is applied only to documents retrieved from the knowledge base, while the textual description derived from the structured knowledge graph is used as supplementary information, and the synthesized output is the meta-knowledge document. This document helps to generate precise query strategies, which may involve requesting specific entity properties or finding specific relations in the knowledge graph.

[0036] In addition, this step also involves: 1) Utilization of knowledge graph: The large language model repeatedly accesses the knowledge graph to extract task-related information and generates responses based on the query. 2) Context and feedback learning: By tracking the conversation context and adjusting the meta-knowledge base based on feedback, the query strategy is optimized to improve the efficiency and accuracy of future queries.

[0037] Step 102, combining the meta-knowledge with the short-term memory to obtain prompt word information.

[0038] Step 103, based on the prompt word information, use the large language model to execute the query action, complete the current reasoning, and obtain the corresponding response result. This step generates a series of reasoning steps (including various decisions and their results in the current reasoning process) by simulating the interaction process between the action and the environment, providing data support for the next step of evaluation. The environment mentioned above refers to components such as Ma, Me, Mip, long-term memory, knowledge graph, etc., and the action refers to the interaction between the large language model and such components, such as evaluating the answer, retrieving knowledge, and storing the results.

[0039] The implementation of steps 103-107 is based on a self-correction framework, which is designed to support complex language model applications and integrates three key models: Meta-Executor (Me), Meta-Assessor (Ma), and Meta-Insight Processor (Mip). This framework emphasizes the generation and application of meta-knowledge. By continuously learning and refining the knowledge base from each interaction, the framework ensures that large language models become more and more efficient in processing complex queries. By combining meta-executors, meta-assessors, and meta-insight processors based on a self-correction framework, we can use meta-knowledge to solve complex queries, overcome knowledge gaps, and resolve logical inconsistencies in the question-answering process.

[0040] In an application example, step 103 is implemented by a meta-executor. The meta-executor generates necessary text and actions based on the observed state of the environment through a large language model. In addition, models such as chain thinking are integrated into the meta-executor to explore different levels of text generation and action execution, thereby optimizing the question-answering process based on the knowledge graph. The meta-executor is also equipped with a memory component to enhance its context perception and decision-making capabilities.

[0041] Step 104, evaluate the response result to obtain a reward score. In an application example, step 104 is implemented by a meta-evaluator. The meta-evaluator evaluates the trajectory generated by the meta-executor and calculates a reward score reflecting the performance in a specific question-answering task. The step of evaluating the response result to obtain a reward score includes: evaluating the response result from three dimensions: whether the response result is hallucination information, whether the response result is consistent with the knowledge graph, and whether the response result is consistent with the query, to obtain a reward score. In the specific details of the evaluation, not only the accuracy-based score is considered, but also different large language model instances themselves can be tried as evaluators to generate reward scores. The above multi-dimensional evaluation method enables fine-tuning the question-answering process to ensure that the output is closely aligned with the expected answer of the knowledge graph. Among them, the evaluation process of whether the response result is hallucination information is: cross-validation with the meta-knowledge base to identify whether the response result contains non-factual or false content. The evaluation process of whether the response result is consistent with the knowledge graph is: checking whether the response result is consistent with the known facts in the knowledge graph. The evaluation process of whether the response result is consistent with the query is to evaluate the relevance of the response result to the retrieval question. By combining the iterations in the subsequent steps and continuously learning these evaluation results, the meta-evaluator also continuously optimizes its scoring mechanism and feeds performance insights back to the meta-knowledge base to improve the quality of future evaluations.

[0042] Step 105, if the reward score does not meet the preset conditions, reflective correction information is generated according to the response result; the reflective correction information helps to improve the reasoning strategy by correcting errors and optimizing the reasoning process, and ultimately improves the overall question and answer accuracy. In an application example, step 105 is implemented by a meta-insight processor. The meta-insight processor plays a key role by solving two core challenges: knowledge gaps and logical inconsistencies in generated answers. This active learning mechanism introduced by the meta-insight processor fundamentally changes the reflection process, enabling large language models to provide more accurate and comprehensive answers by introducing targeted knowledge supplementation and logical review. Specifically, the present application introduces three specific reflection dimensions, enhances the reflection framework, and overcomes the limitations existing in previous systems. That is, the step of generating reflective correction information according to the response result includes:

[0043] 51) Based on the active supplement model, the knowledge missing information is identified according to the response results; many hallucinations or errors are caused by incomplete or wrong information retrieval. Mip identifies the areas of missing knowledge through the output generated by the active supplement model and prompts the large language model to generate more comprehensive and complete answers. This ensures that the output of the large language model is not only accurate but also contextually rich, minimizing knowledge gaps, thereby avoiding the further spread of misunderstandings or misinformation.

[0044] 52) Performing logical consistency focusing on the response result to obtain logical correction information; wherein, during the logical consistency focusing process, the response result is logically checked from three dimensions: whether the response result is hallucination information (whether the generated answer contains non-factual or irrelevant information), whether the response result is consistent with the knowledge graph (whether the result is consistent with the verified external data source), and whether the response result is consistent with the query (whether the answer fully and correctly responds to the original intention of the query), and corresponding corrections are performed.

[0045] The reflective framework focuses on evaluating the logical consistency of the meta-evaluator output. After evaluating the initial trajectory generated by the meta-executor, Mip checks whether there are logical contradictions, conflicts with the knowledge graph retrieval results, or deviations from the original query intent. If inconsistencies are found, the system will re-execute the process and realign the output to ensure logical and factual correctness. In other words, in this process, knowledge from multiple sources is integrated and feedback loops are learned to correct logical errors detected by Mip and re-execute the task using insights from the meta-knowledge base. This process enables large language models to be highly adaptable and error-resistant.

[0046] 53) Perform logical evaluation and knowledge supplement on the process of identifying the missing knowledge information and the process of determining the logical correction information to obtain reflective correction information. This dimension introduces an additional level of reflection to the reflective process: the evaluation of the evaluation. If Mip does not find significant logical flaws in the previous steps, it allows the system to generate the final output. However, if inconsistencies or errors are found in the logical evaluation, it will return to the reflection and correction cycle and re-execute. This iterative process ensures not only accuracy, but also the adaptability of the model when processing complex queries.

[0047] Specifically, a logical evaluation and knowledge supplement are performed on the process of identifying the missing knowledge information and the process of determining the logical correction information to obtain reflective correction information, including: if there are logical defects in the process of identifying the missing knowledge information and the process of determining the logical correction information, the missing knowledge information and the logical correction information are discarded, and the knowledge missing information is re-identified and the logical correction information is re-determined; if there are no logical defects in the process of identifying the missing knowledge information and the process of determining the logical correction information, the reflective correction information is determined based on the knowledge missing information and the logical correction information.

[0048] By introducing the above three reflection frameworks, the self-correction ability of the large language model is significantly improved, making the output more accurate and error-resistant. Through active learning and iterative self-reflection in subsequent steps, Mip can ensure that each step is carefully evaluated, improved, and corrected when necessary, resulting in more robust and reliable output.

[0049] Step 106, the prompt word information and the corresponding reflection correction information are stored in the long-term memory and short-term memory respectively to realize data update, and the meta-knowledge base is updated based on the updated long-term memory, and then the process returns to the step of determining the corresponding meta-knowledge based on the meta-knowledge base and the search question to perform the next round of reasoning. Among them, the update of long-term memory can optimize the memory and the corrected action trajectory, and continuously improve the quality and efficiency of the reasoning process. This setting ensures that the accumulated experience of each reasoning can be effectively used in future reasoning, and continuously improves the adaptive ability of the question-answering framework.

[0050] In a specific application, long-term memory and short-term memory are combined to form a multi-storage memory module, thus obtaining a comprehensive information processing and retention framework, creating a powerful synergistic effect through the interaction between long-term memory and short-term memory. Among them, short-term memory facilitates the immediate processing and manipulation of information, which is crucial for real-time tasks and problem solving; long-term memory provides a rich library of accumulated knowledge and strategies, enhancing the ability to generalize and apply learned concepts to new situations. The complementary relationship between the two types of memory makes information processing more efficient, decision-making more accurate, and more adaptive and resilient when dealing with complex cognitive tasks (such as knowledge graph question answering).

[0051] Specifically, the short-term memory is used to temporarily store information related to the retrieval question; wherein the information related to the retrieval question includes the retrieval strategy of the retrieval question, the current execution stage of the retrieval strategy, and the reflective correction information.

[0052] Short-term memory can also be understood as working memory, which is used to temporarily store information and actively participate in the operation and processing of information in the process. Short-term memory is the "workspace" of the brain for cognitive tasks (such as problem solving, reasoning, and understanding). It emphasizes that information is in an active state during processing and is used in complex cognitive activities. In the context of knowledge graph question-answering tasks, entities, relationships, and reasoning paths are usually complex, so real-time monitoring of task execution and feedback from environmental variables are required. This process of adjusting action trajectories in real time improves the accuracy of task execution.

[0053] In large language models, prompts are usually used as a kind of most commonly used working memory. In order to optimize the action trajectory in the execution of knowledge graph question answering, some language models will repeatedly access the knowledge graph to extract task-related information. Before each access, the working memory will store the plan of the entire action trajectory. As the task progresses, successful steps will continue to be executed according to the action trajectory in the working memory, while failures prompt the model to simulate the human reflection process: identifying possible causes of failure, reflecting on them, and developing alternative strategies. Through this iterative process of planning, error analysis, and trajectory adjustment, the working memory gradually optimizes the action trajectory of KGQA, thereby improving the accuracy of the response.

[0054] The long-term memory is used to: store procedural memory, semantic memory and episodic memory; analyze the episodic memory to generate a general retrieval strategy for knowledge graph retrieval; wherein the procedural memory includes operational knowledge of the knowledge graph; the semantic memory includes conceptual understanding of entities, attributes and relationships in the knowledge graph; the episodic memory includes any historical retrieval question and the corresponding knowledge graph retrieval process (such as specific success or failure in a knowledge graph question-answering task).

[0055] Specifically, long-term memory, as a huge experience and strategy library, stores the experience gained from actual knowledge graph question and answer applications, saves new strategies for dealing with failures, potential causes of failure, changes in wording, and problems encountered in similar tasks and their solutions. This dynamic multi-level memory structure promotes a deeper understanding and strategy formulation for knowledge graph question and answer applications. Long-term memory also reflects the complexity of human memory, not only storing specific feedback and task trajectories, but also refining this information into reflective insights and general strategies for future applications. This process forms a hierarchical memory structure that provides an overview of specific feedback and action trajectories, and ultimately summarizes high-quality reflections and general strategies for knowledge graph applications.

[0056] Based on the above-mentioned long-term memory and short-term memory segments, this application also updates the memory update strategy in accordance with the continuous development of long-term memory and different knowledge graph question-and-answer scenarios. The memory update mechanism used in this application is similar to (vector database) computer storage: the information in long-term memory is similar to the data stored on the hard disk, which is stored for a long time and can be accessed when needed. The long-term memory is also used to: use the (Ebbinghaus forgetting curve) exponential decay model to describe the memory forgetting process, so as to forget and delete the stored memory.

[0057] During the memory update process of this application, each time a new knowledge graph question-answering task is encountered, the experience and strategies used to deal with the task must be recorded. This process is similar to a continuous review cycle, which helps to consolidate the acquired knowledge. Through this method, this application has not only accumulated rich practical experience, but also developed a set of effective strategies to deal with various knowledge graph question-answering scenarios. This continuous learning and review mechanism has enabled the framework of this application to continue to evolve, gradually enhancing its capabilities and efficiency in handling KGQA tasks. Each task execution is not only a process of solving a problem, but also an opportunity to deepen understanding and accumulate knowledge. This makes this application more and more handy in dealing with more complex KGQA challenges in the future.

[0058] During the loop, the continuous updating of the meta-knowledge base plays a key role in guiding the interaction of the large language model with the knowledge graph. It enhances the ability of the large language model to generalize from past interactions, synthesize complex reasoning strategies based on past successes and failures, and refine the question-answering method. Meta-knowledge not only captures the correct answer, but also summarizes the lessons learned from mistakes, enabling the large language model to avoid past mistakes and improve accuracy and efficiency in future queries.

[0059] Step 107, if the reward score meets the preset conditions, the response result is marked as the question-and-answer result. In an application example, if the reward score fails to meet the preset conditions, the loop is terminated according to the maximum number of trials. That is, if the reward score meets the preset conditions, or the maximum number of trials is reached, the iteration is stopped, and the current response result is marked as the final question-and-answer result, indicating that the self-correction is passed and the reasoning process can be ended. In each cycle, the reasoning process is continuously improved based on the current memory and the new correction until the final answer is obtained. This process ensures that each reasoning result is as accurate as possible by gradually optimizing the reasoning strategy, and finally provides the user with an accurate answer.

[0060] The above steps 101 to 107 complete the answer to a search question, and after completing the answers to multiple search questions, statistics and analysis of multiple search questions can be performed based on long-term memory. Compared with the prior art, the present application has the following advantages: (1) Improving reasoning accuracy: The present application greatly improves the accuracy of the system in multiple rounds of reasoning through the combination of self-reflection, long-term memory and short-term memory. (2) Enhancing task adaptability: The present application can adapt to various complex and dynamic KGQA tasks, and optimizes query efficiency through modular processing of knowledge graphs. (3) Self-optimization capability: The self-reflection mechanism in the present application can continuously optimize the reasoning process based on past errors and improve the long-term performance of the system. That is, the self-reflection mechanism enables the system to learn from errors, and the memory model ensures the continuous accumulation and call of effective information, thereby ensuring the efficiency and reliability of the framework.

[0061] Based on the same inventive concept, the embodiment of the present application also provides a question-answering system that combines a large language model with a knowledge graph. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more system embodiments provided below can refer to the limitations on the method above, and will not be repeated here.

[0062] The question-answering system of the present application that combines a large language model with a knowledge graph includes a knowledge retrieval module, a self-reflection mechanism module and a multi-storage memory module; the multi-storage memory module includes short-term memory and long-term memory. The framework that combines a large language model with a knowledge graph, which is composed of the above-mentioned multiple modules, has self-reflection and knowledge retrieval functions, aiming to improve the accuracy and adaptability of the knowledge graph question-answering system in complex queries. The system can be applied to smart terminals, servers, and cloud computing platforms, etc., to handle multi-round reasoning tasks, graph data retrieval, and knowledge-based reasoning and decision-making. For the relevant contents of the knowledge retrieval module, the short-term memory, the self-reflection mechanism module, and the long-term memory, please refer to the records in the above method. In an application example, the self-reflection mechanism module includes a meta-executor, a meta-evaluator, and a meta-insight processor. For specific functions, please refer to the records in the above method.

[0063] In another application example, let's take the search question of how many temperaments are there between Harriers and Chinese Crested Dogs:

[0064] (1) Knowledge retrieval is performed in the prior knowledge base of the meta-knowledge base based on the above retrieval questions to obtain meta-knowledge containing relevant common sense knowledge and reasoning path information, for example: Knowledge description: This knowledge graph includes some common sense questions, [... reasoning and deep understanding...]; Query action: The action taken is get_relations(Harrier), which is used to query the attribute relations of the Harrier, including knowledge related to its temperament; Problem entities: Harrier, Chinese Crested.

[0065] Then, based on the above meta-knowledge, the prompt word information is generated to start the knowledge query process. In this process, the binary signal from the database is received to start the query and perform preliminary reasoning.

[0066] (2) First reasoning:

[0067] Update of short-term memory: Based on the prompt word information obtained from the query of short-term memory, the temperament attributes related to the Harrier are preliminarily obtained. On the basis that the bonus score of the above result does not meet the preset conditions (such as not obtaining complete temperament information), reflective correction information is generated and the short-term memory is updated to identify in subsequent reasoning that the "get_neighbors" function was not used correctly when executing the query and correct the query method; at the same time, the reflective correction information is stored in the long-term memory to avoid similar errors in the future.

[0068] (3) Second reasoning: Memory search of the search problem in the meta-knowledge base’s empirical knowledge base to find ways to avoid errors, such as using “get_neighbors” correctly to obtain the temperament information of Harrier dogs. Previous errors and corrections will be saved in short-term memory, and prompt word information will be generated in combination with short-term memory to execute the “get_neighbors(Harrier,base.petbreeds.dog_breed.temperament)” query to ensure that the query syntax is correct. If the bonus score of the above result also does not meet the preset conditions, continue to generate reflection correction information, correct the query syntax, and update short-term memory, long-term memory, and empirical knowledge base.

[0069] (4) Other word reasoning and intersection operations: When it is necessary to find the temperaments shared by Harriers and Chinese Crested Dogs, the intersection() function is used to identify the common temperaments and store the intersection results in the empirical knowledge base to provide support for similar problems in the future.

[0070] (5) Final calculation and answer output: Based on the empirical knowledge in long-term memory, combined with the information in short-term memory, the prompt word "Execute count (#3)" is generated to calculate the number of shared temperaments; in order to find the number of shared temperaments, the count function will be used, and the result (the number of shared temperaments) will be obtained. On the basis that the bonus score of the above result meets the preset conditions, the final answer is output: variable #3 (representing the number of shared temperaments). Correspondingly, the long-term memory is updated to include the application of the count() function in the intersection calculation and provide the correct query steps for future questions.

[0071] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0072] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A question-answering method combining a large language model and a knowledge graph, characterized in that: include: Based on the meta-knowledge base, corresponding meta-knowledge is determined according to the search question; the meta-knowledge includes knowledge description, query action and question entity; the meta-knowledge base includes knowledge graph; combining the meta-knowledge with short-term memory to obtain cue word information; Based on the prompt word information, a large language model is used to execute the query action, and the current reasoning is completed to obtain a corresponding response result; Evaluating the response results to obtain reward points; If the reward score does not meet the preset conditions, generating reflection correction information according to the response result; The prompt word information and the corresponding reflection correction information are stored in the long-term memory and the short-term memory respectively to realize data update, and the meta-knowledge base is updated based on the updated long-term memory, and then the process of determining the corresponding meta-knowledge according to the retrieval question based on the meta-knowledge base is returned to perform the next round of reasoning; If the reward score meets the preset conditions, the response result is marked as the question and answer result.

2. The question-answering method combining a large language model and a knowledge graph according to claim 1, characterized in that: The meta-knowledge base includes an experience database and a priori database; The experience database contains procedural memory, semantic memory and episodic memory; the priori database contains procedural memory and semantic memory; wherein the procedural memory includes operational knowledge of the knowledge graph; the semantic memory includes conceptual understanding of entities, attributes and relationships in the knowledge graph; the episodic memory includes any historical retrieval question and the corresponding knowledge graph retrieval process.

3. The question-answering method combining a large language model and a knowledge graph according to claim 1, characterized in that: The step of evaluating the response result to obtain a reward score comprises: The response result is evaluated from three dimensions: whether the response result is hallucination information, whether the response result is consistent with the knowledge graph, and whether the response result is consistent with the query, so as to obtain a reward score; Among them, the evaluation process of whether the response result is hallucination information is: cross-validation with the meta-knowledge base to identify whether the response result contains non-factual or false content; the evaluation process of whether the response result is consistent with the knowledge graph is: checking whether the response result is consistent with the known facts in the knowledge graph; the evaluation process of whether the response result is consistent with the query is: evaluating the relevance of the response result to the retrieval question.

4. The question-answering method combining a large language model and a knowledge graph according to claim 1, characterized in that: The step of generating reflection correction information according to the response result includes: Based on the active supplement model, identifying knowledge missing information according to the response results; Performing logical consistency focusing on the response result to obtain logical correction information; wherein, in the process of logical consistency focusing, performing a logical check on the response result from three dimensions: whether the response result is hallucination information, whether the response result is consistent with the knowledge graph, and whether the response result is consistent with the query, and performing corresponding corrections; The process of identifying the knowledge missing information and the process of determining the logic correction information are logically evaluated and supplemented with knowledge to obtain reflective correction information.

5. The question-answering method combining a large language model and a knowledge graph according to claim 2, characterized in that: The short-term memory is used to temporarily store information related to the retrieval question; wherein the information related to the retrieval question includes the retrieval strategy of the retrieval question, the current execution stage of the retrieval strategy and the reflection correction information; The long-term memory is used to: store the procedural memory, the semantic memory and the episodic memory; analyze the episodic memory and generate a general retrieval strategy for knowledge graph retrieval.

6. The question-answering method combining a large language model and a knowledge graph according to claim 5, characterized in that: The long-term memory is also used to: use an exponential decay model to describe the memory forgetting process, so as to forget and delete the stored memory.

7. The question-answering method combining a large language model and a knowledge graph according to claim 1, characterized in that: Based on the meta-knowledge base, the steps of determining the corresponding meta-knowledge according to the search question include: The following formula is used to match the relevant documents of the search question: D q =KR(q,KB); Where Dq is the set of relevant documents retrieved from the meta-knowledge base according to the retrieval question vector q, KB is the meta-knowledge base, and KR is the knowledge retrieval module; The cosine similarity is calculated using the following formula: Among them, cosine_similarity(q,d) is the cosine similarity between the retrieval question vector q and the relevant document d; The ranking of related documents is calculated using the following ranking function: Among them, BM25(q,d) is the ranking of the retrieval question vector q and the related document d, n is the number of search terms in the retrieval question, IDF(qi) is the inverse document frequency, tf(qi,d) is the number of times the i-th search term vector qi appears in the related document d, k1 and b are adjustment parameters, and avgdl is the average document length.

8. A question-answering system combining a large language model and a knowledge graph, characterized in that: The question-answering system combining a large language model and a knowledge graph includes a knowledge retrieval module, a self-reflection mechanism module and a multi-storage memory module; the multi-storage memory module includes short-term memory and long-term memory; The knowledge retrieval module is used to: determine the corresponding meta-knowledge according to the search question based on the meta-knowledge base; the meta-knowledge includes knowledge description, query action and question entity; the meta-knowledge base includes knowledge graph; The short-term memory is used to: combine with the meta-knowledge to obtain prompt word information; The self-reflection mechanism module is used to: based on the prompt word information, use a large language model to execute the query action, complete the current reasoning, and obtain a corresponding response result; evaluate the response result to obtain a reward score; If the reward score does not meet the preset conditions, generating reflection correction information according to the response result; The prompt word information and the corresponding reflection correction information are stored in the long-term memory and the short-term memory respectively to realize data update; if the reward score meets the preset conditions, the response result is marked as the question and answer result; The long-term memory is used to update the meta-knowledge base.

Images