Robust retrieval enhancement method and device for large language model

Abstract

The invention relates to the technical field of big language model reasoning, and provides a robust retrieval enhancement method and device oriented to a big language model. According to the method, semantic analysis is carried out on user query in a real scene, a key entity is positioned, then a core entity is taken as an initial retrieval point, a credible iterative retrieval strategy is executed, associated information in a knowledge graph is obtained step by step, factual verification is carried out, the limitation of a traditional RAG framework on the key entity in the real scene is broken through, and the reliability of the key entity in the real scene is improved. Robust reasoning and knowledge utilization of the LLM under the conditions that reliable predefined associated entities are lacked and query semantics are complex are achieved, and robustness and fact consistency of the model in a complex knowledge graph question and answer task are improved.

Description

Technical Field

[0001] This application relates to the field of large language model reasoning technology, and in particular to a robust retrieval enhancement method and apparatus for large language models. Background Technology

[0002] With the increase in computing resources and model parameters, Large Language Models (LLMs) have been widely used in various vertical fields due to their powerful data awareness and semantic understanding capabilities. However, at present, the inference ability of LLMs mainly relies on the massive internal parameters to model and simulate the real world. This process incurs huge time overhead, affecting the application value of LLMs in vertical fields.

[0003] Retrieval-augmented generation (RAG) algorithms can effectively alleviate the application difficulties caused by the fixed parameters of LLM (Limited Least Model) and provide an efficient means to enhance model reasoning. Specifically, RAG algorithms typically utilize external high-quality knowledge bases (such as knowledge graphs) to analyze and obtain key entities relevant to the user's query as initial nodes for retrieval. They then perform path searches within the knowledge graph to retrieve knowledge links related to the current reasoning problem, thereby compensating for the deficiencies in the model's inherent knowledge and improving the generation accuracy of LLM.

[0004] Existing RAG algorithms typically assume that key entities in a knowledge graph are unique and easy to capture. However, real-world scenarios often encounter user queries with complex semantic relationships and non-standard writing, where key entities are difficult to locate accurately and may be multiple, posing a challenge to accurately retrieving key information from the knowledge graph. Furthermore, current methods generally assume that the information contained in the external knowledge base is completely correct and free of factual conflicts, an assumption that usually does not hold true in real-world scenarios. During the construction of external knowledge graphs, due to technological imperfections and the difficulty in verifying the authenticity of some information, a certain amount of inaccurate knowledge is usually introduced into the knowledge graph. This noise inevitably interferes with the LLM reasoning process, leading to unreliable reasoning results. Summary of the Invention

[0005] In view of this, embodiments of this application provide a robust retrieval enhancement method and apparatus for large language models, in order to solve the problem of insufficient inference performance of existing LLM retrieval enhancement algorithms when facing real-world scenarios.

[0006] A first aspect of this application provides a robust retrieval enhancement method for large language models, comprising:

[0007] A predictive agent based on a large language model (LLM) is used to analyze the problem and identify key candidate words.

[0008] Identify the candidate entities corresponding to the key candidate words in the preset vector index database; the vector index database is constructed based on an external knowledge graph, and each entity in the vector index database corresponds to a node in the external knowledge graph;

[0009] The retrieval agent iteratively retrieves each candidate entity and constructs a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification.

[0010] After each round of iterative retrieval, an LLM-based supervised agent is used to determine the reasoning result of the problem through the knowledge subgraph constructed in this round. If no reasoning result can be obtained, the next round of iterative retrieval continues until a reasoning result is obtained or the iteration termination condition is met.

[0011] A second aspect of this application provides a robust retrieval enhancement device for large language models, comprising:

[0012] The recognition module is configured to use a predictive agent based on a large language model (LLM) to parse the question and identify key candidate words.

[0013] The determination module is configured to determine the candidate entities corresponding to key candidate words in a preset vector index database; the vector index database is built based on an external knowledge graph, and each entity in the vector index database corresponds to a node in the external knowledge graph;

[0014] The iterative retrieval module is configured to use a retrieval agent to iteratively retrieve each candidate entity and construct a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification.

[0015] The reasoning module is configured to determine the reasoning result of the problem through the knowledge subgraph constructed in this round after each round of iterative retrieval using an LLM-based supervised agent, and continue to the next round of iterative retrieval if no reasoning result can be obtained, until a reasoning result is obtained or the iteration termination condition is met.

[0016] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0018] The beneficial effects of the embodiments in this application compared with the prior art are:

[0019] This application embodiment performs semantic analysis on user queries in real-world scenarios to locate key entities. Then, starting with the core entities as the retrieval points, it executes a reliable iterative retrieval strategy to gradually obtain related information in the knowledge graph and perform factual verification. This breaks through the limitations of the traditional RAG framework on key entities in real-world scenarios, and realizes robust reasoning and knowledge utilization of LLM under conditions of lacking reliable predefined related entities and complex query semantics. This improves the robustness and factual consistency of the model in complex knowledge graph question answering tasks. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a robust retrieval enhancement method for large language models provided in an embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating the method for constructing a knowledge subgraph in each iteration through relation filtering, tail entity filtering, and evidence chain verification, as provided in the embodiments of this application.

[0023] Figure 3 This is a schematic diagram of a partial structure of an external knowledge graph provided in an embodiment of this application.

[0024] Figure 4 This is a flowchart illustrating the method for determining the core entity among candidate entities based on the aggregated evaluation results of the adjacent relationship paths of each node, as provided in the embodiments of this application.

[0025] Figure 5 This is a system block diagram for implementing the robust retrieval enhancement method for large language models provided in the embodiments of this application.

[0026] Figure 6 This is a schematic diagram of a robust retrieval enhancement device for large language models provided in an embodiment of this application.

[0027] Figure 7 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] The following will describe in detail, with reference to the accompanying drawings, a robust retrieval enhancement method and apparatus for large language models according to embodiments of this application.

[0030] As mentioned above, existing RAG algorithms typically assume that key entities in the knowledge graph are unique and easy to capture, and that existing reasoning methods typically assume that the information contained in the external knowledge base is completely correct and there are no factual conflicts. These assumptions are inconsistent with real-world scenarios, which may lead to unreliable reasoning results when applying LLM for reasoning in real-world situations.

[0031] In view of this, this application provides a robust retrieval enhancement method for large language models. By performing semantic analysis on user queries in real-world scenarios, key entities are located. Then, starting with the core entities as the retrieval points, a reliable iterative retrieval strategy is executed to gradually obtain the associated information in the knowledge graph and perform factual verification. This method breaks through the limitations of traditional RAG frameworks on key entities in real-world scenarios, and realizes robust reasoning and knowledge utilization of LLM under conditions of lacking reliable predefined associated entities and complex query semantics. This improves the robustness and factual consistency of the model in complex knowledge graph question answering tasks.

[0032] Figure 1 This is a flowchart illustrating a robust retrieval enhancement method for large language models provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0033] In step S101, an LLM-based predictive agent is used to analyze the question and identify key candidate words.

[0034] In step S102, the candidate entities corresponding to the key candidate words in the preset vector index database are determined.

[0035] The vector index database is built on an external knowledge graph, and each entity in the vector index database corresponds to a node in the external knowledge graph.

[0036] In step S103, the retrieval agent iteratively retrieves each candidate entity and constructs a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification.

[0037] In step S104, after each round of iterative retrieval, the LLM-based supervised agent determines the reasoning result of the problem through the knowledge subgraph constructed in this round, and continues the next round of iterative retrieval if no reasoning result can be obtained, until a reasoning result is obtained or the iteration termination condition is met.

[0038] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0039] In some embodiments of this application, a question can be obtained and a predictive agent equipped with LLM can be used to parse the question and identify key candidate words.

[0040] In some examples, the question can be a natural language question input by the user. A predictive agent can first normalize the natural language question, including correcting spelling errors, before identifying key candidate words.

[0041] In some embodiments of this application, candidate entities corresponding to key candidate words in a preset vector index database can be determined, and then the key candidate words can be linked to nodes in an external knowledge graph.

[0042] The external knowledge graph can be used for retrieval enhancement. The knowledge graph includes multiple nodes, each representing an entity, which can be a head entity or a tail entity. An edge between two nodes represents a relationship.

[0043] Vector index databases can be built based on external knowledge graphs. For example, the names of all nodes can be collected from an external knowledge graph, and then these node names can be encoded using a pre-trained language model (such as the MiniLM model). Based on this, a vector index database can be built, such as the FAISS vector index database, thereby achieving efficient retrieval of entity names.

[0044] Candidate entities can be obtained by performing semantic similarity retrieval using the FAISS index constructed from each candidate keyword in the vector index database. Since each entity in the vector index database corresponds to a node in an external knowledge graph, links can be established between the candidate keywords and nodes in the external knowledge graph.

[0045] In some embodiments of this application, a retrieval agent can be used to iteratively retrieve each candidate entity, constructing a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification. Furthermore, after each round of iterative retrieval, an LLM-based supervisory agent is used to determine the reasoning result of the problem based on the knowledge subgraph constructed in this round. If no reasoning result can be obtained after this round of iterative retrieval, the next round of iterative retrieval continues until a reasoning result is obtained or the iteration termination condition is met.

[0046] The iteration termination condition could be, for example, that the number of iterations exceeds a preset threshold, or that there are no more active search agents.

[0047] In some implementations, each core entity can be configured with a corresponding retrieval agent to perform iterative retrieval to construct a knowledge subgraph, and in each iteration, the retrieval of each core entity is performed in parallel.

[0048] In some implementations, during the evidence chain verification step of each iteration, if a retrieval agent fails to find any valid path after performing evidence chain verification, the retrieval agent will immediately terminate the iteration and release computing resources. If all retrieval agents terminate the iteration, it can be determined that there are currently no active retrieval agents.

[0049] According to the technical solution provided in the embodiments of this application, by performing semantic analysis on user queries in real-world scenarios, key entities are located. Then, with the core entity as the starting point, a reliable iterative retrieval strategy is executed to gradually obtain the associated information in the knowledge graph and perform factual verification. This breaks through the limitations of the traditional RAG framework on key entities in real-world scenarios, and realizes robust reasoning and knowledge utilization of LLM under conditions of lack of reliable predefined associated entities and complex query semantics. This improves the robustness and factual consistency of the model in complex knowledge graph question answering tasks.

[0050] Figure 2 This is a flowchart illustrating the method for constructing a knowledge subgraph in each iteration using relation filtering, tail entity filtering, and evidence chain verification, as provided in the embodiments of this application. Figure 2 As shown, the method includes the following steps:

[0051] In step S201, for each candidate entity, all adjacent relationship paths of its corresponding node in the external knowledge graph are determined, and the core entity in the candidate entity is determined based on the aggregation evaluation results of the adjacent relationship paths of each node.

[0052] In step S202, for each core entity, b candidate relations and b candidate tail entities are determined by using an LLM-based retrieval agent through relation filtering and tail entity filtering, respectively.

[0053] b is a positive integer.

[0054] In step S203, each core entity is combined with its corresponding b candidate relations and b candidate tail entities to form candidate reasoning paths.

[0055] In step S204, evidence chain verification is performed on each candidate reasoning path, and the reasoning evidence for each core entity is determined based on the verification results.

[0056] In step S205, the reasoning evidence of all core entities is integrated to obtain the knowledge subgraph after this round of iterative retrieval.

[0057] In some embodiments of this application, in each iteration, for each core entity, all adjacent relationship paths of its corresponding node in the external knowledge graph can be determined, and the core entity among the candidate entities can be determined based on the aggregation evaluation results of the adjacent relationship paths of each node.

[0058] Furthermore, for each core entity, an LLM-based retrieval agent can be used to determine b candidate relations and b candidate tail entities through relation filtering and tail entity filtering, respectively.

[0059] Here, any core entity is denoted as the target core entity. The b candidate relations of the target core entity can be the b relations in the external knowledge graph that have the highest semantic similarity to the question. The b candidate tail entities of the target core entity can be the b adjacent tail entities in the external knowledge graph that have the highest semantic similarity to the question among all the adjacent tail nodes of the b candidate relations.

[0060] Then, each core entity can be combined with its corresponding b candidate relations and b candidate tail entities to form multiple candidate reasoning paths. Evidence chains are then verified for each candidate reasoning path, and the reasoning evidence for each core entity is determined based on the verification results. Finally, by integrating the reasoning evidence of all core entities, the knowledge subgraph after this round of iterative retrieval is obtained.

[0061] In some embodiments of this application, the adjacency path of each node includes the adjacency path formed by the node and all its adjacent nodes, and each adjacency path includes the node, its adjacent nodes, and the relationship between the node and its adjacent nodes.

[0062] Figure 3 This is a schematic diagram of a partial structure of an external knowledge graph provided in an embodiment of this application. For example... Figure 3 As shown, node A has adjacent nodes B and C, and node B has adjacent nodes D and E. Any two adjacent nodes are connected by a directed line segment, as shown in the diagram. Therefore, for node A, its adjacent paths are "B—>A" and "A—>C".

[0063] Figure 4 This is a flowchart illustrating the method for determining the core entity among candidate entities based on the aggregated evaluation results of the adjacent relationship paths of each node, as provided in an embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0064] In step S401, for each node corresponding to a candidate entity, the predictive agent is used to determine the semantic similarity between each adjacent relation path of the node and the question.

[0065] In step S402, the average of the k adjacent relationship paths with the highest semantic similarity scores of this node is calculated to obtain the comprehensive relevance score of this node.

[0066] In step S403, the entities corresponding to the m nodes with the highest comprehensive relevance scores are determined as core entities.

[0067] Where k and m are both positive integers.

[0068] In some embodiments of this application, for each candidate entity's corresponding node in the external knowledge graph, a predictive agent can be used to query all adjacent relationship paths of this node and calculate the semantic similarity between each adjacent relationship path of this node and the question. Then, the k adjacent relationship paths with the highest semantic similarity scores of this node are aggregated, for example, by averaging the k semantic similarity scores, to obtain the comprehensive relevance score of this node.

[0069] Finally, the predictive agent can select the entities corresponding to the m nodes with the highest comprehensive relevance scores as the core entities.

[0070] In some embodiments of this application, determining b candidate relationships of a target core entity using a retrieval agent may include: determining all direct connection relationships of the target core entity using a retrieval agent; using LLM to perform semantic similarity scoring on all direct connection relationships and questions of the target core entity; determining the b direct connection relationships with the highest scores as b candidate relationships of the target core entity; wherein, the target core entity is any core entity.

[0071] Meanwhile, using a retrieval agent to determine b tail entities of the target core entity may include: using the retrieval agent to determine all adjacent tail entities of the target core entity; using LLM to perform semantic similarity scoring on all adjacent tail entities and questions of the b candidate relations; and determining the b adjacent tail entities with the highest scores as b candidate tail entities of the target core entity.

[0072] In other words, the retrieval agent can obtain all directly connected relationships of the core entity, submit them to the LLM for semantic relevance scoring, and retain the top-b relationships with the highest scores. Furthermore, the retrieval agent can also obtain adjacent tail entities from the candidate relationships, similarly submitting them to the LLM for semantic similarity comparison between the tail entities and the question, and retaining the top-b tail entities based on the similarity scores.

[0073] In some embodiments of this application, for each core entity, the core entity, the candidate relationship corresponding to the core entity, and the tail entity corresponding to the core entity can be integrated into a triple, and each triple can be integrated to form multiple candidate reasoning paths.

[0074] Each candidate inference path consists of a string of triples. In each triple string, the head entity of the next triple is the same as the tail entity of the previous triple. An exemplary candidate inference path could be (e1, r1, e2), (e2, r2, e3), or (e3, r3, e4), where e1, e2, e3, and e4 are entities, and r1, r2, and r3 are relations.

[0075] If any triple in the triple string corresponding to the target core entity is taken as the target triple, then the head entity and tail entity of the target triple are the entities in the target core entity and its corresponding b candidate tail entities, and the relationship of the target triple is the relationship in the b candidate relationships corresponding to the target core entity.

[0076] For each candidate reasoning path, evidence chain verification can be performed separately. During evidence chain verification, enhanced information can be retrieved from the external knowledge graph based on all core entities. Then, combining the enhanced information with the inherent knowledge of the LLM, it can be determined whether there is erroneous information in each candidate reasoning path. If so, the triples containing the erroneous information are filtered out, and the remaining triples are saved as reasoning evidence in the global shared memory. Otherwise, all triples in this candidate reasoning path are saved as reasoning evidence in the global shared memory.

[0077] Taking the candidate inference paths (e1, r1, e2), (e2, r2, e3), and (e3, r3, e4) as an example, if the judgment result indicates that the triple (e2, r2, e3) is an incorrect fact, then the triple can be considered noise information, and the triple (e2, r2, e3) is deleted from the candidate inference path, while only (e1, r1, e2) and (e3, r3, e4) are saved to the global shared memory. Otherwise, if the judgment result indicates that there are no incorrect facts in the candidate inference path, i.e., no noise information, then all triples are saved to the global shared memory.

[0078] The global shared memory can be a pre-defined storage space used to store the reasoning evidence retrieved by the retrieval agent. Both the retrieval agent and the supervisory agent can access this global shared memory.

[0079] After each round of iterative retrieval, the supervising agent integrates all reasoning evidence from the globally shared memory to obtain a knowledge subgraph. Then, the supervising agent performs reasoning within this knowledge subgraph to determine if a reasoning result can be obtained. If so, the reasoning result is output and the process ends. Otherwise, if not, control is transferred to the retrieval agent to continue the next round of iterative retrieval until a reasoning result is obtained or the iteration termination condition is met.

[0080] In some embodiments of this application, if the reasoning result of the problem cannot be obtained after the iterative retrieval is terminated, the LLM can be used to generate the reasoning result of the problem using the Chain of Thought (CoT) method.

[0081] Figure 5 This is a system block diagram for implementing the robust retrieval enhancement method for large language models provided in the embodiments of this application. For example... Figure 5 As shown, users can input questions into the predictive agent via user queries. The predictive agent interacts with an external knowledge graph, obtaining a set of key entities by inputting candidate words into the external knowledge graph. To facilitate retrieval, a vector index database can also be built based on the external knowledge graph.

[0082] For each key entity, an iterative retrieval agent can be used to obtain an evidence chain consisting of candidate reasoning paths. The supervisory agent then integrates the verified evidence chains for each key entity to arrive at the final answer.

[0083] The technical solution provided in this application can overcome the limitations of traditional RAG frameworks on key entities in real-world scenarios, enabling robust reasoning and knowledge utilization of large models even in the absence of reliable predefined related entities and under conditions of complex query semantics. By constructing a multi-agent collaborative mechanism, key information in external knowledge bases is located, and the retrieved evidence chains are fact-checked, thereby improving the robustness and factual consistency of the model in complex knowledge graph question-answering tasks.

[0084] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0085] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0086] Figure 6This is a schematic diagram of a robust retrieval enhancement device for large language models provided in an embodiment of this application. Figure 6 As shown, the device includes:

[0087] The recognition module 601 is configured to use a predictive agent based on a large language model (LLM) to parse the question and identify key candidate words.

[0088] The determination module 602 is configured to determine the candidate entities corresponding to the key candidate words in the preset vector index database; the vector index database is constructed based on an external knowledge graph, and each entity in the vector index database corresponds to a node in the external knowledge graph.

[0089] The iterative retrieval module 603 is configured to use a retrieval agent to iteratively retrieve each candidate entity and construct a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification.

[0090] The reasoning module 604 is configured to, after each round of iterative retrieval, use an LLM-based supervised agent to determine the reasoning result of the problem through the knowledge subgraph constructed in this round, and continue to the next round of iterative retrieval if no reasoning result can be obtained, until a reasoning result is obtained or the iteration termination condition is met.

[0091] According to the technical solution provided in the embodiments of this application, by performing semantic analysis on user queries in real-world scenarios, key entities are located. Then, with the core entity as the starting point, a reliable iterative retrieval strategy is executed to gradually obtain the associated information in the knowledge graph and perform factual verification. This breaks through the limitations of the traditional RAG framework on key entities in real-world scenarios, and realizes robust reasoning and knowledge utilization of LLM under conditions of lack of reliable predefined associated entities and complex query semantics. This improves the robustness and factual consistency of the model in complex knowledge graph question answering tasks.

[0092] In some implementations, in each iteration, a knowledge subgraph is constructed through relation filtering, tail entity filtering, and evidence chain verification. This includes: for each candidate entity, determining all adjacent relation paths of its corresponding node in the external knowledge graph, and determining the core entity among the candidate entities based on the aggregation evaluation results of the adjacent relation paths of each node; for each core entity, using an LLM-based retrieval agent, determining b candidate relations and b candidate tail entities respectively through relation filtering and tail entity filtering; b is a positive integer; combining each core entity with its corresponding b candidate relations and b candidate tail entities to form candidate reasoning paths; performing evidence chain verification on each candidate reasoning path, and determining the reasoning evidence of each core entity based on the verification results; and integrating the reasoning evidence of all core entities to obtain the knowledge subgraph after this iteration.

[0093] In some implementations, the adjacency paths of each node include the adjacency paths formed by the node and all its adjacent nodes. Each adjacency path includes the node, its adjacent nodes, and the relationship between the node and its adjacent nodes. The core entities among the candidate entities are determined based on the aggregated evaluation results of the adjacency paths of each node. This includes: for each node corresponding to a candidate entity, using a predictive agent to determine the semantic similarity between each adjacency path of the node and the question; averaging the k adjacency paths with the highest semantic similarity scores for the node to obtain a comprehensive relevance score for the node; and determining the entities corresponding to the m nodes with the highest comprehensive relevance scores as core entities; where k and m are both positive integers.

[0094] In some implementations, the retrieval agent is used to determine b candidate relations of the target core entity, including: using the retrieval agent to determine all direct connection relations of the target core entity; using LLM to perform semantic similarity scoring on all direct connection relations and questions of the target core entity; and determining the b direct connection relations with the highest scores as b candidate relations of the target core entity; wherein, the target core entity is any core entity.

[0095] In some implementations, the retrieval agent is used to determine b tail entities of the target core entity, including: using the retrieval agent to determine all adjacent tail entities of b candidate relations; using LLM to perform semantic similarity scoring on all adjacent tail entities and questions of the b candidate relations; and determining the b adjacent tail entities with the highest scores as b candidate tail entities of the target core entity.

[0096] In some implementations, candidate reasoning paths include triple strings, where the head entity of the next triple in each triple string is the same as the tail entity of the previous triple; the head and tail entities of the target triple are entities from the target core entity and its corresponding b candidate tail entities, and the relation of the target triple is the relation from the b candidate relations corresponding to the target core entity; the target triple is any triple in the triple string corresponding to the target core entity; evidence chain verification is performed on each candidate reasoning path, including: obtaining enhanced information based on all core entities in the external knowledge graph; combining the enhanced information and the inherent knowledge of LLM to determine whether there is erroneous information in each candidate reasoning path; if so, filtering out the triple containing the erroneous information and saving the remaining triples as reasoning evidence to the global shared memory; otherwise, saving all triples in this candidate reasoning path as reasoning evidence to the global shared memory.

[0097] In some implementations, the method further includes: in response to determining that a reasoning result for the problem cannot be obtained after the iterative retrieval has terminated, using an LLM-based thought chain approach to generate a reasoning result for the problem.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0099] Figure 7 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 7 of this embodiment includes a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, it implements the steps in the various method embodiments described above. Alternatively, when the processor 701 executes the computer program 703, it implements the functions of each module / unit in the various device embodiments described above.

[0100] Electronic device 7 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 7 may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of electronic device 7 and does not constitute a limitation on electronic device 7. It may include more or fewer components than shown, or different components.

[0101] The processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0102] The memory 702 can be an internal storage unit of the electronic device 7, such as a hard disk or RAM of the electronic device 7. The memory 702 can also be an external storage device of the electronic device 7, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 7. The memory 702 can also include both internal and external storage units of the electronic device 7. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0105] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A robust retrieval enhancement method for large language models, characterized in that, include: A predictive agent based on a large language model (LLM) is used to analyze the problem and identify key candidate words. Determine the candidate entities corresponding to the key candidate words in the preset vector index database; The vector index database is built based on an external knowledge graph, and each entity in the vector index database corresponds to a node in the external knowledge graph. The retrieval agent iteratively retrieves each candidate entity and constructs a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification. After each round of iterative retrieval, an LLM-based supervised agent is used to determine the reasoning result of the problem through the knowledge subgraph constructed in this round. If no reasoning result can be obtained, the next round of iterative retrieval continues until a reasoning result is obtained or the iteration termination condition is met.

2. The method according to claim 1, characterized in that, In each iteration, a knowledge subgraph is constructed through relation filtering, tail entity filtering, and evidence chain verification, including: For each candidate entity, determine all adjacent relationship paths of its corresponding node in the external knowledge graph, and determine the core entity among the candidate entities based on the aggregation evaluation results of the adjacent relationship paths of each node. For each core entity, an LLM-based retrieval agent is used to determine b candidate relations and b candidate tail entities through relation filtering and tail entity filtering, respectively; b is a positive integer. Each core entity is combined with its corresponding b candidate relations and b candidate tail entities to form candidate reasoning paths; For each candidate reasoning path, the chain of evidence is verified, and the reasoning evidence for each core entity is determined based on the verification results. By integrating the reasoning evidence of all core entities, a knowledge subgraph is obtained after this round of iterative retrieval.

3. The method according to claim 2, characterized in that, Each node's adjacency path includes the adjacency path formed by this node and all its adjacent nodes. Each adjacency path includes this node, its adjacent nodes, and the relationship between this node and its adjacent nodes. Based on the aggregation evaluation results of the adjacent relationship paths of each node, the core entities among the candidate entities are determined, including: For each node corresponding to a candidate entity, the predictive agent is used to determine the semantic similarity between each adjacent relation path of the node and the question. The average of the k adjacent paths with the highest semantic similarity scores for this node is calculated to obtain the comprehensive relevance score for this node. The entities corresponding to the m nodes with the highest comprehensive relevance scores are identified as the core entities. Where k and m are both positive integers.

4. The method according to claim 2, characterized in that, The retrieval agent is used to determine b candidate relations for the target core entity, including: Use a retrieval agent to determine all direct connections of the target core entity; LLM is used to perform semantic similarity scoring on all direct connections of the target core entity and the question. The b highest-scoring direct connections are identified as the b candidate connections for the target core entity. The target core entity can be any core entity.

5. The method according to claim 2, characterized in that, Using a retrieval agent, determine b tail entities of the target core entity, including: The retrieval agent is used to determine all adjacent tail entities of the b candidate relations; LLM is used to perform semantic similarity scoring on all adjacent tail entities of the b candidate relations and the question; The b adjacent tail entities with the highest scores are identified as the b candidate tail entities of the target core entity.

6. The method according to claim 2, characterized in that, The candidate reasoning path includes a triplet string, in which the head entity of the next triplet in an adjacent triplet is the same as the tail entity of the previous triplet. The head entity and tail entity of the target triple are the entities in the target core entity and its corresponding b candidate tail entities, and the relation of the target triple is the relation in the b candidate relations corresponding to the target core entity. The target triplet is any triplet in the triplet string corresponding to the target core entity; For each candidate reasoning path, a chain of evidence verification is performed, including: Enhanced information is obtained by retrieving all core entities from the external knowledge graph. By combining the enhanced information and the inherent knowledge of the LLM, it is determined whether there is erroneous information in each candidate reasoning path; If so, filter out the triples containing the erroneous information and save the remaining triples as inference evidence to the global shared memory; Otherwise, save all triples in this candidate reasoning path as reasoning evidence to the global shared memory.

7. The method according to claim 1, characterized in that, The method further includes: In response to the determination that the reasoning result for the problem cannot be obtained after the iterative retrieval terminates, the reasoning result for the problem is generated using the thought chain method with LLM.

8. A robust retrieval enhancement device for large language models, characterized in that, include: The recognition module is configured to use a predictive agent based on a large language model (LLM) to parse the question and identify key candidate words. The determination module is configured to determine the candidate entities corresponding to the key candidate words in a preset vector index database; The vector index database is built based on an external knowledge graph, and each entity in the vector index database corresponds to a node in the external knowledge graph. The iterative retrieval module is configured to use a retrieval agent to iteratively retrieve each candidate entity and construct a knowledge subgraph through relation filtering, tail entity filtering, and evidence chain verification. The reasoning module is configured to, after each round of iterative retrieval, use an LLM-based supervised agent to determine the reasoning result of the problem through the knowledge subgraph constructed in this round, and continue to the next round of iterative retrieval if no reasoning result can be obtained, until a reasoning result is obtained or the iteration termination condition is met.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images