Graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning

Abstract

The invention discloses a graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning, and belongs to the technical field of knowledge graph and natural language processing, and the method comprises the following steps: S1, knowledge distillation and student model fine adjustment; s2, performing retrieval and pruning; s3, questioning and answering. According to the method, efficient and high-quality knowledge graph construction is realized through knowledge distillation, sub-graph pruning is optimized based on a reinforcement learning algorithm, and knowledge selection of a searcher and a generator is aligned to filter irrelevant knowledge as much as possible; unsupervised knowledge graph construction and accurate and efficient knowledge retrieval can be completed in a low-resource environment so as to enhance the question and answer effect of a large model.

Description

Technical Field

[0001] The present invention relates to the technical fields of knowledge graphs and natural language processing, and particularly to a graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a method to improve the generation quality of Large Language Models (LLMs) by introducing external information, and it is also an effective way to address the "hallucination" problem of LLMs. One of the common sources of external information is documents. By segmenting and retrieving documents, LLMs can obtain context knowledge in a specific domain, thereby answering relevant questions more accurately. However, traditional document retrieval usually takes the whole document or paragraph as a unit, and this coarse-grained processing method may limit the depth of the model's understanding of global semantics. In contrast, Knowledge Graphs (KGs), as a structured data form, can provide refined context information and support the model to analyze complex association relationships between entities, thereby endowing it with stronger complex problem reasoning capabilities.

[0003] Currently, the existing research on the GraphRAG model based on graph structure further improves the retrieval efficiency and knowledge representation ability by storing knowledge in the form of a knowledge graph. However, there are still many deficiencies in the existing graph retrieval enhanced generation methods. On the one hand, in terms of knowledge graph construction, many methods rely on existing knowledge graphs and lack customized processing of specific domain document sets; even if some methods try to construct knowledge graphs from documents, their focus is often on optimizing the retrieval module, and the knowledge graph construction process is too simplified to accurately extract complete knowledge. In addition, existing unsupervised knowledge graph construction methods usually need to call the LLM multiple times, resulting in low efficiency and being difficult to apply to large-scale data sets. On the other hand, in the retrieval part, although existing methods have optimized the retriever, they have ignored the problem of aligning the knowledge preferences between the retriever and the generator, which limits the effect of the generator when using the retrieved knowledge and cannot fully exert the value of the retrieved knowledge.

[0004] To optimize the knowledge extraction and retrieval capabilities of the graph retrieval enhanced generation method, the present invention proposes a graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning. Summary of the Invention

[0005] The technical problem to be solved by the present invention is as follows: how to solve the problems of poor knowledge graph construction quality and misalignment between the retriever optimization and the knowledge preferences of the generator in the existing graph retrieval enhanced generation methods, and then optimize the knowledge extraction and retrieval capabilities of the graph retrieval enhanced generation method. A graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning is provided. This method efficiently transfers the deep semantics and complex reasoning capabilities contained in the teacher model to the lightweight student model through knowledge distillation, achieving the construction of a knowledge graph that balances efficiency and quality; and models the retrieval process as a reinforcement learning task based on Direct Preference Optimization (DPO), and automatically learns the optimal subgraph pruning strategy through answer quality feedback, enabling the retriever to implicitly learn the preference characteristics of the generator for the knowledge structure.

[0006] As Figure 2 shown, the present invention solves the above technical problems through the following technical solutions. The present invention includes the following steps:

[0007] S1: Knowledge distillation and fine-tuning of the student model

[0008] Use the teacher model to perform knowledge extraction on the input text, construct a knowledge graph to collect training data, and then use knowledge distillation to screen the results extracted by the teacher model as the training data for the student model to fine-tune the student model;

[0009] S2: Retrieval and pruning

[0010] Extract the topic entity from the query q, and extract the multi-hop subgraph centered on the topic entity. Then use the retriever optimized by reinforcement learning to perform pruning, enabling the student model to screen out a set of knowledge triples with higher relevance to the query q

[0011] S3: Question answering

[0012] Concatenate the query q and the set of knowledge triples and input them into the generator, and infer the answers to the multi-hop questions through the generator.

[0013] Furthermore, in the step S1, both the teacher model and the baseline model of the student model are LLM models.

[0014] Furthermore, in the step S1, the specific process of the teacher model performing knowledge extraction on the input text is as follows:

[0015] S11: Use the few-shot prompting method to prompt the teacher model to extract knowledge triples from the input text;

[0016] S12: Use the few-shot prompting method to prompt the teacher model to define the relationships in each knowledge triple extracted in step S11;

[0017] S13: Standardize the entities and relationships in the extracted knowledge triples. Using the entities and relationships obtained after the processing in step S12 as a reference, let the teacher model extract again to obtain preliminary training data.

[0018] Furthermore, in step S12, when defining relationships, if there is a target pattern, try to match the existing relationship types. If there is no predefined pattern, create a new relationship type to form a consistent pattern structure, where the pattern refers to the relationship type.

[0019] Furthermore, in step S1, the rules for screening the results extracted by the teacher model include the integrity rule and the accuracy rule for extracting knowledge triples; among them, the integrity rule includes: the generation results of each step are output in a specified format; each knowledge triple in the final extraction result contains a complete head entity, relationship, and tail entity; the accuracy rule includes: each element in the knowledge triple of the final extraction result does not have another element with a semantic similarity exceeding the set threshold; the elements in the knowledge triple of the final extraction result exist in the input text or have words with a semantic similarity exceeding the set threshold, and the semantic similarity is represented by cosine similarity.

[0020] Furthermore, in step S2, the specific processing process is as follows:

[0021] S21: Use the few-shot prompting method to identify the topic entity in the query q;

[0022] S22: Map the topic entity to the knowledge graph through semantic similarity, and retrieve the n-hop subgraph g centered on the topic entity through breadth-first search, where n ≥ 2;

[0023] S23: Use answer preference optimization rewards to enhance the retriever's preference for selecting strongly relevant triples, thereby optimizing the pruning results of the retriever.

[0024] Furthermore, in step S23, the specific processing process is as follows:

[0025] S231: Use the LLM to sample different outputs

[0026]

[0027] Among them, Prompt prune is a prompt word designed for subgraph pruning, and this LLM is used as a retriever;

[0028] S232: Concatenate different outputs with the query q respectively and input them into the generator to obtain different answers, calculate the evaluation metric F1 score, use the score of the metric as the reward, and select the one with the highest reward and the one with the lowest reward as the positive example and the negative example respectively;

[0029] S233: Maximize the probability to enable the retriever to filter out a set of knowledge triples more relevant to the query q

[0030] Furthermore, in the step S3, the answer inference formula is as follows:

[0031]

[0032] where response is the inferred answer, Generator is the generator, and Prompt QA is a prompt template designed for the graph retrieval enhanced generation task.

[0033] Furthermore, in the step S3, when the output of the generator is "unknown", trace back to the source text of each triple in the set of knowledge triples through the source text alignment mechanism, and use these texts as context knowledge to assist the generator in answering questions.

[0034] The present invention has the following advantages compared with the prior art: This graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning realizes efficient and high-quality knowledge graph construction through knowledge distillation, optimizes subgraph pruning based on the reinforcement learning algorithm, aligns the knowledge selection of the retriever and the generator to filter out irrelevant knowledge as much as possible; it can complete unsupervised knowledge graph construction and accurate and efficient knowledge retrieval in a low-resource environment to enhance the question answering effect of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a framework diagram of the graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning in an embodiment of the present invention;

[0036] Figure 2 is an overall process schematic diagram of the graph retrieval enhanced large model generation method based on knowledge distillation and reinforcement learning of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The embodiments of the present invention will be described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0038] Example 1

[0039] In this example, first, the general process of the graph retrieval enhancement generation method will be described:

[0040] The dataset contains a document set Among them, each document consists of multiple text blocks: Performing knowledge triple extraction on the text blocks in all documents can obtain a triple set: where head j , rel j and tail j respectively represent the head entity, relation, and tail entity of the j-th triple.

[0041] For a given input query q, first use a retriever to retrieve a triple set related to q in the knowledge graph: Then concatenate the retrieved knowledge with q and input it into the generator. The generator performs reasoning based on the provided knowledge to generate the final response:

[0042] Such as Figure 1 shown, this example provides a technical solution: a graph retrieval enhancement large model generation method based on knowledge distillation and reinforcement learning. To balance the efficiency and quality of knowledge graph construction, the knowledge distillation method is introduced. A large-scale teacher model that has undergone multi-step knowledge extraction and fusion is used to generate a high-quality knowledge graph as training data, and then a small-scale student model is fine-tuned to achieve efficient and end-to-end knowledge graph extraction; at the same time, to align the knowledge preferences of the generator and the retriever, the DPO method is adopted. The reward is calculated based on the final generation result of the generator to optimize the preference training of the retriever, which specifically includes the following steps:

[0043] Step 1: To reduce the number of large model calls to simplify the process of knowledge graph construction, the present invention adopts the knowledge distillation method, and uses the high-quality data generated by a larger LLM model (teacher model) through the process of complete knowledge graph construction and optimization for end-to-end fine-tuning of the student model;

[0044] Step 2: To accurately retrieve the knowledge triples related to the query q, the present invention extracts the topic entities from the query q, extracts the multi-hop subgraphs centered on these topic entities, and then uses the retriever optimized based on reinforcement learning for pruning.

[0045] Step 3: In the question - answering stage, concatenate the query q and the knowledge triple set and input them into the generator. The generator can infer the answer to the multi - hop question based on the provided knowledge triple set.

[0046] The specific process of Step 1 is described in more detail below:

[0047] Use the teacher model to construct the knowledge graph to collect training data. Autonomously identify and extract structured information without pre - defining any schema. Mainly adopt the few - shot prompt method to prompt the LLM (teacher model) to extract knowledge triples from the input text. To enable the large - model to further understand the relationships between entities, also use the few - shot prompt method to prompt the LLM to write a definition for the relationship in each knowledge triple extracted in the previous step. If there is a target schema (schema refers to the relationship type), try to match the existing relationship types. If there is no pre - defined schema, dynamically create new relationship types to form a consistent schema structure. Finally, further standardize the entities and relationships in the extracted knowledge triples. Use the previously extracted entities and relationships as a reference, and let the model extract the input text again, thereby reducing redundancy and improving consistency.

[0048] It should be noted that the few - shot prompt method is to give the model some examples to make the model better understand the corresponding task.

[0049] The above - mentioned knowledge graph construction method does not require pre - defined schemas, is flexible in extraction, and the graph is complete. However, there are problems such as slow extraction speed due to multiple calls to the LLM, and errors in intermediate steps may affect subsequent results. Therefore, the present invention proposes to use the knowledge distillation method to simplify the multi - step call of the teacher model into an end - to - end extraction of the student model. The present invention obtains a large amount of supervised data through the extraction results obtained by the teacher model using the above - mentioned method. However, due to the hallucination problem of the LLM, the labels of these data still have problems. Therefore, to ensure the quality of the data set, the present invention proposes the following filtering rules to screen the data from two aspects: the integrity and accuracy of the extracted knowledge triples:

[0050] (1) Integrity rule 1: The generation result of each step is output in the specified format;

[0051] (2) Integrity rule 2: Each triple in the final extraction result contains a complete head entity, relationship, and tail entity;

[0052] (3) Accuracy rule 1: Each element in the final extraction result does not have another element with a high semantic similarity to it;

[0053] (4) Accuracy Rule 2: All elements in the triples of the extraction results exist in the input text or have words with a relatively high semantic similarity.

[0054] It should be noted that in this embodiment, a relatively high semantic similarity means that the cosine similarity is greater than 0.85.

[0055] The high-quality training data after filtering is used to fine-tune the student model. The fine-tuned model can significantly improve the extraction efficiency. Although there may still be certain errors, compared with the original method, the method proposed by the present invention achieves a better balance between efficiency and quality, providing a more efficient and practical solution for knowledge graph construction.

[0056] The following provides a more detailed description of the specific process of Step 2:

[0057] In the retrieval and pruning stage, in order to retrieve more relevant information from the knowledge graph, a few-shot prompting method is used to identify the topic entities in the query q. Then, these topic entities are mapped to the knowledge graph through semantic similarity, and an n-hop subgraph g centered on these topic entities is retrieved through breadth-first search (BFS). The answers to multi-hop reasoning questions usually exist within this subgraph. However, directly using the entire subgraph as input may be too verbose, making it difficult for the model to understand. In addition, too much redundant information may introduce noise, thus hindering the performance of the model. Therefore, after retrieving the n-hop subgraph g, it is crucial to perform pruning to refine and optimize the information presented to the generator.

[0058] Since the LLM itself has a certain reasoning ability, it can be directly used as a retriever to prune the subgraph, but there may be a risk of misjudging weakly relevant triples. Therefore, in order to optimize the subgraph pruning ability of the LLM, the present invention uses answer preference optimization rewards to enhance the model's preference for strongly relevant triples (strongly relevant triples are knowledge triples that are more relevant to the query q). First, the LLM samples different outputs where Prompt prune is a prompt word designed for subgraph pruning. In order to measure whether these outputs can help the generator answer questions, the present invention concatenates different with q respectively and inputs them into the generator to obtain different answers, and calculates the evaluation metric (F1 score). The scores of these metrics are used as rewards. The present invention selects the one with the highest reward value as the positive example and the lowest as the negative example. The training objective is to maximize the probability to enable the LLM to screen out a set of knowledge triples that are more relevant to the query q. The present invention uses the DPO algorithm to optimize the parameters of the retriever.

[0059] The following provides a more detailed description of the specific process of Step 3:

[0060] After retrieval and pruning, the query q and the set of related knowledge triples are input into the generator for answer reasoning: Among them, Prompt QA is a prompt template designed for the graph retrieval enhancement generation task. Since the results of unsupervised knowledge graph construction may be incomplete, it is inevitable that a few knowledge triples are missing. To address the limitation that the model cannot answer questions caused by this, the present invention designs a Source Text Reconciliation (STR) mechanism. Specifically, when the output of the generator is "unknown", this mechanism will trace the source text of each triple and use these texts as context knowledge to assist the generator in answering questions.

[0061] Example 2

[0062] Comparison of the method for constructing existing models and the performance of the model proposed by the present invention

[0063] The present invention will compare the question-answering performance of the existing method based on chain-of-thought retrieval IRCoT, the method based on graph traversal agent KGP-T5, the method based on double-layer retrieval architecture LightRAG, the method based on heterogeneous graph index MiniRAG, and the graph retrieval enhancement large model generation method based on knowledge distillation and reinforcement learning proposed by the present invention on the multi-hop inference question-answering dataset. The metrics used are the commonly used exact match (EM) and F1 score in the inference question-answering task. Both of these metrics evaluate the accuracy of the generated response compared to the standard answer.

[0064] The data used in this example are the publicly available datasets HotPotQA and 2WikiMQA for multi-hop question-answering tasks. The present invention compares with the methods of IRCoT, KGP-T5, MiniRAG, and LightRAG, and the results are shown in Table 1:

[0065] (1) IRCoT: Adopts a chain-of-thought-based method to guide the model to use the chain of thought for retrieval to optimize the performance of multi-hop question-answering.

[0066] (2) KGP-T5: Constructs a knowledge graph with text paragraphs and document structures as nodes, and selectively accesses the nodes related to the query through a large model-based graph traversal agent.

[0067] (3) LightRAG: Adopts a double-layer retrieval architecture, enhances the understanding of the query by extracting low-order and high-order keywords from the query, and thus retrieves relevant information correspondingly in the knowledge graph.

[0068] (4) MiniRAG: Achieve retrieval-enhanced generation based on knowledge graphs through heterogeneous graph indexing and a lightweight heuristic retrieval mechanism.

[0069] Table 1 Experimental Results of Multi-hop Reasoning Question Answering

[0070]

[0071] As shown in Table 1, the method proposed in the present invention outperforms all comparative methods on two multi-hop question answering benchmark datasets. Compared with the previous state-of-the-art method MiniRAG, the EM metrics on the HotPotQA and 2WikiMQA datasets are improved by 21.1% and 17.5% respectively. Therefore, the method proposed in the present invention is very significant for improving the performance of the multi-hop reasoning question answering task.

[0072] In summary, the method for generating a graph retrieval-enhanced large model based on knowledge distillation and reinforcement learning in the above embodiments realizes efficient and high-quality knowledge graph construction through knowledge distillation, optimizes subgraph pruning based on the reinforcement learning algorithm, aligns the knowledge selection of the retriever and the generator to filter out irrelevant knowledge as much as possible; can complete unsupervised knowledge graph construction and accurate and efficient knowledge retrieval in a low-resource environment to enhance the question answering effect of the large model, and in the experiments on two public datasets for multi-hop reasoning question answering, both metrics are better than other existing models, and the EM metrics on the two datasets are improved by 21.1% and 17.5% respectively compared with the current state-of-the-art methods.

[0073] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for enhancing the generation of large models in graph retrieval based on knowledge distillation and reinforcement learning, characterized in that, It includes the following steps: S1: Knowledge distillation and fine-tuning of the student model Use the teacher model to extract knowledge from the input text, construct a knowledge graph to collect training data, and then use the method of knowledge distillation to screen the results extracted by the teacher model as the training data for the student model, and fine-tune the student model; S2: Retrieval and pruning Extract the topic entity from the query q, and extract the multi-hop subgraph centered on the topic entity. Then, use the retriever optimized by reinforcement learning for pruning, so that the student model can screen out a set of knowledge triples with higher relevance to the query q S3: Question answering The query q and the knowledge triple set are concatenated and input into the generator, and the generator infers the answers to the multi-hop questions.

2. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 1, wherein In the step S1, the baseline models of both the teacher model and the student model are LLM models.

3. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 1, wherein In the step S1, the specific process of the teacher model extracting knowledge from the input text is as follows: S11: Use the few-shot prompting method to prompt the teacher model to extract knowledge triples from the input text; S12: Use the few-shot prompting method to prompt the teacher model to define the relationships in each knowledge triple extracted in S11; S13: Standardize the entities and relationships in the extracted knowledge triples, use the entities and relationships obtained after the process of S12 as a reference, and let the teacher model extract again to obtain preliminary training data.

4. The method for generating a graph retrieval enhanced large model based on knowledge distillation and reinforcement learning according to claim 3, wherein In the step S12, when defining the relationship, if there is a target pattern, try to match the existing relationship types. If there is no predefined pattern, create a new relationship type to form a consistent pattern structure, where the pattern refers to the relationship type.

5. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 1, wherein In the step S1, the rules for screening the results extracted by the teacher model include the integrity rule and the accuracy rule for extracting knowledge triples; among them, the integrity rule includes: the generation results of each step are output in the specified format; each knowledge triple in the final extraction result contains a complete head entity, relationship, and tail entity; the accuracy rule includes: each element in the knowledge triple of the final extraction result does not have another element with a semantic similarity exceeding the set threshold; each element in the knowledge triple of the final extraction result exists in the input text or has a word with a semantic similarity exceeding the set threshold, and the semantic similarity is represented by cosine similarity.

6. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 3, wherein In the step S2, the specific processing process is as follows: S21: Use the few-shot prompting method to identify the topic entity in the query q; S22: Map the topic entity to the knowledge graph through semantic similarity, and retrieve the n-hop subgraph g centered on the topic entity through breadth-first search, where n≥2; S23: Use the answer preference optimization reward to enhance the retrieval preference of the retriever for strongly relevant triples, and then optimize the pruning result of the retriever.

7. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 6, wherein In the step S23, the specific processing process is as follows: S231: Sample different outputs using the LLM Among them, Prompt prune is a prompt word designed for subgraph pruning, and this LLM is used as a retriever; S232: Combine different outputs with the query q respectively, input them into the generator to obtain different answers, calculate the evaluation metric F1 score, use the score of the metric as the reward, and select the one with the highest reward and the one with the lowest reward as the positive example and the negative example respectively; S233: By maximizing the probability enable the retriever to filter out a set of knowledge triples that are more relevant to the query q 8. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 7, wherein In the step S3, the answer inference formula is as follows: Among them, response is the inferred answer, Generator is the generator, and Prompt QA is a prompt template designed for the graph retrieval enhanced generation task.

9. The method for enhancing the generation of a large model for graph retrieval based on knowledge distillation and reinforcement learning according to claim 8, characterized in that, When the output of the generator is "unknown" in the step S3, trace the source text of each triple in the knowledge triple set through the source text alignment mechanism, and use these texts as context knowledge to assist the generator in answering questions.

Images