Retrieval enhancement generation method based on domain knowledge graph

CN119046480APending Publication Date: 2024-11-29KEDADUOCHUANG CLOUD NETWORK TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411248127.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

[0006]本发明所要解决的技术问题在于:如何最小化模型调用次数并缓解幻觉和领域知识稀缺的问题,进而优化低资源场景下复杂问题的知识图谱问答任务,提供了一种基于领域知识图谱的检索增强生成方法

Benefits of technology

[0019] Compared with the existing technology, the present invention has the following advantages: the retrieval enhancement generation method based on domain knowledge graph first obtains PLM through fine-tuning A Used to simultaneously extract topic entities and predict the relationship paths required to answer questions; then retrieve subgraphs related to the topic entities, and prune the subgraphs based on the generated relationship paths and semantic similarities between questions and triples; and finally fine-tune Get PLM B To complete the mapping from "question + contextual knowledge" to answers; thus, accurate reasoning on complex domain problems can be completed in a low-resource environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046480A_ABST
    Figure CN119046480A_ABST
Patent Text Reader

Abstract

The invention discloses a retrieval enhancement generation method based on a domain knowledge graph, which belongs to the technical field of knowledge graph and natural language processing, and comprises a subject entity identification and relation path prediction stage, a retrieval and pruning stage and an answer reasoning stage. The method comprises the following steps: firstly, performing fine adjustment to obtain a PLMA for simultaneously extracting a subject entity and predicting a relationship path required by an answer question; secondly, searching a subgraph related to the subject entity, and pruning the subgraph according to the generated relation path and the semantic similarity between the problem and the triad; and finally, performing fine tuning to obtain PLMB to complete mapping from'question + context knowledge 'to answers. And accurate reasoning of complex field problems can be completed in a low-resource environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge graphs and natural language processing, and in particular to a retrieval enhancement generation method based on a domain knowledge graph. Background Art

[0002] Retrieval-Augmented Generation (RAG), which is to enhance the generation results of Large Language Model (LLM) by retrieving external information, is an effective solution to the problem of LLMs hallucination. A common form of external information is documents. By segmenting and retrieving documents, LLMs can obtain contextual domain knowledge to answer relevant questions more accurately. However, document retrieval usually relies on entire documents or paragraphs. This coarse-grained retrieval leads to limited understanding of global semantics. In contrast, Knowledge Graph (KG), as a structured data source, can provide concise contextual knowledge and facilitate the model's analysis of complex relationships between entities, thereby giving it the reasoning ability to solve complex problems.

[0003] At present, the methods combining KG and LLM are mainly divided into two paradigms: KG-enhanced LLM and LLM-enhanced KG. The main idea of ​​the KG-enhanced LLM paradigm is to retrieve information from the knowledge graph and then integrate the retrieved information into the prompts provided to the LLM. However, such methods aim to fully utilize the reasoning ability of LLM and decompose the reasoning process into multiple sub-steps to enhance interpretability, resulting in multiple interactions with LLM to answer a question. Therefore, the time and computational costs required for such methods are often infeasible for practical applications. In contrast, the LLM-enhanced KG method aims to prompt the LLM to convert natural language questions into query statements, which are then executed against the knowledge base to obtain answers. However, this method relies heavily on the quality of the generated query statements, and when there are errors in the query statements themselves, it will fail to execute, thereby hindering the retrieval of accurate answers.

[0004] In addition, existing methods only focus on reasoning about multi-hop problems, but in reality, complex problems also include two types: comparative and parallel. Comparative problems usually involve multiple subject entities, and it is necessary to retrieve relevant subgraphs for each subject entity, and then use these subgraphs together as contextual knowledge for LLM reasoning; parallel problems contain multiple subquestions about the same subject entity, and it is necessary to retrieve all relevant subgraphs to infer all answers.

[0005] The above problems need to be solved urgently. To this end, the present invention proposes a retrieval enhancement generation method based on domain knowledge graph. Summary of the invention

[0006] The technical problem to be solved by the present invention is: how to minimize the number of model calls and alleviate the problems of hallucination and scarcity of domain knowledge, thereby optimizing the knowledge graph question answering task of complex problems in low-resource scenarios, and providing a retrieval enhancement generation method based on domain knowledge graph.

[0007] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0008] S1: In the stage of subject entity recognition and relationship path prediction, a joint fine-tuning task is designed to identify the subject entity in the question and predict the relationship path, and then fine-tune the PLM to obtain the PLM A ;

[0009] S2: In the retrieval and pruning stage, the n-hop subgraphs related to the subject entity are extracted, and then pruned according to the predicted relationship paths and the similarity between the triples and the question; the PLM is obtained by fine-tuning the PLM accordingly by taking the question and the pruned subgraph as contextual cues and the answer as the target. B ;

[0010] S3: In the answer reasoning stage, by B Provide questions and corresponding triples as hints to accurately reason about the answers.

[0011] Furthermore, in the step S1, in the joint fine-tuning task, the input of the PLM is the question, and the goal is to extract the subject entity and relationship path from the question, and then fine-tune the PLM to obtain the PLM A .

[0012] Furthermore, in step S1, when using the fine-tuned PLM A After the model extracts the topic entities and relationship paths, it uses the Sentence-Transformer model as an encoder to encode the entities and relationships in the knowledge graph, as well as the relationships in the extracted topic entities and relationship paths, and maps the extracted topic entities and relationship paths to the entities and relationships in the knowledge graph with the highest similarity through cosine similarity calculation.

[0013] Furthermore, in step S2, the specific processing process of pruning is as follows:

[0014] S21: Use the Sentence-Transformer model to encode the triples in the question and subgraph;

[0015] S22: Calculate the cosine similarity between each triple and the question;

[0016] S23: For the triples containing the relations that appear in the predicted relation path, add 0.2 to the cosine similarity score as the final score;

[0017] S24: Sort the triples according to their scores and select the top k triples with the highest scores as contextual knowledge.

[0018] Furthermore, in the steps S1 and S3, the PLM is the same model.

[0019] Compared with the prior art, the present invention has the following advantages: the retrieval enhancement generation method based on domain knowledge graph first obtains PLM by fine-tuning A It is used to simultaneously extract the subject entity and predict the relationship path required to answer the question; then retrieve the subgraph related to the subject entity and prune the subgraph according to the generated relationship path and the semantic similarity between the question and the triple; finally fine-tune to obtain the PLM B To complete the mapping from "question + contextual knowledge" to the answer; and then to complete accurate reasoning on complex domain problems in a low-resource environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the retrieval enhancement generation method based on the domain knowledge graph of the present invention. DETAILED DESCRIPTION

[0021] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0022] like Figure 1 As shown, this embodiment provides a technical solution: a retrieval enhancement generation method based on a domain knowledge graph, comprising three stages: subject entity recognition and relationship path prediction stage, retrieval and pruning stage, and answer reasoning stage.

[0023] First, in the topic entity recognition and relationship path prediction stage, a joint fine-tuning task is designed to identify the topic entities in the question and predict possible relationship paths to fine-tune the PLM A Secondly, in the retrieval and pruning stage, n-hop subgraphs related to the subject entity are extracted and then pruned based on the predicted relationship paths and the similarity between triples and questions; the PLM is optimized by taking the question and the pruned subgraph as contextual cues and the answer as the target. B Finally, in the answer reasoning stage, the fine-tuned PLM B Provide questions and corresponding triples as hints to accurately reason about the answers.

[0024] It should be noted that different fine-tuning is performed on the same PLM to obtain the PLM A 、PLM B , in order to distinguish and explain, A and B are used as subscripts respectively.

[0025] For an input question, in order to extract relevant knowledge from the knowledge graph as much as possible, it is necessary to first perform subject entity recognition. The relationship path can capture semantic entity relationships and assist in reasoning tasks. For questions like "How to purchase the 6-month version of Package A?", generating the relationship path "6-month version→handling method" helps to retrieve relevant knowledge from the knowledge graph and obtain the answer "handling method_5". Existing methods use a small number of samples or zero-sample prompts to guide LLM to extract subject entities, but because its training corpus lacks domain knowledge, it is difficult to fully identify domain nouns. In addition, the lack of domain knowledge graph in LLM also makes it difficult to complete reasoning path prediction. Therefore, the present invention designs a joint fine-tuning task for subject entity recognition and relationship path prediction. The input of the model (PLM) is the question, and the goal is "subject entity; relationship path". Using the fine-tuned PLM A After the model extracts the topic entities and relationship paths, it uses the Sentence-Transformer model as an encoder to encode the entities and relationships in the knowledge graph, as well as the relationships in the extracted topic entities and relationship paths, and maps them to the knowledge graph entities or relationships with the highest similarity through cosine similarity.

[0026] In the retrieval and pruning stage, given the subject entity e0, an n-hop subgraph centered on e0 can be retrieved, and the answer to the question is usually contained in this subgraph. However, directly taking the entire subgraph as input may be too lengthy, making it difficult for the model to understand. In addition, too much redundant information may introduce noise, thereby affecting the reasoning performance of the model. Therefore, after retrieving the n-hop subgraph, pruning is required to provide the model with streamlined knowledge. Existing methods directly use the generated relationship path for retrieval and take the end point of the reasoning path as the answer. However, this method cannot be generalized well to all types of questions, such as comparative or parallel questions. Therefore, the present invention proposes a strategy that considers both the relationship path and semantic similarity to formulate pruning rules. First, the Sentence-Transformer model is used to encode the triples in the question and the subgraph. Then, the cosine similarity between each triple and the question is calculated. For triples containing relations that appear in the predicted relationship path, the cosine similarity score is added with 0.2 as the final score. Finally, these triples are sorted according to their scores, and the top k triples with the highest scores are selected as contextual knowledge.

[0027] Since the domain KG contains a large number of domain-specific words, many of which are knowledge outside the PLM training corpus, it is difficult to infer the answer to the question directly based on the input text subgraph. In order to enable the model to understand this task and distill the knowledge in the KG into the model, the present invention performs fine-tuning training. The input of the fine-tuning task includes the question and the text subgraph, and the goal is to infer the answer to the question. The fine-tuned PLM B Can accurately infer answers to domain-related questions.

[0028] This paper compares the question-answering performance of the existing large model prompting method KGRAG, the graph neural network-based method G-Retriever, the collaborative reasoning-based method RoG, and the retrieval enhancement method based on the domain knowledge graph proposed in this invention on the operator domain knowledge graph question-answering dataset, and uses the indicators Hit@1 and F1_score commonly used in knowledge graph question-answering tasks. Hit@1 mainly measures whether the true answer ranks first in the prediction result, while F1_score is mainly used to evaluate whether all true answers are included in the prediction result.

[0029] The data used in this embodiment is from a public data set in the operator field, including a knowledge graph in the operator field and 5,000 question-answer pairs. The present invention is compared with the KGRAG, G-Retriever and RoG methods, and the results are shown in Table 1:

[0030] (1) KGRAG: It adopts the conventional method of prompting LLM, prompting LLM to extract subject entities, and prompting LLM to infer the answer to the question after retrieving related subgraphs.

[0031] (2) G-Retriever: It combines the advantages of graph neural networks (GNNs), large language models (LLMs), and retrieval-augmented generation (RAG), and is fine-tuned with soft prompts to enhance graph understanding capabilities.

[0032] (3) RoG: We adopt a novel graph reasoning approach that synergistically leverages LLM and KG to achieve reliable and explainable reasoning.

[0033] Table 1. Experimental results of knowledge graph question answering

[0034]

[0035] As shown in Table 1, the proposed method outperforms all the comparison methods on the operator domain knowledge graph question answering dataset, with Hit@1 accuracy and F1_score reaching 95.2 and 90.8 respectively. Compared with the best G-Retriever method among the comparison methods, both evaluation indicators have been significantly improved by more than 100%. Therefore, the proposed method is very effective in improving the performance of domain knowledge graph question answering tasks.

[0036] In summary, the retrieval enhancement generation method based on the domain knowledge graph in the above embodiment first obtains the PLM by fine-tuning A It is used to simultaneously extract the subject entity and predict the relationship path required to answer the question; then retrieve the subgraph related to the subject entity and prune the subgraph according to the generated relationship path and the semantic similarity between the question and the triple; finally fine-tune to obtain the PLM B to complete the mapping from "question + contextual knowledge" to the answer; and then to complete accurate reasoning on complex domain problems in a low-resource environment. In the experiment of the operator domain knowledge graph question-answering dataset, both indicators are far better than other existing models, with Hit@1 and F1_score respectively increasing by 110.6% and 138.9% compared with the current most advanced methods.

[0037] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A retrieval enhancement generation method based on domain knowledge graph, characterized in that: The following steps are involved: S1: In the stage of subject entity recognition and relationship path prediction, a joint fine-tuning task is designed to identify the subject entity in the question and predict the relationship path, and then fine-tune the PLM to obtain the PLM A ; S2: In the retrieval and pruning stage, the n-hop subgraphs related to the subject entity are extracted, and then pruned according to the predicted relationship paths and the similarity between the triples and the question; the PLM is obtained by fine-tuning the PLM accordingly by taking the question and the pruned subgraph as contextual cues and the answer as the target. B ; S3: In the answer reasoning stage, by B Provide questions and corresponding triples as hints to accurately reason about the answers.

2. According to claim 1, a retrieval enhancement generation method based on a domain knowledge graph is characterized in that: In step S1, in the joint fine-tuning task, the input of PLM is the question, the goal is to extract the subject entity and relationship path from the question, and then fine-tune PLM to obtain PLM A .

3. A retrieval enhancement generation method based on domain knowledge graph according to claim 2, characterized in that: In step S1, when using the fine-tuned PLM A After the model extracts the topic entities and relationship paths, it uses the Sentence-Transformer model as an encoder to encode the entities and relationships in the knowledge graph, as well as the relationships in the extracted topic entities and relationship paths, and maps the extracted topic entities and relationship paths to the entities and relationships in the knowledge graph with the highest similarity through cosine similarity calculation.

4. A method for generating retrieval enhancement based on domain knowledge graph according to claim 3, characterized in that: In step S2, the specific processing process of pruning is as follows: S21: Use the Sentence-Transformer model to encode the triples in the question and subgraph; S22: Calculate the cosine similarity between each triple and the question; S23: For the triples containing the relations that appear in the predicted relation path, add 0.2 to the cosine similarity score as the final score; S24: Sort the triples according to their scores and select the top k triples with the highest scores as contextual knowledge.

5. A method for generating retrieval enhancement based on domain knowledge graph according to claim 4, characterized in that: In the steps S1 and S3, the PLM is the same model.

Citation Information

Patent Citations

  • Complex question knowledge base question answering method based on embedded and candidate subgraph pruning

    CN112766507A

  • Knowledge graph question and answer method based on sub-graph retrieval and query path ranking

    CN117786047A

  • Inference method and device for multi-hop question and answer of knowledge graph and storage medium

    CN118503366A