RAG system factuality verification method based on knowledge graph
By building a knowledge graph in the RAG system and combining large language models for factual verification, the factual errors in the RAG system generated answers and the lack of professional knowledge application are solved, and higher answer accuracy and reliability are achieved. It is suitable for complex query scenarios and expands the application scope of RAG technology.
Patent Information
- Application Number
- CN202411224818.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-03
AI Technical Summary
The RAG system is prone to factual errors when generating answers, and lacks in-depth understanding and flexible application of professional domain knowledge, making it difficult to make full use of domain-specific knowledge structures and relationships in complex queries, resulting in poor answer quality.
Using a knowledge graph-based method, we use a knowledge graph in a custom field, combine it with a large language model for factual verification, detect and correct errors in the preliminary answers, and generate a final answer that has been checked and corrected.
It improves the accuracy and reliability of answers, is suitable for fields that require highly specialized and accurate information, broadens the scope of application of RAG technology, and promotes the advancement of artificial intelligence in the fields of information retrieval and natural language processing.
Smart Images

Figure CN119066208A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing technology, and in particular to a knowledge graph-based RAG system factual verification method. Background Art
[0002] In recent years, with the rapid development of artificial intelligence technology, large language models have made significant progress in the field of natural language processing. Among them, the retrieval-augmented generation (RAG) system, as an advanced technology that combines information retrieval and language generation, has been widely used in applications such as question-answering systems and intelligent customer service. The RAG system enhances the answering ability of large language models by retrieving relevant documents, and performs well in providing more accurate and relevant information.
[0003] However, despite the excellent performance of the RAG system in many scenarios, it still faces some challenges. One of the most prominent issues is factual errors. When generating answers, large language models sometimes produce content that is inconsistent with the facts. This may be caused by factors such as limitations of training data, reasoning bias of the model, or incomplete retrieved information. Such factual errors not only reduce the reliability of the system, but may also cause users to make wrong judgments or decisions, especially in professional fields such as medicine, law, and finance that require extremely high accuracy.
[0004] Existing solutions mainly focus on optimizing retrieval algorithms and expanding training data sets, but these methods often fail to completely eliminate factual errors. Some studies have tried to introduce external knowledge bases for verification, but most methods either affect the real-time performance of the system or are difficult to adapt to different professional fields.
[0005] In addition, traditional RAG systems usually lack the ability to deeply understand and flexibly apply professional domain knowledge. When dealing with complex queries in a specific domain, the system may not be able to fully utilize the domain-specific knowledge structure and relationships, resulting in unsatisfactory answer quality.
[0006] Based on the above problems, the present invention proposes a factual verification method for a RAG system based on a knowledge graph. Summary of the invention
[0007] In order to remedy the deficiencies of the prior art, the present invention provides a simple and efficient knowledge graph-based RAG system factual verification method.
[0008] The present invention is achieved through the following technical solutions:
[0009] A method for factual verification of a RAG system based on a knowledge graph, characterized in that it comprises the following steps:
[0010] Step S1, constructing a knowledge graph;
[0011] First, collect data from custom fields, extract entities and relationships using natural language processing technology after preprocessing, build a comprehensive knowledge graph, and store it in a high-performance graph database to provide a basis for subsequent factual verification;
[0012] Step S2: query processing and generating a preliminary answer;
[0013] When a user query is received, the vector retrieval technology is used to search for relevant information in the pre-prepared document library; the retrieved information is combined with the original query and input into the large language model to generate a preliminary answer;
[0014] Step S3, factual verification;
[0015] Extract key information from the preliminary answer, and verify the preliminary answer using a large language model in combination with the knowledge graph, and find potential errors or inconsistencies by comparing the statements in the answer with the facts in the knowledge graph;
[0016] Step S4, correction and secondary answer;
[0017] If a factual error is detected, a correction is generated based on the correct information in the knowledge graph. Subsequently, the original query is combined with the correction and fed back into the large language model to generate a final answer that has been verified and corrected.
[0018] If no factual errors are detected, the preliminary answer is final;
[0019] Step S5: Result output and continuous optimization;
[0020] The final answer is formatted and presented through the user interface; at the same time, user feedback and performance data are continuously collected, and the knowledge graph and optimization algorithm are regularly updated to ensure the long-term and efficient operation of the system.
[0021] In step S1, structured and unstructured data in a custom domain are collected, and natural language processing technology is used to implement entity recognition and relationship extraction, so as to construct a comprehensive knowledge graph including entities, attributes and relationships.
[0022] In step S2, when the user inputs a query, firstly, the vector retrieval technology is used to retrieve relevant information from the pre-prepared document library; then, the retrieved information is combined with the original query through a pre-built prompt template to form a rich context; the query integrating the retrieved information is input into the large language model, and the large language model generates a preliminary answer result.
[0023] In step S3, a prompt template is set for verification by a large language model. The prompt template receives a preliminary answer generated by the large language model, and combines it with the knowledge graph and sends it to the large language model for in-depth analysis of the answer. The large language model combines the knowledge graph to determine whether there are factual errors in the preliminary answer.
[0024] In step S4, when the factual verification module detects an error, the correction speech generation module is started; based on the correct information in the knowledge graph, the correction speech is intelligently generated through the prompt template set by the large language model.
[0025] In step S5, when the final answer is presented through the user interface, a process description of factual verification is provided.
[0026] A RAG system factual verification system based on knowledge graph, comprising:
[0027] The knowledge graph construction module is responsible for building and maintaining the knowledge graph for factual verification;
[0028] First, we collect structured and unstructured data in custom domains, and build a comprehensive knowledge graph containing entities, attributes, and relationships by using natural language processing technology, entity recognition, and relationship extraction.
[0029] RAG retrieval enhancement generation module is responsible for receiving and processing user query requests;
[0030] When a user enters a query, the RAG retrieval enhancement generation module first uses vector retrieval technology to retrieve relevant information from the pre-prepared document library; then, the retrieved information is combined with the original query through a pre-built prompt template to form a rich context, and the query that incorporates the retrieval information is input into the large language model, which generates a preliminary answer result;
[0031] The factual verification module is responsible for setting a prompt template for verification by the large language model. The prompt template receives the preliminary answer generated by the RAG search enhancement generation module and combines it with the knowledge graph to send it to the large language model for in-depth analysis of the answer. The large language model combines the knowledge graph to determine whether there are factual errors in the preliminary answer.
[0032] The correction speech generation module is responsible for intelligently generating correction speech based on the correct information in the knowledge graph and the prompt template set by the large language model when the factual verification module detects an error;
[0033] The secondary answer generation module is responsible for integrating the original query and the generated correction words, and inputting the original query and the generated correction words into the large language model again for factual verification until there are no factual errors in the answer; the result output module is responsible for formatting the answer and presenting the final answer. It will receive the secondary generated answer, or directly process the initial answer if no error is detected.
[0034] The result output module provides a process description of factual verification.
[0035] A knowledge graph-based RAG system factual verification device, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method when executing the computer program.
[0036] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method when executed by a processor.
[0037] The beneficial effects of the present invention are as follows: the factual verification method of the RAG system based on the knowledge graph combines the advantages of structured knowledge and large language models, improves the accuracy and reliability of answers, and can be flexibly adjusted according to different fields and application scenarios, broadens the application scope of RAG technology, and promotes the progress of artificial intelligence in the fields of information retrieval and natural language processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0039] Attached Figure 1 Schematic diagram of the factual verification method of the RAG system based on the knowledge graph of the present invention. DETAILED DESCRIPTION
[0040] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0041] The RAG system factual verification method based on knowledge graph includes the following steps:
[0042] Step S1, constructing a knowledge graph;
[0043] First, we collect domain-specific data from multiple reliable sources, extract entities and relationships using natural language processing technology after preprocessing, build a comprehensive knowledge graph, and store it in a high-performance graph database to provide a basis for subsequent factual verification.
[0044] Step S2: query processing and generating a preliminary answer;
[0045] When a user query is received, the vector retrieval technology is used to search for relevant information in the pre-prepared document library; the retrieved information is combined with the original query and input into the large language model to generate a preliminary answer;
[0046] Step S3, factual verification;
[0047] Extract key information from the preliminary answer, and verify the preliminary answer using a large language model in combination with the knowledge graph, and find potential errors or inconsistencies by comparing the statements in the answer with the facts in the knowledge graph;
[0048] Step S4, correction and secondary answer;
[0049] If a factual error is detected, a correction is generated based on the correct information in the knowledge graph. Subsequently, the original query is combined with the correction and fed back into the large language model to generate a final answer that has been verified and corrected.
[0050] If no factual errors are detected, the preliminary answer is final;
[0051] Step S5: Result output and continuous optimization;
[0052] The final answer is formatted and presented through the user interface; at the same time, user feedback and performance data are continuously collected, and the knowledge graph and optimization algorithm are regularly updated to ensure the long-term and efficient operation of the system.
[0053] In step S1, structured and unstructured data in a custom field are collected, and natural language processing technology is used to implement entity recognition and relationship extraction. These data are deeply analyzed and processed, and a comprehensive knowledge graph including entities, attributes and relationships is constructed based on the processed information.
[0054] In step S2, when the user inputs a query, firstly, the vector retrieval technology is used to retrieve relevant information from the pre-prepared document library; then, the retrieved information is combined with the original query through a pre-built prompt template to form a rich context; the query integrating the retrieved information is input into the large language model, and the large language model generates a preliminary answer result.
[0055] In step S3, a prompt template is set for verification by a large language model. The prompt template receives a preliminary answer generated by the large language model, and combines it with the knowledge graph and sends it to the large language model for in-depth analysis of the answer. The large language model combines the knowledge graph to determine whether there are factual errors in the preliminary answer.
[0056] In step S4, when the factual verification module detects an error, the correction speech generation module is started; based on the correct information in the knowledge graph, the correction speech is intelligently generated through the prompt template set by the large language model.
[0057] In step S5, when the final answer is presented through the user interface, a process description of factual verification is provided.
[0058] Through this process, the knowledge graph-based RAG system factual verification method can effectively improve the accuracy and reliability of the RAG system's answers, and is particularly suitable for fields that require highly specialized and precise information.
[0059] The knowledge graph-based RAG system factual verification system includes:
[0060] The knowledge graph construction module is responsible for building and maintaining the knowledge graph for factual verification;
[0061] First, collect structured and unstructured data in custom domains, which may be user-uploaded databases or other reliable sources. Then, deeply analyze and process these data by using natural language processing technology, entity recognition and relationship extraction. Based on the processed information, build a comprehensive knowledge graph containing entities, attributes and relationships;
[0062] RAG retrieval enhancement generation module is the core of the system and is responsible for receiving and processing user query requests;
[0063] When a user enters a query, the RAG retrieval enhancement generation module first uses vector retrieval technology to retrieve relevant information from a pre-prepared document library; then, the retrieved information is combined with the original query through a pre-built prompt template to form a rich context, and the query that incorporates the retrieval information is input into the large language model, which generates preliminary answer results; this method not only utilizes the accuracy of retrieval, but also gives full play to the generation ability of the large language model to provide users with information-rich and relevant answers.
[0064] The factual verification module is responsible for setting up a prompt template for verification through the large language model; the prompt template receives the preliminary answer generated by the RAG retrieval enhancement generation module, and combines it with the knowledge graph and sends it to the large language model for in-depth analysis of the answer; the large language model combines the knowledge graph to determine whether there are factual errors in the preliminary answer; this process can not only detect obvious errors, but also detect subtle inaccurate information, thereby greatly improving the reliability of the system's answers.
[0065] The correction speech generation module is responsible for intelligently generating correction speech based on the correct information in the knowledge graph and the prompt template set by the large language model when the factual verification module detects an error. This process is not just a simple replacement of the wrong information, but it takes into account the context and expression of the original answer to generate a smooth, natural and easy-to-understand correction speech. This design ensures that the correction information can be seamlessly integrated into the final answer, improving the user experience.
[0066] The secondary answer generation module is responsible for integrating the original query and the generated correction words, and inputting the original query and the generated correction words into the large language model again for factual verification until there are no factual errors in the answer; in this way, the system can generate a final answer that has been verified and corrected. This process not only corrects previous errors, but also maintains the coherence and completeness of the answer, ensuring that the final output information is both accurate and easy to understand.
[0067] The output module is responsible for formatting the answers to ensure readability and clarity. It presents the final answer. It receives the second-generated answer or directly processes the initial answer if no error is detected.
[0068] According to the system settings, the result output module provides a process description of the factual verification to increase the transparency and credibility of the system.
[0069] Through the collaborative work of the above modules, the knowledge graph-based RAG system factual verification system realizes comprehensive and effective factual verification of the content generated by the RAG system, significantly improving the accuracy and reliability of the answers. This method is not only particularly suitable for professional fields that require high accuracy, but its modular and configurable design also enables it to easily adapt to different application scenarios and knowledge fields, providing a powerful and flexible solution for various complex question-answering needs.
[0070] The knowledge graph-based RAG system factual verification device includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method when executing the computer program.
[0071] The readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0072] In summary, the factual verification method of the RAG system based on the knowledge graph builds a pre-prepared knowledge graph and uses it as an external fact source to verify the answers generated by the RAG system in real time to ensure the accuracy of the output content. When it is detected that the model's answer deviates from the facts in the knowledge graph, the system automatically generates a correction statement and resubmits it to the big model together with the original query, triggering a secondary generation process to correct the error and improve the quality of the answer.
[0073] Compared with the existing technology, it has the following characteristics:
[0074] First, it not only maintains the original flexibility and efficiency of the RAG system, but also integrates the advantages of structured knowledge graphs. Without sacrificing the original performance of the RAG system, it achieves precise control over the generated content. It is particularly suitable for data question-and-answer scenarios that require high accuracy and specialization.
[0075] Second, by selectively enabling knowledge graph validation, the stability and reliability of the RAG system when processing rare or complex data sets are significantly improved.
[0076] Third, the modular design enables the system to easily adapt to a variety of different fields and application scenarios, thereby broadening the application scope of RAG technology and promoting the advancement of artificial intelligence in the fields of information retrieval and natural language processing.
[0077] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.
Claims
1. A factual verification method for RAG system based on knowledge graph, Features: The following steps are involved: Step S1, constructing a knowledge graph; First, collect data from custom fields, extract entities and relationships using natural language processing technology after preprocessing, build a comprehensive knowledge graph, and store it in a high-performance graph database to provide a basis for subsequent factual verification; Step S2: query processing and generating a preliminary answer; When a user query is received, the vector retrieval technology is used to search for relevant information in the pre-prepared document library; the retrieved information is combined with the original query and input into the large language model to generate a preliminary answer; Step S3, factual verification; Extract key information from the preliminary answer, and verify the preliminary answer using a large language model in combination with the knowledge graph, and find potential errors or inconsistencies by comparing the statements in the answer with the facts in the knowledge graph; Step S4, correction and secondary answer; If factual errors are detected, corrective statements are generated based on the correct information in the knowledge graph; Then, the original query is combined with the corrected words and input into the large language model again to generate the final answer after verification and correction. If no factual errors are detected, the preliminary answer is final; Step S5, result output and continuous optimization; The final answer is formatted and presented through the user interface; at the same time, user feedback and performance data are continuously collected, and the knowledge graph and optimization algorithm are regularly updated to ensure the long-term and efficient operation of the system.
2. According to the RAG system factual verification method based on knowledge graph according to claim 1, Features: In step S1, structured and unstructured data in a custom domain are collected, and natural language processing technology is used to implement entity recognition and relationship extraction, so as to construct a comprehensive knowledge graph including entities, attributes and relationships.
3. According to the RAG system factual verification method based on knowledge graph according to claim 1, Features: In step S2, when the user inputs a query, firstly, the vector retrieval technology is used to retrieve relevant information from the pre-prepared document library; then, the retrieved information is combined with the original query through a pre-built prompt template to form a rich context; the query integrating the retrieved information is input into the large language model, and the large language model generates a preliminary answer result.
4. According to the knowledge graph-based RAG system factual verification method of claim 3, Features: In step S3, a prompt template is set for verification by a large language model. The prompt template receives a preliminary answer generated by the large language model, and combines it with the knowledge graph and sends it to the large language model for in-depth analysis of the answer. The large language model combines the knowledge graph to determine whether there are factual errors in the preliminary answer.
5. According to the RAG system factual verification method based on knowledge graph according to claim 4, Features: In step S4, when the factual verification module detects an error, the correction speech generation module is started; based on the correct information in the knowledge graph, the correction speech is intelligently generated through the prompt template set by the large language model.
6. According to the knowledge graph-based RAG system factual verification method of claim 1, Features: In step S5, when the final answer is presented through the user interface, a process description of factual verification is provided.
7. A knowledge graph-based RAG system factual verification system, Features: include: The knowledge graph construction module is responsible for building and maintaining the knowledge graph for factual verification; First, we collect structured and unstructured data in custom domains, and build a comprehensive knowledge graph containing entities, attributes, and relationships by using natural language processing technology, entity recognition, and relationship extraction. RAG retrieval enhancement generation module is responsible for receiving and processing user query requests; When a user enters a query, the RAG retrieval enhancement generation module first uses vector retrieval technology to retrieve relevant information from the pre-prepared document library; Then, the retrieved information is combined with the original query through a pre-built prompt template to form a rich context. The query that incorporates the retrieved information is input into the large language model, which generates a preliminary answer result. The factual verification module is responsible for setting a prompt template for verification through a large language model; The prompt template receives the preliminary answer generated by the RAG search enhancement generation module, and combines it with the knowledge graph and sends it to the large language model for in-depth analysis of the answer. The large language model combines the knowledge graph to determine whether there are factual errors in the preliminary answer. The correction speech generation module is responsible for intelligently generating correction speech based on the correct information in the knowledge graph and the prompt template set by the large language model when the factual verification module detects an error; The secondary answer generation module is responsible for integrating the original query and the generated correction words, and inputting the original query and the generated correction words into the large language model again for factual verification until there are no factual errors in the answer; The output module is responsible for formatting the answer and presenting the final answer. It will receive the second-generated answer, or directly process the initial answer if no error is detected.
8. According to claim 7, the RAG system factual verification system based on knowledge graph, Features: The result output module provides a process description of factual verification.
9. A knowledge graph-based RAG system factual verification device, Features: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 6 when executing the computer program.
10. A readable storage medium, Features: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Error positioning method for retrieval enhancement generation system, electronic equipment and medium
CN119474276A
Activity assessment method for factual knowledge question, product, equipment and medium
CN120180060A
Large model question and answer generation system and method, computer equipment and storage medium
CN121787538A
