Question and answer tracing method and system based on large language model and retrieval enhancement generation
By introducing a fine-grained tracing process into the RAG system, the user-selected content is compared and analyzed with each slice, and the original sentences are marked. This solves the problems of coarse tracing granularity and information fusion distortion in the existing system, and realizes accurate and reliable tracing of answers and interactive verification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA HYDROELECTRIC ENGINEERING CONSULTING GROUP CHENGDU RESEARCH HYDROELECTRIC INVESTIGATION DESIGN AND INSTITUTE
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-15
AI Technical Summary
Existing RAG question-answering systems have significant shortcomings in terms of the traceability and credibility verification of answers. The traceability granularity is coarse and the information fusion is distorted, making it difficult to assess the credibility of the answers.
By introducing a fine-grained source tracing process into the large language model, the user-selected source content is compared and analyzed with each retrieved slice one by one, the original sentences are identified and marked, and the accurate mapping from the answer clause to the original sentence is achieved, and the marking is visualized on the front-end interface.
It enables precise location of any clause in the answer and the original evidence, improves the credibility and auditability of the question-and-answer results, allows users to quickly and intuitively verify the source of the content, and significantly enhances the application value in serious fields.
Smart Images
Figure CN122045358A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a question-answering source tracing method and system based on a large language model and retrieval enhancement generation. Background Technology
[0002] Question answering systems based on Large Language Models (LLMs) demonstrate powerful capabilities in generating fluent and coherent text. However, their inherent "illusion" problem—generating content that is inconsistent with or lacks supporting evidence from the input—limits their application in scenarios requiring high accuracy and reliability, such as professional consultation, legal analysis, and medical assistance. To alleviate this problem, Retrieval-Augmented Generation (RAG) technology has been widely adopted.
[0003] The typical workflow of a RAG system is as follows: First, external knowledge documents are segmented (sliced) and quantized to construct a searchable vector knowledge base. When a user asks a question, the system retrieves several text slices from the vector base that are most semantically relevant to the question. Then, these retrieved slices are used as contextual information and input along with the user's question into an LLM (Local Level Model), which synthesizes the information to generate the final answer. This method, by introducing external knowledge sources, effectively enhances the factual basis of the model's answer and reduces the generation of unsubstantiated content.
[0004] However, existing RAG question-answering systems still have significant shortcomings in terms of the traceability and credibility verification of answers, mainly in the following two aspects: First, the granularity of source tracing is coarse and lacks precision. Existing technologies can typically only establish a macroscopic relationship between the "overall answer" and the "set of reference slices." When providing an answer, it often only lists one or more document slices referenced in generating the entire answer, but cannot further reveal which specific statement, fact, or opinion within the answer corresponds to which original text segment in the reference slices. When users want to verify the authenticity of a sentence in the answer, they have to manually read through the entire cited slice to find the corresponding source, a tedious and inefficient process. This coarse-grained source tracing method fails to achieve true precision, significantly reducing the practical usability of the tracing results.
[0005] Second, information fusion distortion weakens credibility. When LLM generates answers based on the fusion of multiple retrieval slices, existing technologies cannot clearly define and present the independent source of each part of the information in the answer. This leads to two prominent problems: First, the model may engage in uncontrolled "creative" processing during the fusion process, producing statements or inferences that do not exist in the source slices, a process that cannot be detected in existing source tracing results; second, when different reference slices have minor conflicts or differences in details, users cannot identify which specific source a conclusion in the answer is based on, making it difficult to assess its authority and accuracy. This ambiguity of source causes the credibility assessment of the answer to remain at the level of overall feeling, seriously weakening the credibility of the answer and its practical value in serious scenarios. Summary of the Invention
[0006] This invention aims to address the problem that the coarse granularity of existing RAG systems makes it difficult to assess the credibility of question-and-answer results, and proposes a question-and-answer source tracing method and system based on a large language model and retrieval enhancement generation.
[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a question-answering source tracing method based on a large language model and retrieval enhancement generation, the method comprising: Deploy a large language model and its corresponding retrieval enhancement generation system. Use the retrieval enhancement generation system to slice, vectorize, and store knowledge documents in a vector library to build a knowledge base. The system receives user questions, retrieves multiple slices related to the user questions from the vector library through the retrieval enhancement generation system, inputs the multiple slices into the large language model to generate a final answer, and associates the multiple slices with the final answer. Receive the user's selected source-tracing content from the final answer, and combine the source-tracing content with each of the multiple slices to form a comparison request with the same number of slices; For each comparison request, the large language model is invoked to compare and analyze the corresponding source content to be traced with the corresponding slice, to determine whether there is a source sentence related to the source content in the corresponding slice, and to return the source tracing result; the source tracing result is a list or an empty list that includes at least one source sentence. Based on the source tracing results, the original sentences existing in the slice are located and marked on the front-end interface to complete the source tracing of the content to be traced.
[0008] Furthermore, the large language model is invoked to compare and analyze the corresponding source content with the corresponding slice, specifically including: The source content to be traced and a slice are submitted as input to the large language model, which returns at least one original sentence in the slice that is related to the source content to be traced; if no related original sentence exists, an empty list is returned.
[0009] Furthermore, the original sentences are marked on the front-end interface, including: Based on each original sentence in the source tracing results, the text position of the original sentence in its corresponding slice is determined by matching; In the front-end interface, the text content within the specified text location range is marked.
[0010] Furthermore, the marking includes: bold display or highlighting with a specified color.
[0011] Furthermore, receiving the user's selected source content from the final answer is specifically achieved by capturing the user's selection or checkmark operations on the final answer text in the front-end interface.
[0012] Furthermore, the method also includes: Before starting the source tracing process, reset the marking status of all slices on the front-end interface; After obtaining the source tracing result, if the source tracing result is not empty, the front-end identifier corresponding to the slice of the matched original text sentence is set to the marked state.
[0013] Furthermore, after retrieving multiple slices related to the user's question from the vector library, the retrieval enhancement generation system also rearranges the retrieved slices according to their relevance using a rearrangement model, and then selects the top N slices to input into the large language model to generate the final answer.
[0014] Furthermore, if the tracing results for all slices are empty lists, the front-end interface will display a message indicating that no relevant content was found.
[0015] In a second aspect, the present invention provides a question-answering source tracing system based on a large language model and retrieval enhancement generation, for implementing the question-answering source tracing method based on a large language model and retrieval enhancement generation as described in the first aspect, the system comprising: The knowledge base construction module is used to deploy a large language model and its corresponding retrieval enhancement generation system. The retrieval enhancement generation system slices, vectorizes, and stores knowledge documents in a vector library to construct the knowledge base. The answer generation module is used to receive user questions, retrieve multiple slices related to the user questions from the vector library through the retrieval enhancement generation system, input the multiple slices into the large language model to generate the final answer, and associate the multiple slices with the final answer. The request combination module is used to receive the content to be traced from the final answer selected by the user, and combine the content to be traced with each of the multiple slices to form a comparison request with the same number of slices. The comparison and analysis module is used to call the large language model to compare and analyze the corresponding source content to be traced with the corresponding slice for each comparison request, determine whether there is a source sentence related to the source content in the corresponding slice, and return the source tracing result; the source tracing result is a list or an empty list including at least one source sentence; The results display module is used to locate the original sentences in the slice based on the source tracing results, and mark the original sentences on the front-end interface to complete the source tracing of the content to be traced.
[0016] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed, implements the steps of the question-answering tracing method based on a large language model and retrieval enhancement as described in the first aspect.
[0017] The beneficial effects of this invention are as follows: The question-answering tracing method and system based on large language model and retrieval enhancement provided by this invention achieves fine-grained and precise mapping from "answer clauses" to "original sentences" by performing LLM comparisons one by one between the user-selected content to be traced and multiple slices retrieved by RAG. This solves the problem of coarse granularity in traditional tracing, which requires users to manually screen the entire slice. By marking the matched original sentences on the front end, an intuitive interactive verification loop is constructed, which allows users to clearly identify and distinguish between "facts supported by the original text" and "potential illusions without matching results" in the answer. Ultimately, without reconstructing the existing RAG system, the credibility, auditability, and application value in serious fields of the answer are significantly improved. Attached Figure Description
[0018] Figure 1 A flowchart illustrating the question-answering source tracing method based on a large language model and retrieval enhancement provided for this embodiment; Figure 2 This is a flowchart illustrating the process of comparing and analyzing the content to be traced with the corresponding slice, as provided in the example. Figure 3 This is a schematic diagram of the structure of a question-answering tracing system based on a large language model and retrieval enhancement provided for an embodiment. Detailed Implementation
[0019] Existing RAG-based question-answering systems can only establish a coarse-grained relationship between the overall answer and the reference slice set, failing to achieve a fine-grained and precise mapping between specific statements within the answer and the original evidence. Furthermore, they struggle to trace and distinguish the specific sources of the merged content, making it difficult to effectively verify the credibility of the answer. Therefore, there is an urgent need for a question-answering method that enables fine-grained and precise source tracing, allowing users to quickly and intuitively locate the original evidence supporting any clause in the answer, thereby completing interactive verification and precise evaluation of the credibility of the generated content.
[0020] Based on this, the technical solution of this invention is proposed. In this invention, an independent, fine-grained tracing process is introduced on top of the standard RAG-generated answer. When a user requests tracing of any selected content (the content to be traced) in the generated answer, instead of directly displaying all related reference slices, the selected content is independently compared with each original slice retrieved by RAG. During the comparison, the powerful semantic understanding and content analysis capabilities of LLM are utilized to determine the relevance between the "content to be traced" and the full text of a single "slice," identifying and extracting the original sentences in the slice that directly support or correspond to that content. Subsequently, the comparison results of each slice are summarized, and these located sentences in the original knowledge document are precisely visualized and marked on the front-end interface. The entire process achieves a precise mapping from the "user-selected answer clause" to "specific evidence fragments scattered throughout the various original slices."
[0021] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0022] Figure 1 A flowchart illustrating a question-answering source tracing method based on a large language model and retrieval enhancement is shown below. Figure 1 The method includes the following steps: Step 1: Deploy the large language model and its corresponding retrieval enhancement generation system. The retrieval enhancement generation system is used to slice, vectorize, and store knowledge documents in a vector library to build a knowledge base.
[0023] In practical applications, the first step is to deploy a large language model and build a corresponding retrieval enhancement and generation system. Then, users upload knowledge documents to be answered through the retrieval enhancement and generation system. The system automatically processes the knowledge documents into text slices, and then converts each slice into a vector representation through word embedding technology. Finally, these vectorized slices are stored in a vector library, thereby completing the construction of the knowledge base and providing structured knowledge reserves for subsequent retrieval and question answering.
[0024] Step 2: Receive user questions, retrieve multiple slices related to the user questions from the vector library through the retrieval enhancement generation system, input the multiple slices into the large language model to generate the final answer, and associate the multiple slices with the final answer.
[0025] In practical applications, after receiving a user's question, the retrieval enhancement generation system performs a semantic search in the pre-built vector library based on the user's question, obtaining a series of relevant document slices. Next, it calls the reordering model in the retrieval enhancement generation system to sort these retrieved slices according to their relevance to the question, and selects the top N most relevant slices based on the user's preset or system default number N. Then, these N slices are used as contextual information and input into the deployed large language model, which understands, integrates, and generates a final answer that conforms to human reading habits. At the same time, this final answer is associated with the N slices used to generate it, and the answer and the list of associated slices are returned to the front-end interface for display, thus completing the entire process from asking a question to generating an answer and retaining the source evidence.
[0026] Step 3: Receive the user's selected source-tracing content from the final answer, and combine the source-tracing content with each of the multiple slices to form a comparison request with the same number of slices.
[0027] In practical applications, when a user views the generated final answer on the front-end interface, they can specify the content to be traced from the answer text by selecting or checking with the mouse. After capturing the selected content, the front-end system obtains a list of all N slices associated with this answer. Subsequently, the content to be traced is combined with these N slices. Specifically, the back-end interface receives a request from the front-end and pairs the content to be traced with each slice separately, submitting it N times per slice, thus forming N independent comparison requests. Each request contains the same content to be traced and a specific slice, preparing for subsequent detailed comparison of each slice.
[0028] Step 4: For each comparison request, call the large language model to compare and analyze the corresponding source content to be traced with the corresponding slice, determine whether there is a source sentence related to the source content in the corresponding slice, and return the source tracing result; the source tracing result is a list or an empty list including at least one source sentence.
[0029] Please see Figure 2In this embodiment, the large language model is invoked to compare and analyze the corresponding source content to be traced with the corresponding slice. Specifically, this includes: submitting the source content to be traced and a slice as input to the large language model, and the large language model returning at least one original sentence in the slice that is related to the source content to be traced; if there is no related original sentence, an empty list is returned.
[0030] In practical applications, for each comparison request generated in step 3, the backend program calls the deployed large language model to perform the comparison analysis task. Specifically, each call takes the "content to be traced" and a corresponding "slice" full text as input, instructing the large language model to analyze and determine whether there are semantically supporting, explaining, or corresponding original sentences in the slice. After the large language model completes the analysis, it directly outputs the relevant original sentences in the slice as the result. If there are multiple relevant sentences, it returns a list containing multiple sentences; if it is determined that there are no relevant sentences, it returns an empty list. After each slice goes through this independent comparison process, it generates a "tracing result", that is, a list of sentences or an empty list, and returns it to the front end, thereby providing accurate data basis for subsequent visualization.
[0031] Step 5: Based on the source tracing results, locate the original sentences in the slice and mark the original sentences on the front-end interface to complete the source tracing of the content to be traced.
[0032] In this embodiment, the original text sentence is marked on the front-end interface, including: Based on each original sentence in the source tracing results, the text position of that original sentence within its corresponding slice is determined; in the front-end interface, the text content within the range of the stated text position is marked. The marking includes: bolding or highlighting with a specified color.
[0033] In practical applications, after receiving the source tracing results of all N slices returned by the backend, the frontend system first performs parsing and location: for each non-empty source tracing result, the frontend performs precise matching between the original sentence and the corresponding slice's full text to determine the specific text position of each original sentence within that slice (e.g., start and end character indices); subsequently, it performs visual marking in the user interface: the system adds a highlighted mark to the reference list sidebar for successfully matched slices. Simultaneously, if the user has opened the slice's details pop-up, it searches for these sentences in the pop-up content using secure regular expressions and performs visual marking. For example, using HTML tags with specific styles (e.g., specifying background color) (e.g.) <mark>The system wraps and renders the matched original sentences using tags to achieve precise and eye-catching highlighting; for slices with empty lists of source tracing results, the front end does not perform any marking processing; finally, the system gives the user a corresponding prompt based on whether there are highlighted results, such as "Source tracing complete"; if the source tracing results for all slices are empty lists, the front end interface will prompt "No relevant content found", thus completing the complete source tracing presentation from data matching to visual interaction.
[0034] In this embodiment, the method further includes: Before starting the source tracing process, reset the marking status of all slices on the front-end interface; After obtaining the source tracing result, if the source tracing result is not empty, the front-end identifier corresponding to the slice of the matched original text sentence is set to the marked state.
[0035] Specifically, when a user selects the answer text on the front-end interface to trigger source tracing, the interface state will be reset before initiating a comparison request. This means clearing any markup styles that may exist in the front-end interface for all reference slices associated with the current answer, and clearing the globally stored set of marked sentences. This ensures that each source tracing interaction starts from a clear visual baseline, avoiding visual confusion between previous and subsequent source tracing results on the interface.
[0036] In summary, the question-answering source tracing method based on large language models and retrieval enhancement provided in this embodiment achieves fine-grained and precise mapping between any clause in the answer and specific evidence sentences in the original document by performing LLM comparison analysis on each of the user-selected content to be traced and each retrieved slice. This overcomes the limitation of traditional RAG systems, which can only coarsely associate answers with slice sets. By accurately locating and marking the matched original sentences on the front end, and combining the state reset and result feedback mechanism before source tracing, an intuitive interactive verification loop is constructed, enabling users to directly and quickly confirm the source of each statement in the answer and effectively distinguish between verifiable facts and untraceable model illusions. Finally, without modifying the existing RAG core architecture, the credibility, auditability, and application feasibility of the question-answering results are significantly improved.
[0037] Based on the above technical solutions, this embodiment also provides a question-answering source tracing system based on a large language model and retrieval enhancement generation, used to implement the question-answering source tracing method based on a large language model and retrieval enhancement generation as described in the embodiment. Please refer to [link to relevant documentation]. Figure 3 The system includes: The knowledge base construction module is used to deploy a large language model and its corresponding retrieval enhancement generation system. The retrieval enhancement generation system slices, vectorizes, and stores knowledge documents in a vector library to construct the knowledge base. The answer generation module is used to receive user questions, retrieve multiple slices related to the user questions from the vector library through the retrieval enhancement generation system, input the multiple slices into the large language model to generate the final answer, and associate the multiple slices with the final answer. The request combination module is used to receive the content to be traced from the final answer selected by the user, and combine the content to be traced with each of the multiple slices to form a comparison request with the same number of slices. The comparison and analysis module is used to call the large language model to compare and analyze the corresponding source content to be traced with the corresponding slice for each comparison request, determine whether there is a source sentence related to the source content in the corresponding slice, and return the source tracing result; the source tracing result is a list or an empty list including at least one source sentence; The results display module is used to locate the original sentences in the slice based on the source tracing results, and mark the original sentences on the front-end interface to complete the source tracing of the content to be traced.
[0038] It is understood that the question-answering source tracing system based on large language model and retrieval enhancement described in this embodiment is a system for implementing the question-answering source tracing method based on large language model and retrieval enhancement described in the embodiment. As the system disclosed in the embodiment corresponds to the method disclosed in the embodiment, the description is relatively simple. For relevant parts, please refer to the description of the method. It will not be repeated here.
[0039] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.< / mark>
Claims
1. A question-answering source tracing method based on large language models and retrieval enhancement generation, characterized in that, The method includes: Deploy a large language model and its corresponding retrieval enhancement generation system. Use the retrieval enhancement generation system to slice, vectorize, and store knowledge documents in a vector library to build a knowledge base. The system receives user questions, retrieves multiple slices related to the user questions from the vector library through the retrieval enhancement generation system, inputs the multiple slices into the large language model to generate a final answer, and associates the multiple slices with the final answer. Receive the user's selected source-tracing content from the final answer, and combine the source-tracing content with each of the multiple slices to form a comparison request with the same number of slices; For each comparison request, the large language model is invoked to compare and analyze the corresponding source content to be traced with the corresponding slice, to determine whether there is a source sentence related to the source content in the corresponding slice, and to return the source tracing result; the source tracing result is a list or an empty list that includes at least one source sentence. Based on the source tracing results, the original sentences existing in the slice are located and marked on the front-end interface to complete the source tracing of the content to be traced.
2. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 1, characterized in that, The large language model is invoked to compare and analyze the corresponding source-tracing content with the corresponding slices, specifically including: The source content to be traced and a slice are submitted as input to the large language model, which returns at least one original sentence in the slice that is related to the source content to be traced; if no related original sentence exists, an empty list is returned.
3. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 1, characterized in that, The original sentences are marked on the front-end interface, including: Based on each original sentence in the source tracing results, the text position of the original sentence in its corresponding slice is determined by matching; In the front-end interface, the text content within the specified text location range is marked.
4. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 3, characterized in that, The markings include: bold display or highlighting with a specified color.
5. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 1, characterized in that, The system receives the user's selected source content from the final answer, specifically by capturing the user's selection or checkmark operations on the final answer text in the front-end interface.
6. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 1, characterized in that, The method further includes: Before starting the source tracing process, reset the marking status of all slices on the front-end interface; After obtaining the source tracing result, if the source tracing result is not empty, the front-end identifier corresponding to the slice of the matched original text sentence is set to the marked state.
7. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 1, characterized in that, After retrieving multiple slices related to the user's question from the vector library, the retrieval enhancement generation system rearranges the retrieved slices according to their relevance using a rearrangement model, and then selects the top N slices to input into the large language model to generate the final answer.
8. The question-answering source tracing method based on a large language model and retrieval enhancement generation according to claim 1, characterized in that, If the tracing results for all slices are empty lists, the front-end interface will display a message indicating that no relevant content was found.
9. A question-answering source tracing system based on a large language model and retrieval enhancement, characterized in that: The system is used to implement the question-answering source tracing method based on a large language model and retrieval enhancement as described in any one of claims 1 to 8, the system comprising: The knowledge base construction module is used to deploy a large language model and its corresponding retrieval enhancement generation system. The retrieval enhancement generation system slices, vectorizes, and stores knowledge documents in a vector library to construct the knowledge base. The answer generation module is used to receive user questions, retrieve multiple slices related to the user questions from the vector library through the retrieval enhancement generation system, input the multiple slices into the large language model to generate the final answer, and associate the multiple slices with the final answer. The request combination module is used to receive the content to be traced from the final answer selected by the user, and combine the content to be traced with each of the multiple slices to form a comparison request with the same number of slices. The comparison and analysis module is used to call the large language model to compare and analyze the corresponding source content to be traced with the corresponding slice for each comparison request, determine whether there is a source sentence related to the source content in the corresponding slice, and return the source tracing result; the source tracing result is a list or an empty list including at least one source sentence; The results display module is used to locate the original sentences in the slice based on the source tracing results, and mark the original sentences on the front-end interface to complete the source tracing of the content to be traced.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the steps of the question-answering tracing method based on a large language model and retrieval enhancement as described in any one of claims 1 to 8.