Retrieval enhancement generation-based answer tracing method, device and equipment

Abstract

The invention discloses an answer tracing method, device and equipment based on retrieval enhancement generation. The method comprises the following steps: firstly, receiving a user query request, and retrieving knowledge fragments and corresponding source identifiers; based on the retrieval context and the user query request, the large language model generates an answer text; analyzing the answer text, and extracting an answer fragment set; retrieving the answer fragment set through a high-precision matching algorithm to obtain a corresponding target source text fragment; establishing a mapping relation list among the answer fragment, the target source text fragment and the source identifier; and displaying the answer text, and marking the answer fragment. According to the method, accurate mapping of the generated key information points and the knowledge source text is realized by firstly generating and then positioning. And decoupling the traceability task and the generation task. The LLM generates smooth and natural answers. And traceability is carried out through a high-precision matching algorithm. Through the architecture of black box generation and white box verification, the smoothness of LLM and the reliability of traceability are both considered.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more specifically, to a method, apparatus, and system for tracing the source of answers based on retrieval enhancement generation. Background Technology

[0002] Large Language Models (LLMs) are widely used in the field of intelligent question answering, but they have an inherent "hallucination" problem, in which the model may fabricate facts, data or information sources, which limits their application in professional fields that require high reliability (such as medicine, law and finance).

[0003] Retrieval-Augmented Generation (RAG) assists LLM in generating answers by retrieving relevant information from external knowledge bases as context, reducing the "illusion" to some extent. However, the standard RAG generation process remains a "black box." Even if a user receives the answer and a list of references, they cannot know which specific sentence or data in the answer originates from which paragraph of which reference. Because of the lack of a precise mapping between the answer and the source, the reliability of the answer cannot be guaranteed. Users cannot quickly verify the answer, especially for critical data (such as numbers, dates, and amounts), where current technology cannot provide phrase-level source tracing verification. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus and device for tracing the source of answers based on retrieval enhancement generation, which solves the problem that existing retrieval enhancement generation methods cannot provide word-level source verification.

[0005] To achieve the above objectives, the following solution is proposed: A method for tracing the source of answers based on retrieval enhancement includes: Receive user query requests, retrieve relevant knowledge fragments and corresponding source identifiers from the knowledge base, and obtain the retrieval context; Based on the retrieval context and user query request, the large language model generates the answer text; Analyze the response text and extract a set of response fragments; The answer fragment set is retrieved using a high-precision matching algorithm to obtain the corresponding target source text fragment in the retrieval context; Establish a list of mapping relationships between answer fragments, target source text fragments, and source identifiers; Display the answer text and tag the set of answer fragments so that users can trace the source based on the mapping list.

[0006] Preferably, the knowledge base construction process includes: The original document is divided into blocks to obtain knowledge fragments; Assign a globally unique source identifier to each knowledge fragment; Each knowledge fragment is mapped to a vector representation; The knowledge base is obtained by storing the vector representations of each knowledge fragment and the mapping relationship of the corresponding source identifiers.

[0007] Preferably, the process of retrieving relevant knowledge fragments and their corresponding source identifiers from the knowledge base includes: Vectorize the query request to obtain the query vector; Based on the similarity calculation between the query vector and the knowledge fragments in the knowledge base, K related source identifiers are obtained; Based on the source identifier, the corresponding knowledge fragments are extracted to form the retrieval context.

[0008] Preferably, the process of analyzing the answer text and extracting the answer fragment set includes: Extract sets of entities with explicit semantic types from the response text using a named entity recognition model; Extracting key information sets from vertical domains based on rule sets; Merge the entity set and the key information set and remove predefined stop words to obtain the answer fragment set.

[0009] Preferably, the process of retrieving the answer fragment set using a high-precision matching algorithm includes: Initialize the mapping list; Preprocessing of knowledge fragments based on normalization functions; Iterate through each answer fragment in the answer fragment set, and retrieve the source text fragments in the context; If the string of the answer fragment is in the knowledge fragment, then the knowledge fragment is used as the target source text fragment; If the string of the answer fragment is not in the knowledge fragment, then iterate through and search each knowledge fragment in the context, and extract the knowledge fragment of the same type as the target source text fragment; If the search query does not find any knowledge fragments of the same type as the answer fragment, then the knowledge fragment whose edit distance is less than a preset value will be used as the target source text fragment.

[0010] Preferably, the process of establishing the mapping relationship list between the answer fragment, the target source text fragment, and the source identifier includes: When a target source text fragment is retrieved, a mapping entry is created that associates the response fragment with the target source text fragment; A list of mapping relationships is constructed based on each mapping entry. Preferably, the process of displaying the answer text and marking the set of answer fragments includes: Display the plain text of the answer; Based on the mapping list, add interactive visual markers to each answer fragment in the answer text so that users can query the target source text fragment.

[0011] Preferably, the process of a user querying a target source text fragment includes: The user triggers a visual marker for the answer segment, and the system searches for the relevant mapping entries based on the mapping relationship list. Retrieve the corresponding source identifier and target source text fragment based on the mapping entries; Retrieve the corresponding complete knowledge fragment from the knowledge base based on the source identifier; The complete knowledge fragment is displayed, and the corresponding target source text fragment is marked.

[0012] A response tracing device based on retrieval enhancement includes: The user interaction module receives user query requests, retrieves relevant knowledge fragments and corresponding source identifiers from the knowledge base, and obtains the search context. The text generation module generates response text based on the retrieval context and user query request, using a large language model. The text analysis module analyzes the response text and extracts a set of response fragments; The text retrieval module uses a high-precision matching algorithm to retrieve the set of response fragments and obtain the target source text fragments corresponding to the retrieval context. The mapping relationship construction module establishes a list of mapping relationships between answer fragments, target source text fragments, and source identifiers; The display module shows the answer text and marks the set of answer fragments so that users can trace the source based on the mapping list.

[0013] A response tracing device based on retrieval enhancement includes: a memory and a processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the response tracing method based on retrieval enhancement.

[0014] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The retrieval-enhanced response (LLM) method provided by this invention first receives a user query request, retrieves relevant knowledge fragments and corresponding source identifiers to obtain the retrieval context; based on the retrieval context and the user query request, a large language model (MLM) generates response text; the response text is analyzed to extract a set of response fragments; a high-precision matching algorithm is used to retrieve the response fragment set to obtain the target source text fragments corresponding to the retrieval context; a mapping relationship list between response fragments, target source text fragments, and source identifiers is established; the response text is displayed, and the response fragments are marked so that users can trace the source based on the mapping relationship list. This invention adopts a generation-then-location approach, achieving accurate mapping between generated key information points and knowledge source text. It decouples the "source tracing" task from the "generation" task. LLM focuses on generating fluent and natural responses. The "source tracing" task is performed using a high-precision matching algorithm. Through a "black-box generation + white-box verification" architecture, it balances the fluency of LLM with the reliability of source tracing. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 A flowchart of a response tracing method based on retrieval enhancement provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a response tracing device based on retrieval enhancement provided in an embodiment of the present invention; Figure 3 This is a hardware structure block diagram of a response tracing device based on retrieval enhancement provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] First, combined Figure 1 This invention introduces a method for tracing the source of answers based on retrieval enhancement, as provided in an embodiment of the invention. The method process is as follows: Step S01: Receive user query request and retrieve relevant knowledge fragments and corresponding source identifiers from the knowledge base.

[0019] Specifically, each knowledge chunk in the knowledge base is assigned a unique "source identifier". Upon receiving a user query, the system retrieves a batch of relevant knowledge chunks and their corresponding "source identifiers", using this batch of knowledge chunks as the retrieval context.

[0020] (1) The knowledge base preprocessing and index construction process is as follows: Set up a vertical domain knowledge base ,Depend on M The original document consists of:

[0021] in, Indicates the first i The original document.

[0022] The original document is divided into blocks, and the knowledge base is... Each original document in Apply a predefined block function Divide it into a group A knowledge fragment c i,j .

[0023] Ci = f chunk ( d i ) = c i, 1 , c i, 2 , . . . , c i, Ni in, For the original document Knowledge fragments group, j For knowledge fragments in the original document The index number in the database.

[0024] For each knowledge segment c i,j Assign a globally unique source identifier s i,j : s i, j = ( id ( d i ), j ) in, id ( d i (This is the original document) A unique document ID.

[0025] Introduce a text embedding model E model Take any piece of knowledge c Mapped to a k dimensional vector v ∈ R k All knowledge fragments c i,j Mapping to vector representation v i,j : v i, j = E model ( c i, j ) Vector representation of each knowledge fragment v i,j and the corresponding source identifier s i,j The mapping relationships are stored to obtain a vector index knowledge base. DB index : DB index = ( v i, j , s i, j )| i ∈[1, M ], j ∈[1, N i ].

[0026] (2) The retrieval and context building process is as follows: Receive the user's query request Q, and vectorize the query request Q to obtain the query vector: v Q = E model ( Q ).

[0027] use v Q In vector index database DB index In the similarity search, obtain the K most relevant source identifiers. S retrieved = { s (1) ,s (2) , . . . , s (K)}

[0028] Based on source identifier S retrieved From the knowledge base D KB Extract the corresponding K knowledge fragments (source text) to form the retrieval context. C context : C context = c (1) , c (2) , . . . , c (K) in, c (k) yes s (k) The corresponding source text.

[0029] Step S02: Based on the retrieval context and user query request, the large language model generates the answer text.

[0030] Specifically, based on the retrieval context and query request, a response text is generated freely from the Large Language Model (LLM). First, a prompt P is constructed, which contains the query request Q and the retrieval context. C context .

[0031] P = f prompt ( Q , C context ).

[0032] Then, the large language model M is called. LLM Execute prompt P to allow it to generate its own response text. A raw : A raw = M LLM ( P ).

[0033] For example, A raw "According to the 2023 financial report, xxx company's revenue in Chengdu was 150 million yuan."

[0034] Step S03: Analyze the response text and extract the response fragment set.

[0035] Specifically, the response text is analyzed to identify one or more response segments that need to be traced back to their source, constructing a set of response segments. Examples include numbers, place names, times, and proper nouns. A composite segment recognition function is then applied. f identify To answer text A raw The above is used to extract one or more traceable spans of the response that need to be traced. T A .

[0036] f identify Defined as a multi-strategy process: First, a pre-trained named entity recognition model is used. M NER From the answer text A raw Extract entity sets with explicit semantic types T NER Named entity recognition models can be NER models based on BERT or Transformer.

[0037] T NER = t | t ∈ M NER ( A raw ), type(t) ∈ ORG , LOC , PER , PRODUCT , ... For example: T NERR = {("xxx company"),("Chengdu")}.

[0038] Then, a rule set containing multiple regular expressions is applied. R set To match M NER A set of critical information that is not covered but is important in vertical sectors (such as finance and healthcare). R set At least including r date Used to match dates and years (e.g., "2023", "October 31").

[0039] r money Used to match currency amounts (e.g., "150 million yuan", "$200,000").

[0040] r numeric Used to match other key numbers and percentages (e.g., "25% increase", "dosage"). 50mg).

[0041] T Rule = t | t = match ( A raw , r ), r ∈R set For example: T Rule = {("2023"),("150 million yuan")}.

[0042] Finally, merge the entity sets. T NER and key information set T Rule Then, predefined stop words or meaningless short segments are removed to obtain the final set of answer segments. T A .

[0043] Each of them t A,p ∈ T A Each is a tuple containing the text and its corresponding response text. A raw The position in the middle, t A,p =(text p ,start p end p ), text p To answer the text fragment, start p To answer the question about the start position of the segment, end p () indicates the end position of the answer segment.

[0044] Step S04: Retrieve the set of answer fragments using a high-precision matching algorithm to obtain the target source text fragments corresponding to the retrieval context.

[0045] Specifically, within the retrieval context (i.e., within the retrieved batch of knowledge fragments), a high-precision matching algorithm (such as string, pattern, or fuzzy matching) is used to retrieve the answer text fragments in order to accurately locate the target source text fragments corresponding to them within the knowledge fragments.

[0046] First, initialize an empty list of mapping relationships: M final = .

[0047] Then, define a normalization function. f norm ( s ), used to compare texts before comparison. s Preprocessing is performed. f norm At least: ① Remove punctuation marks, spaces, and control characters.

[0048] ② Convert Chinese numerals to Arabic numerals (e.g., "one million five hundred thousand") "1500000").

[0049] ③ Convert units (e.g., “150 million” → “150,000,000”; “150 million yuan” → “150,000,000”).

[0050] ④ Remove the organization suffix (e.g., "xxx company" → "xxx").

[0051] ⑤ Convert to lowercase (applicable to English).

[0052] Secondly, traverse the set of answer fragments. T A Excerpts from various answers t A,p and retrieving source text fragments from the context. c (k) .

[0053] Finally, a multi-strategy high-precision positioning function is applied. f locate ,exist c (k) Search and t A,p Answer fragment text p Matching source text fragments T C,k,p .

[0054] f locate Matching will be performed with the following priority: Strategy 1: Exact string matching.c (k) Directly search for text p If the string of the answer fragment is in the knowledge fragment, that is... t C,q = (text q ,start q end q If the text is a source text fragment, then the source text fragment will be used as the target source text fragment. q For knowledge fragment text, start q This is the starting position of the knowledge fragment text, end. q This indicates the end position of the knowledge fragment text.

[0055] Strategy 2: Normalized string matching. If Strategy 1 does not find a match, then calculate: text' p = fn orm (text p ).

[0056] Then, iterate through... c (k) Extract all related to t A,p Candidate knowledge fragments of the same type (e.g., both numbers or both organization names) c candidate .like f norm ( c candidate ) = text' p Then c candidate Corresponding source text fragment t C,q As the target source text fragment.

[0057] Strategy 3: Fuzzy string matching. If strategies 1 and 2 both fail, then calculate the text. p and c (k) Edit distance of candidate source text fragments d L .if d L If the value is less than a preset threshold θ (e.g., θ=1 or θ=2 to allow for minor typos), then the source text fragment will be... t C,q As the target source text fragment. Source text fragment set. T C, k, p : T C, k, p= f locate ( text p , c (k) ); t C,q ∈ T C, k, p .

[0058] T C, k, p It is a list containing c (k) All matched target source text fragments.

[0059] Step S05: Establish a list of mapping relationships between answer fragments, target source text fragments, and source identifiers.

[0060] Specifically, a high-precision mapping list is established between the answer fragment (its position in the answer) and the target source text fragment (its position in the source fragment) and its "source identifier".

[0061] When a target source text fragment is retrieved, a mapping entry is created. m Mapping entries m Related answer snippets and target source text snippets: m = ( t A,p , s (k) , t C,q ) Right now m = ((text p ,start p end p ), s(k),(text q ,start q end q )) Construct a mapping relationship list based on each mapping entry, and m Add to the final mapping list M final middle.

[0062] Step S06: Display the answer text and mark the set of answer fragments so that users can trace the source based on the mapping relationship list.

[0063] Specifically, on the user interface, the answer text is highlighted, and users are allowed to view the corresponding source text fragment and its context through interactions such as clicking and hovering.

[0064] First, display the plain text of the answer to the user. A raw Then, iterate through the list of mapping relationships. M final According to the mapping relationship list M final , to mapping entries m Excerpts from various answers t A,p In response to the text A raw Corresponding position (start) p to end p Add interactive visual markers (e.g., highlighting, underlining) to allow users to search for source text fragments.

[0065] When a user triggers a visual marker for an answer segment by clicking or hovering, firstly, based on a list of mapping relationships... M final Find answer snippets t A,p Related mapping entries m Obtain the corresponding source identifier based on the mapping entry. s (k) and source text fragments t C,q Then, based on the source identifier... s (k) Retrieve the corresponding complete knowledge fragment from the knowledge base. c (k) Finally, the complete knowledge fragment... c (k) Display it, and based on its position (start) q to end q ) knowledge fragments c (k) Source text fragments t C,q By marking words, a precise correspondence between phrases can be achieved.

[0066] The following describes the answer tracing device based on retrieval enhancement provided in the embodiments of the present invention. The answer tracing device based on retrieval enhancement described below can be referred to in correspondence with the answer tracing method based on retrieval enhancement described above.

[0067] First, combine Figure 2 This section introduces a retrieval-enhanced response tracing device, such as... Figure 2 As shown, the response tracing device based on retrieval enhancement may include: User interaction module 100 receives user query requests, retrieves relevant knowledge fragments and corresponding source identifiers from the knowledge base, and obtains the retrieval context; The text generation module 200 generates response text based on the retrieval context and user query request, using a large language model. The text analysis module 300 analyzes the response text and extracts a set of response fragments; The text retrieval module 400 retrieves the answer fragment set through a high-precision matching algorithm to obtain the target source text fragment corresponding to the retrieval context; The mapping relationship construction module 500 establishes a list of mapping relationships between answer fragments, target source text fragments, and source identifiers. Display module 600 displays the answer text and marks the set of answer fragments so that users can trace the source based on the mapping relationship list.

[0068] The retrieval-enhanced response tracing device provided in this embodiment of the invention can be applied to retrieval-enhanced response tracing equipment. Figure 3 The hardware structure block diagram of the answer tracing device based on retrieval enhancement is shown, with reference to... Figure 3 The hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4; In this embodiment of the invention, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4. Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device; The memory stores a program, and the processor can call the program stored in the memory. The program is used to implement the various processing flows in the aforementioned response tracing scheme based on retrieval enhancement. This invention also provides a storage medium that can store a program suitable for execution by a processor, the program being used to implement the various processing flows in the aforementioned response tracing scheme based on retrieval enhancement.

[0069] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0070] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0071] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for tracing the source of answers based on retrieval enhancement, characterized in that, include: Receive user query requests, retrieve relevant knowledge fragments and corresponding source identifiers from the knowledge base, and obtain the retrieval context; Based on the retrieval context and user query request, the large language model generates the answer text; Analyze the response text and extract a set of response fragments; The answer fragment set is retrieved using a high-precision matching algorithm to obtain the corresponding target source text fragment in the retrieval context; Establish a list of mapping relationships between answer fragments, target source text fragments, and source identifiers; Display the answer text and tag the set of answer fragments so that users can trace the source based on the mapping list.

2. The answer tracing method based on retrieval enhancement generation according to claim 1, characterized in that, The knowledge base construction process includes: The original document is divided into blocks to obtain knowledge fragments; Assign a globally unique source identifier to each knowledge fragment; Each knowledge fragment is mapped to a vector representation; The knowledge base is obtained by storing the vector representations of each knowledge fragment and the mapping relationship of the corresponding source identifiers.

3. The answer tracing method based on retrieval enhancement generation according to claim 1, characterized in that, The process of retrieving relevant knowledge fragments and their corresponding source identifiers from the knowledge base includes: Vectorize the query request to obtain the query vector; Based on the similarity calculation between the query vector and the knowledge fragments in the knowledge base, K related source identifiers are obtained; Based on the source identifier, the corresponding knowledge fragments are extracted to form the retrieval context.

4. The answer tracing method based on retrieval enhancement generation according to claim 1, characterized in that, The process of analyzing the response text and extracting a set of response fragments includes: Extract sets of entities with explicit semantic types from the response text using a named entity recognition model; Extracting key information sets from vertical domains based on rule sets; Merge the entity set and the key information set and remove predefined stop words to obtain the answer fragment set.

5. The answer tracing method based on retrieval enhancement generation according to claim 1, characterized in that, The process of retrieving the set of answer fragments using a high-precision matching algorithm includes: Initialize the mapping list; Preprocessing of knowledge fragments based on normalization functions; Iterate through each answer fragment in the answer fragment set, and retrieve the source text fragments in the context; If the string of the answer fragment is in the knowledge fragment, then the knowledge fragment is used as the target source text fragment; If the string of the answer fragment is not in the knowledge fragment, then iterate through and search each knowledge fragment in the context, and extract the knowledge fragment of the same type as the target source text fragment; If the search query does not find any knowledge fragments of the same type as the answer fragment, then the knowledge fragment whose edit distance is less than a preset value will be used as the target source text fragment.

6. The answer tracing method based on retrieval enhancement generation according to claim 5, characterized in that, The process of establishing a list of mapping relationships between answer fragments, target source text fragments, and source identifiers includes: When a target source text fragment is retrieved, a mapping entry is created that associates the response fragment with the target source text fragment; A list of mapping relationships is constructed based on each mapping entry.

7. The answer tracing method based on retrieval enhancement generation according to claim 1, characterized in that, The process of displaying the answer text and marking the set of answer fragments includes: Display the plain text of the answer; Based on the mapping list, add interactive visual markers to each answer fragment in the answer text so that users can query the target source text fragment.

8. The answer tracing method based on retrieval enhancement generation according to claim 7, characterized in that, The process of a user querying a target source text fragment includes: The user triggers a visual marker for the answer segment, and the system searches for the relevant mapping entries based on the mapping relationship list. Retrieve the corresponding source identifier and target source text fragment based on the mapping entries; Retrieve the corresponding complete knowledge fragment from the knowledge base based on the source identifier; The complete knowledge fragment is displayed, and the corresponding target source text fragment is marked.

9. A response tracing device based on retrieval enhancement generation, characterized in that, include: The user interaction module receives user query requests, retrieves relevant knowledge fragments and corresponding source identifiers from the knowledge base, and obtains the search context. The text generation module generates response text based on the retrieval context and user query request, using a large language model. The text analysis module analyzes the response text and extracts a set of response fragments; The text retrieval module uses a high-precision matching algorithm to retrieve the set of response fragments and obtain the target source text fragments corresponding to the retrieval context. The mapping relationship construction module establishes a list of mapping relationships between answer fragments, target source text fragments, and source identifiers; The display module shows the answer text and marks the set of answer fragments so that users can trace the source based on the mapping list.

10. A response tracing device based on retrieval enhancement generation, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the response tracing method based on retrieval enhancement generation as described in any one of claims 1-8.

Images