RAG-based vertical domain knowledge multi-round question and answer method
By reconstructing the knowledge base documents and constructing a vector database, and combining a large language model to process user questions, we have achieved accurate understanding of user intent and efficient retrieval of answers in multi-turn question answering. This solves the problem of insufficient understanding of context and flexibility in existing systems in multi-turn question answering, and improves the accuracy and recall of retrieval.
Patent Information
- Application Number
- CN202410997916.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing question-answering systems based on document knowledge bases or knowledge graphs struggle to understand the context of user questions in multi-turn question-answering, fail to provide satisfactory answers that integrate multiple knowledge points, lack flexibility when handling non-standardized or open-ended questions, have low retrieval accuracy, and are prone to missing important semantic information.
By reconstructing the knowledge base documents, dense and sparse vector databases are constructed. A hybrid query and reordering model is used to sort document chunks, perform deduplication and merging, combine a large language model for question understanding and expansion, filter historical dialogues, and insert the final search results into the dialogue records.
It improves the retrieval success rate and recall rate of multi-turn question answering for vertical domain knowledge, avoids the mixing of useless information in historical dialogues, and ensures accurate understanding of user intent and precision of answers.
Smart Images

Figure CN118964556A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and specifically is a vertical domain knowledge multi-round question-answering method based on RAG. Background Art
[0002] Professional question-and-answer in vertical fields is an important scenario for the implementation of artificial intelligence. Traditional question-and-answer systems based on document knowledge bases or knowledge graphs have the following main problems: 1. They cannot understand the context of user questions well, especially in multiple rounds of question-and-answer sessions, where the system has difficulty maintaining the continuity of the conversation; 2. They are difficult to provide satisfactory answers to questions that require the integration of multiple knowledge points or require reasoning and analysis; 3. They lack flexibility when dealing with non-standard or open-ended questions and are difficult to adapt to different ways of asking questions.
[0003] With the widespread use of generative large language models (LLM), vertical field question-answering systems based on retrieval augmented generation (RAG) have solved the above problems to a certain extent. However, there may still be problems such as complex questions not being broken down, resulting in low retrieval accuracy, missing important semantic information, retrieved document blocks losing important semantics and reading order, document blocks not containing answers, historical conversations mixed with useless information, and document blocks being too long to drown out the user's true intentions, resulting in less than ideal performance of the dialogue system. Summary of the invention
[0004] The purpose of the present invention is to provide a vertical domain knowledge multi-round question-answering method based on RAG to solve the problems raised in the above background technology.
[0005] In order to achieve the above object, the present invention provides the following technical solution: a vertical domain knowledge multi-round question-answering method based on RAG, comprising the following steps:
[0006] S1, reconstruct the documents in the knowledge base and build dense and sparse vector databases;
[0007] Preferably, the detailed steps in step S1 are:
[0008] S101, extracting hierarchical structure information of a document, wherein the hierarchical structure information includes but is not limited to a document title, a chapter title, an ordered list, an unordered list, a table, a picture, a table title, and a picture title;
[0009] Preferably, the extraction method includes manual extraction, algorithm-based automatic extraction, and algorithm-based automatic extraction plus manual review;
[0010] S102, dividing the document into large blocks according to the hierarchical structure information;
[0011] Furthermore, an upper limit and a lower limit of the number of characters in a large block are set, the document is divided into large blocks in combination with the hierarchical structure information, and the number of characters in each large block is controlled between the upper limit and the lower limit of the number of characters in the large block;
[0012] S103, completing the block hierarchical semantic information: according to the hierarchical structure information, obtaining the title and content of the upper level of the block, performing named entity recognition on the title and content of the upper level, extracting important entities in the vertical field, and adding the upper level title and upper level important entities to the front of the block;
[0013] Furthermore, the upper level of the large block includes at least one of the following:
[0014] 1) Document title; 2) Previous chunks that have a semantic “total-to-part” relationship with the current chunk;
[0015] S104, splitting each large block into multiple small blocks;
[0016] Further, an upper limit and a lower limit of the number of characters in a small block are set, each large block is divided into small blocks in combination with the hierarchical structure information, and the number of characters in each small block is controlled between the upper limit and the lower limit of the number of characters in the small block;
[0017] S105, small block level semantic information completion: perform named entity recognition on the text content of the large block of the document, extract the domain important entities in the large block of the document, and add the parent title of the large block, the parent important entity, the large block title, and the important entities in the large block to the front of the small block to complete the important semantic information missing in the small block document;
[0018] S106, constructing dense vectors and sparse vectors of small blocks of text;
[0019] Furthermore, the dense vector is a semantic vector whose most feature values are non-zero values constructed by neural networks such as BERT, GPT, and autoencoders; the sparse vector refers to a semantic vector whose most feature values are zero obtained by traditional NLP methods such as bag-of-words model, TF-IDF, and sparse coding;
[0020] Furthermore, the dense and sparse vectors can utilize a multifunctional semantic vector model to obtain both dense and sparse vectors at one time through reasoning;
[0021] S107, construct small block metadata information, and write the dense and sparse vectors of the small block content and the small block metadata information into the vector database, wherein the small block metadata information includes: small block text content, small block ID, large block text content, large block ID, original document ID, and original document name.
[0022] S2, the problem after understanding and expanding the user's original problem;
[0023] Preferably, entity completion and sentence optimization are performed on user questions. The entity completion and sentence optimization utilize the multi-round question-answering understanding and text generation capabilities of the large language model to automatically complete key entities and convert spoken expressions into professional written standard expressions. Furthermore, if there are historical conversations, the large language model is used to integrate the historical rounds of question-answering records and the latest questions of the current user to obtain questions with complete semantic and grammatical structures and more professional question expressions.
[0024] Preferably, the obtained complete question is decomposed to obtain a series of sub-questions that can answer the question. Furthermore, the problem decomposition can split the user question into multiple basic sub-questions through the thinking chain capability of the large language model;
[0025] S3, encode the processed problem to obtain dense vectors and sparse vectors;
[0026] S4, using hybrid query to retrieve multiple document blocks and their metadata information from the vector database;
[0027] Preferably, the hybrid query is to query the dense vector column of the vector database by using the dense vector of the processed question, and query the sparse vector column of the vector database by using the sparse vector, and merge the results of the sparse search and the dense search by using the Reciprocal Rank Fusion (RRF) algorithm;
[0028] S5, using the reranker model to sort the retrieved document chunks;
[0029] Preferably, the similarity is calculated for the question and a plurality of document blocks that form a pair (a text pair consisting of the question and the document block content), sorted according to the similarity score, and the document blocks that do not meet the requirements are filtered according to a threshold; further, the similarity can directly obtain the relevance score of the question and the document through a sorting model such as BGE-Reranker, or obtain the vectors of the text and the document block respectively through a text vector model, and calculate the distance between the vectors;
[0030] S6, de-duplication and merging of the small document blocks sorted by relevance, including:
[0031] S601, traversing the sorted document blocks, if there is a document block whose metadata information has a block ID that is the same as the block ID in the metadata information of the first-ranked document block, then removing the document block with the same block ID that is the last-ranked document block;
[0032] S602, traversing the document blocks processed by S601, if the original document ID in the metadata information of the document block is the same as the original document ID in the metadata information of the block with a higher ranking, merging the document block with a lower ranking into the document block with the same document ID but a higher ranking, and removing the document block with the same document ID but a lower ranking;
[0033] S603, traversing the small document blocks processed in S602, and using the large text content in the metadata information of the small document blocks as the retrieval output document blocks after deduplication and merging processing;
[0034] S7, performing a relevance check on the retrieved output document block to obtain a final retrieval result;
[0035] Furthermore, the relevance check is to check the documents using a large language model: determine the relevance level of each document to the user's question, and filter out the document blocks that are irrelevant to the user's question as the final retrieval result;
[0036] Furthermore, the relevance levels include: 1) direct answer: the document directly answers the user's question; 2) partial answer: the document partially answers the user's question; 3) irrelevant: the document cannot answer the user's question and is irrelevant to the question.
[0037] S8, if the user is not in the first round of conversation, filter the historical conversations;
[0038] Furthermore, the screening process calculates the semantic similarity scores between the user question and the user question in historical rounds, as well as the semantic similarity scores between the user question and the system answer in historical rounds. If one of the two similarity scores is greater than a set threshold, the conversation in that round is retained; otherwise, the conversation in that round is discarded, thereby preventing negative impacts caused by user editing errors or meaningless conversation rounds.
[0039] S9, inserting the final search result into the conversation record;
[0040] Process the final search result content into a user record and insert it into the penultimate round of conversation, for example:
[0041] {"user":"The retrieved internal knowledge base documents are as follows:<retr ieved documents> "};
[0042] Add a new assistant (large language model) record, for example:
[0043] {"assistant":"I have received the internal knowledge base document. I will answer user questions based on the document. Please provide user questions."};
[0044] S10, sending the conversation record as context to the large language model to get an answer.
[0045] The beneficial effects of the present invention are as follows:
[0046] The present invention is designed to reconstruct the documents in the knowledge base and build a dense and sparse vector database as a basis to assist in understanding and expanding the original questions of customers and effectively handle the questions, retrieve multiple small document blocks and metadata information from the vector database using a hybrid query, sort the small document blocks using a re-ranking model, remove duplicates and merge the small document blocks sorted by relevance, and then perform a relevance check, judge and filter whether the user is in the first round of dialogue, insert the final search results into the dialogue record, send the dialogue record as a context to the large language model to obtain an answer;
[0047] Through the above methods, professional and complex problems in vertical fields can be effectively semantically split and hierarchically associated, which improves the retrieval success rate of question-related semantics, the precision and recall rate of original text citations during retrieval. At the same time, it also effectively avoids problems such as historical conversations mixed with useless information and document blocks that are too long to drown out the user's true intentions, and can achieve efficient multi-round question and answering of vertical field knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a schematic flow chart of the method provided by the present invention;
[0049] Figure 2 It is a schematic diagram of the document segmentation and information completion process;
[0050] Figure 3 It is a flow chart of entity completion, sentence optimization and problem decomposition;
[0051] Figure 4 This is a schematic diagram of the process of deduplication and merging of small document blocks;
[0052] Figure 5 It is a flowchart for inserting large blocks of document content. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] like Figures 1 to 5As shown, an embodiment of the present invention provides a vertical domain knowledge multi-round question-answering method based on RAG:
[0055] Reference Figure 1 Shown is a schematic flow chart of the method provided by the present invention:
[0056] In this embodiment, the method includes:
[0057] S1, reconstruct the documents in the knowledge base and build dense and sparse vector databases;
[0058] The detailed steps in step S1 are:
[0059] S101, extracting hierarchical structure information of a document, wherein the hierarchical structure information includes but is not limited to a document title, a chapter title, an ordered list, an unordered list, a table, a picture, a table title, and a picture title, wherein the extraction method includes manual extraction, automatic extraction by an algorithm, and automatic extraction by an algorithm plus manual review;
[0060] In an exemplary embodiment, a large-scale visual language model, such as Qwen-VL, is used to extract document titles, chapter titles, ordered lists, unordered lists, tables, pictures, table titles, and picture titles to obtain a document tree hierarchy. If the document is a PDF and the document content is in the form of a picture, the text content in the picture needs to be recognized and output by OCR technology, and the original document needs to be processed into text content in markdown format, where:
[0061] 1. Title processing, use different numbers of "#" to indicate the level of the title, such as "#" indicates the document title level, "##" indicates the chapter title level, "###" indicates the subordinate section title level, and so on;
[0062] 2. Table processing, using markdown table syntax to express;
[0063] 3. Image processing: if there is no description of the image in the document, a concise description of the image is generated based on the context in which the image is located, and the image is replaced by a data structure of [image: image number, <image description>];
[0064] S102, dividing the document into large blocks according to the hierarchical structure information;
[0065] Furthermore, an upper limit and a lower limit of the number of characters in a large block are set, the document is divided into large blocks in combination with the hierarchical structure information, and the number of characters in each large block is controlled between the upper limit and the lower limit of the number of characters in the large block;
[0066] In an exemplary embodiment, the lower limit and upper limit of the character length of a document chunk are set to 500 and 1200 respectively. When segmenting, firstly, the various chapters of the document are divided into various chunks according to the chapter structure of the document, and then the character length of each chunk is judged from the front to the back whether it is in the range of 500 to 1200. If it is, the chunk is retained and the next chunk is processed. If it is less than the lower limit of 500 characters, the current chunk is merged with the next chunk into the same chunk. If it is greater than the upper limit of 1200 characters, the current chunk is segmented into two chunks, so that the number of characters of the first chunk after segmentation is in the range of 500 to 1200, and then the second chunk after segmentation is judged and processed, and so on.
[0067] S103, completing the block hierarchical semantic information: according to the hierarchical structure information, obtaining the title and content of the upper level of the block, performing named entity recognition on the title and content of the upper level, extracting important entities in the vertical field, and adding the upper level title and upper level important entities to the front of the block;
[0068] Furthermore, the upper level of the large block includes at least one of the following:
[0069] 1) Document title; 2) Previous chunks that have a semantic “total-to-part” relationship with the current chunk;
[0070] S104, splitting each large block into multiple small blocks;
[0071] Further, an upper limit and a lower limit of the number of characters in a small block are set, each large block is divided into small blocks in combination with the hierarchical structure information, and the number of characters in each small block is controlled between the upper limit and the lower limit of the number of characters in the small block;
[0072] S105, small block level semantic information completion: perform named entity recognition on the text content of the large block of the document, extract the domain important entities in the large block of the document, and add the parent title of the large block, the parent important entity, the large block title, and the important entities in the large block to the front of the small block to complete the important semantic information missing in the small block document;
[0073] like Figure 2 This is a schematic diagram of the document segmentation and information completion process according to an embodiment of the present invention:
[0074] The name of the knowledge base document entered in the figure is "a.docx", and part of the document content is shown in the gray background text box below "a.docx" in the figure. Its title is "Nursing Routine for Third Ventricular Tumors" and contains chapters such as "Overview", "Causes", and "Diagnostic Points". Figure 2In step S101 shown in the figure, a tree-like hierarchical structure is extracted for the document, and the extracted hierarchical structure is shown in the white background box below S101; in step S102, the document is segmented into large blocks according to the hierarchical structure, and "a.docx" is split into multiple large blocks such as "a_1.docx", "a_2.docx", etc., where "a_1.docx" represents the first large block, "a_2.docx" represents the second large block, and so on. The text content of one of the large blocks is shown in the gray background text box below S102 in the figure; in step S103, the hierarchical semantic information of the large blocks is completed, and the partial text content of one of the large blocks after completion is shown in the gray background text box below S103 in the figure, where the completed upper-level title and important entities are shown in the gray background text box below S103. The entity is shown as a virtual box in the gray background text box; in step S104, the large block of the document is split into small blocks to obtain small blocks named "a_1_1.docx", "a_1_2.docx", ..., "a_n_m.docx", where "a_1_1.docx" represents the first small block of the first large block, "a_n_m.docx" represents the mth small block of the nth large block, and so on, and part of the text content of one of the small blocks is shown in the gray background text box below S104; in step S105, the hierarchical semantic information of the small blocks is completed, and part of the text content of one of the small blocks after completion is shown in the gray background text box below S105, and the completed upper-level title and important entity are shown in the virtual box in the gray background text box;
[0075] S106, constructing dense vectors and sparse vectors of small blocks of text;
[0076] Furthermore, the dense vector is a semantic vector whose most feature values are non-zero values constructed by neural networks such as BERT, GPT, and autoencoders; the sparse vector refers to a semantic vector whose most feature values are zero obtained by traditional NLP methods such as bag-of-words model, TF-IDF, and sparse coding;
[0077] Furthermore, the dense and sparse vectors can utilize a multifunctional semantic vector model to obtain both dense and sparse vectors at one time through reasoning;
[0078] S107, constructing small block metadata information, and writing the dense and sparse vectors of the small block content and the small block metadata information into a vector database, wherein the small block metadata information includes: small block text content, small block ID, large block text content, large block ID, original document ID, and original document name;
[0079] In an exemplary embodiment, a multifunctional vector model, such as BGE-M3, is used to encode semantic vectors of small document blocks, and dense semantic vectors and sparse semantic vectors can be obtained at the same time. A vector database, such as Milvus, is used to write the dense and sparse vectors of the small document blocks and metadata information of the small document blocks, namely, the text content of the small block, the ID of the small block, the text content of the large block, the ID of the large block, the ID of the original document, and the name of the original document into the vector database;
[0080] S2, the problem after understanding and expanding the user's original problem;
[0081] Among them, entity completion and sentence optimization are performed on user questions. The entity completion and sentence optimization utilize the multi-round question-answering understanding and text generation capabilities of the large language model to automatically complete key entities and convert spoken expressions into professional written standard expressions. Furthermore, if there are historical conversations, the large language model is used to integrate the historical rounds of question-answering records and the latest questions of the current user to obtain questions with complete semantic and grammatical structures and more professional question expressions.
[0082] The obtained complete question is decomposed to obtain a series of sub-questions that can answer the question. Furthermore, the problem decomposition can split the user question into multiple basic sub-questions through the thinking chain capability of the large language model;
[0083] Figure 3 It is a schematic diagram of entity completion, sentence optimization and problem decomposition process according to the present invention:
[0084] As shown in the figure, in one embodiment of the present invention, the first round of dialogue in the figure is a new conversation without historical dialogue, as shown in the first white background text box on the right side of "First Round" in the figure, the user's first round of questions is shown in the first white background text box below "User Question" in the figure, and the question is "Please introduce the third ventricle tumor?"; in entity completion and sentence optimization, the question is sentence optimized, and the optimized question is shown in the first white background box below "Entity Completion and Sentence Optimization"; in problem decomposition, the question is decomposed to obtain "no problem decomposition", as shown in the first white background box below "Problem Decomposition"; in the second round of dialogue in the figure, the historical dialogue content is shown in the first white background text box on the right side of "Second Round" in the figure, and its content includes the user's first question and the first answer of the dialogue system. , the first question is "Can you introduce the third ventricle tumor?", and the first answer is "The third ventricle tumor refers to the tumor that originates in the third ventricle or grows from outside the third ventricle into the third ventricle"; the second round of user questions is "What is the difference in treatment compared with the fourth ventricle tumor?", as shown in the second white background text box under "User Question" in the figure; in entity completion and sentence optimization, the question is entity completed and optimized, and the question after entity completion and sentence optimization is shown in the second white background text box under "Entity Completion and Sentence Optimization" in the figure, where the completed entity is shown in the gray background text box; in problem decomposition, the problem after entity completion and sentence optimization is decomposed, and the sub-questions obtained are shown in the second to fourth white background text boxes under "Problem Decomposition" in the figure;
[0085] In another embodiment of the present invention, a large language model is used to complete entity completion and sentence optimization, and the prompt is designed as follows: "You are a model that specializes in entity completion and sentence optimization for the user's latest question. When the user's question omits or refers to key information mentioned in the previous chat, your task is to complete the omitted or referred information and adjust the sentence to make the question professional and complete. If the user's question involves a coherent follow-up question and is very relevant to the previous chat questions and content, complete and optimize the user's question; if the question is clear enough, no important entity information is omitted or referred to, and there is no direct relevance to the previous chat content, then there is no need to complete and optimize.", and obtain a complete question after entity completion and sentence optimization;
[0086] In another embodiment of the present invention, a large language model is used to complete the problem decomposition, and the prompt is designed as follows: "You are a model that can automatically determine whether the user's problem needs to be decomposed into sub-problems. When encountering a complex problem, you need to determine whether it needs to be decomposed. If so, it should be reasonably disassembled into several basic sub-problems that can be processed in sequence. Each sub-problem should be concise and help to gradually solve the entire problem. After gradually solving these sub-problems, the answer to the user's question is obtained in a comprehensive manner. If the user's question does not need to be decomposed, the question is answered without decomposition.", and the processed question is obtained;
[0087] S3, encode the processed problem to obtain dense vectors and sparse vectors;
[0088] In an exemplary embodiment, a multifunctional vector model, such as BGE-M3, is used to encode the processed problem to obtain a dense vector and a sparse vector of the processed problem.
[0089] S4, using hybrid query to retrieve multiple document blocks and their metadata information from the vector database;
[0090] The hybrid query is to use the dense vector of the processed problem to query the dense vector column of the vector database, and use the sparse vector to query the sparse vector column of the vector database, and use the reciprocal rank fusion (RRF) algorithm to merge the results of the sparse search and the dense search;
[0091] In an exemplary embodiment, the hybrid search function of mi l vus is used, by using the dense vector of the processed problem to query the dense vector column of the vector database, and using the sparse vector to query the sparse vector column of the vector database, and using the reciprocal rank fusion (RRF, Reciprocal Rank Fusion) or weighted ranking algorithm (WeightedRanker) algorithm to merge the results of the sparse search and the dense search;
[0092] S5, using the reranker model to sort the retrieved document chunks;
[0093] The similarity is calculated for the question and multiple document blocks (text pairs consisting of question and document blocks), sorted according to the similarity score, and the document blocks that do not meet the requirements are filtered according to the threshold. Furthermore, the similarity can directly obtain the relevance score of the question and the document through a sorting model such as BGE-Reranker, or obtain the vectors of the text and document blocks through a text vector model, and calculate the distance between the vectors;
[0094] In an exemplary embodiment, a ranking model, such as BGE-RANKER, is used to rank the search results of step S4;
[0095] S6, de-duplication and merging of the small document blocks sorted by relevance, including:
[0096] S601, traversing the sorted document blocks, if there is a document block whose metadata information has a block ID that is the same as the block ID in the metadata information of the first-ranked document block, then removing the document block with the same block ID that is the last-ranked document block;
[0097] S602, traversing the document blocks processed by S601, if the original document ID in the metadata information of the document block is the same as the original document ID in the metadata information of the block with a higher ranking, merging the document block with a lower ranking into the document block with the same document ID but a higher ranking, and removing the document block with the same document ID but a lower ranking;
[0098] S603, traversing the small document blocks processed in S602, and using the large text content in the metadata information of the small document blocks as the retrieval output document blocks after deduplication and merging processing;
[0099] Figure 4 The following is a schematic diagram of the small block deduplication and merging process according to the present invention, as shown in the figure:
[0100] In one embodiment of the present invention, the document blocks input in the figure are "a_1_1", "a_1_2", "b_1_1", and "a_2_1", as shown in the four white background text boxes in the second column from the left in the figure, and the metadata information of each document block is shown in the four white background text boxes in the first column from the left in the figure, where the metadata information of a_1_1 includes small block text content, small block ID, large block text content, large block ID, and original document ID, which correspond to "<document small block content>", "a_1_1", "<document large block content>", "a_1", and "a" in the figure respectively; in the deduplication of the figure, the input document blocks are deduplicated, and the deduplicated document blocks are shown in the three white background text boxes in the third column from the left in the figure, where "document block a_1_2" and "document block a_2" are not identical. "Block a_1_1" has a duplicated large block ID ("a_1") and is removed; in the backward merging in the figure, the document blocks are merged backward, and the large block text content in the metadata information of the document block is used as the retrieval output document block after deduplication and merging, as shown in the two white background text boxes in the rightmost column in the figure, where "document block a_2_1" has the same original document ID as "document block a_1_1" and is merged, and "large block a_1" and "large block a_2" are used as output document blocks, as shown in the white background text box in the upper right corner of the figure, the dotted box indicates that "document block a_2_1" is merged with "document block a_1_1", and "document large block b_1" is used as the output document block of "document block b_1_1", as shown in the white background text box in the lower rightmost column in the figure;
[0101] S7, performing a relevance check on the retrieved output document block to obtain a final retrieval result;
[0102] Furthermore, the relevance check is to check the documents using a large language model: determine the relevance level of each document to the user's question, and filter out the document blocks that are irrelevant to the user's question as the final retrieval result;
[0103] Furthermore, the relevance levels include: 1) direct answer: the document directly answers the user's question; 2) partial answer: the document partially answers the user's question; 3) irrelevant: the document cannot answer the user's question and is irrelevant to the question.
[0104] In an exemplary embodiment, a large language model is used to complete the document block relevance check, and the prompt is designed as follows: "You are an expert assistant who helps users analyze the relevance of text paragraphs. Please judge the relevance based on the user's question and the provided text paragraph. The following are the relevance criteria: - **Direct answer**: The document directly answers the user's question, that is, the document contains specific requirements or information clearly mentioned in the user's question. - **Partial answer**: The document is somewhat related to the user's question, but not completely answered, that is, the document mentions related topics, but not all the specific information mentioned in the user's question. - **Irrelevant**: The document is irrelevant to the user's question. Please be sure to judge according to the above criteria.", the relevance of the document block and the question is obtained, and the document blocks with direct answers and partial answers are selected, and irrelevant document blocks are removed. The relevance is sorted, and the document blocks with direct answers are ranked before the partial answers to obtain the final retrieval results;
[0105] S8, if the user is not in the first round of conversation, filter the historical conversations;
[0106] Furthermore, the screening process calculates the semantic similarity scores between the user question and the user question in historical rounds, as well as the semantic similarity scores between the user question and the system answer in historical rounds. If one of the two similarity scores is greater than a set threshold, the conversation in that round is retained; otherwise, the conversation in that round is discarded, thereby preventing negative impacts caused by user editing errors or meaningless conversation rounds.
[0107] In an exemplary embodiment, a hybrid semantic vector model (such as BGE-M3) is used to semantically encode user questions to obtain dense semantic vectors, sparse semantic vectors, and multi-vector semantic vectors (using multiple vectors to represent text, such as Col BERT); each round of user questions in the historical dialogue is obtained, and the hybrid semantic vector model is used to obtain dense semantic vectors, sparse semantic vectors, and multi-vector semantic vectors of the user questions; each round of answers of the system in the historical dialogue is obtained, and the dense semantic vectors, sparse semantic vectors, and multi-vector semantic vectors of the system answers are obtained; the weights of the dense semantic vectors, sparse semantic vectors, and multi-vector semantic vectors are set, such as [0.4, 0.2, 0.4], and the final semantic similarity score is:
[0108] Dense semantic similarity × 0.4 + sparse semantic similarity × 0.2 + multi-vector semantic similarity × 0.4;
[0109] Calculate the semantic similarity score between the user's question and the user's question in the previous round, or the semantic similarity score between the user's question and the system's answer in the previous round. If one of the two similarity scores is greater than the set threshold, the conversation in that round is retained. Otherwise, the conversation in that round is discarded. This prevents the negative impact caused by user editing errors or meaningless conversation rounds.
[0110] S9, inserting the final search result into the conversation record;
[0111] Process the final search result content into a user record and insert it into the penultimate round of conversation, for example:
[0112] {"user":"The retrieved internal knowledge base documents are as follows:<retr ieved documents> "};
[0113] Add a new assistant (large language model) record, for example:
[0114] {"assistant":"I have received the internal knowledge base document. I will answer user questions based on the document. Please provide user questions."};
[0115] Figure 5 The figure is a schematic diagram of the process of inserting a large block of document content according to the present invention, as shown in the figure:
[0116] In one embodiment of the present invention, the historical conversations input in the figure are the records of the previous two rounds of conversations of the user, as shown in the text content of the corresponding area on the right side of "Historical Conversation" in the figure; the inserted content generated according to step S9 is shown as the text content of the corresponding area on the right side of "Insert Content" in the figure, where the "<retr ieved documents> " indicates the final search result; new questions raised by users are placed at the end of the conversation.
[0117] S10, sending the conversation record as context to the large language model to get an answer.
[0118] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0119] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A vertical domain knowledge multi-round question-answering method based on RAG, characterized by: The following steps are involved: S1, reconstruct the documents in the knowledge base and build dense and sparse vector databases; S2, the problem after understanding and expanding the user's original problem; S3, encode the processed problem to obtain dense vectors and sparse vectors; S4, using hybrid query to retrieve multiple document blocks and their metadata information from the vector database; S5, uses the re-ranking model to sort the retrieved document chunks; S6, de-duplication and merging of the small document blocks sorted by relevance, including: S601, traversing the sorted document blocks, if there is a document block whose metadata information has a block ID that is the same as the block ID in the metadata information of the first-ranked document block, then removing the document block with the same block ID that is the last-ranked document block; S602, traversing the document blocks processed by S601, if the original document ID in the metadata information of the document block is the same as the original document ID in the metadata information of the block with a higher ranking, merging the document block with a lower ranking into the document block with the same document ID but a higher ranking, and removing the document block with the same document ID but a lower ranking; S603, traversing the small document blocks processed in S602, and using the large text content in the metadata information of the small document blocks as the retrieval output document blocks after deduplication and merging processing; S7, performing a relevance check on the retrieved output document block to obtain a final retrieval result; S8, if the user is not in the first round of conversation, filter the historical conversations; S9, inserting the final search result into the conversation record, processing the final search result content into a user record, and inserting it into the penultimate round of conversation; S10, sending the conversation record as context to the large language model to get an answer.
2. According to claim 1, a vertical domain knowledge multi-round question-answering method based on RAG is characterized by: The detailed steps in step S1 are: S101, extracting hierarchical structure information of a document, wherein the hierarchical structure information includes but is not limited to a document title, a chapter title, an ordered list, an unordered list, a table, a picture, a table title, and a picture title; The extraction methods include manual extraction, algorithm-automatic extraction, and algorithm-automatic extraction plus manual review; S102, dividing the document into large blocks according to the hierarchical structure information: Setting an upper limit and a lower limit of the number of characters in a large block, dividing the document into large blocks in combination with the hierarchical structure information, and controlling the number of characters in each large block to be between the upper limit and the lower limit of the number of characters in the large block; S103, completing the block hierarchical semantic information: according to the hierarchical structure information, obtaining the title and content of the upper level of the block, performing named entity recognition on the title and content of the upper level, extracting important entities in the vertical field, and adding the upper level title and upper level important entities to the front of the block; The upper level of the block includes at least one of the following: 1) Document title; 2) Previous chunks that have a semantic "total-to-part" relationship with the current chunk; S104, split each large block into multiple small blocks: Setting an upper limit and a lower limit of the number of characters in a small block, dividing each large block into small blocks in combination with the hierarchical structure information, and controlling the number of characters in each small block to be between the upper limit and the lower limit of the number of characters in the small block; S105, small block level semantic information completion: perform named entity recognition on the text content of the large block of the document, extract the domain-important entities in the large block of the document, and add the parent title of the large block, the parent important entity, the large block title, and the important entities in the large block to the front of the small block to complete the important semantic information missing in the small block document; S106, construct dense vectors and sparse vectors for small blocks of text: The dense vector is a semantic vector whose most feature values are non-zero values constructed by neural networks such as BERT, GPT, and autoencoders; the sparse vector refers to a semantic vector whose most feature values are zero obtained by traditional NLP methods such as bag-of-words model, TF-IDF, and sparse coding; The dense and sparse vectors utilize a multifunctional semantic vector model to obtain both dense and sparse vectors in one inference; S107, construct small block metadata information, and write the dense and sparse vectors of the small block content and the small block metadata information into the vector database, wherein the small block metadata information includes: small block text content, small block ID, large block text content, large block ID, original document ID, and original document name.
3. The RAG-based vertical domain knowledge multi-round question-answering method according to claim 1, characterized in that: The details of step S2 are: Perform entity completion and sentence optimization on user questions. The entity completion and sentence optimization utilize the multi-round question-answering understanding and text generation capabilities of the large language model to automatically complete key entities and convert spoken expressions into professional written standard expressions. If there are historical conversations, the large language model is used to integrate historical rounds of question-answering records and the latest questions of the current user to obtain questions with complete semantic and grammatical structures and more professional question expressions. Preferably, the obtained complete question is decomposed to obtain a series of sub-questions that can answer the question. The problem decomposition can split the user question into multiple basic sub-questions through the thinking chain capability of the large language model.
4. The RAG-based vertical domain knowledge multi-round question-answering method according to claim 1, characterized in that: The hybrid query in step S4 is to query the dense vector column of the vector database using the dense vector of the processed problem, and query the sparse vector column of the vector database using the sparse vector, and merge the results of the sparse search and the dense search using the inverse ranking fusion algorithm.
5. The RAG-based vertical domain knowledge multi-round question-answering method according to claim 1, characterized in that: The specific content of step S5 is: The similarity is calculated for pairs consisting of questions and multiple document blocks (text pairs consisting of questions and document block contents), sorted according to the similarity scores, and document blocks that do not meet the requirements are filtered according to a threshold. The similarity can directly obtain the relevance score of the question and the document through the sorting model BGE-Reranker, or obtain the vectors of the text and document blocks respectively through the text vector model, and calculate the distance between the vectors.
6. The RAG-based vertical domain knowledge multi-round question-answering method according to claim 1, characterized in that: The relevance check in step S7 is to check the documents using a large language model: determine the relevance level of each document to the user's question, and filter out the document blocks that are irrelevant to the user's question as the final search result. The relevance levels include: 1) direct answer: the document directly answers the user question; 2) partial answer: the document partially answers the user question; 3) irrelevant: the document cannot answer the user question and is irrelevant to the question.
7. The RAG-based vertical domain knowledge multi-round question-answering method according to claim 1, characterized in that: The screening process in step S8 is to calculate the semantic similarity scores between the user question and the user question in historical rounds, as well as the semantic similarity scores between the user question and the system answer in historical rounds. If one of the two similarity scores is greater than a set threshold, the conversation in that round is retained; otherwise, the conversation in that round is discarded, thereby preventing negative impacts caused by user editing errors or meaningless conversation rounds.
Citation Information
Patent Citations
Conversation interaction method and device and electronic equipment
CN117421400A
Traditional Chinese medicine question and answer method and device based on long document retrieval enhancement generation and medium
CN117828050A
Context window-based large language model construction method and system
CN117932038A
Conversational knowledge base
US20240070434A1
Cited By
Automobile intelligent question-answering system and method based on large model and retrieval enhancement
CN119202206A
Large language model RAG optimization method based on tree neighbor context
CN119293195A
Recognition method for multi-round multi-intention mixed dialogue and electronic equipment
CN119396982A
Identification method and electronic device for multi-round multi-intention mixed dialogue
CN119396982B
Knowledge graph and vector retrieval enhancement-based photovoltaic field large model efficiency improvement method
CN119988639A
