A multi-level knowledge base construction method for RAGs
By constructing a multi-level knowledge base, the problems of semantic ambiguity and lack of context caused by unreasonable text segmentation in the RAG system are solved, achieving more efficient and accurate information retrieval and answer generation, and enhancing the stability and adaptability of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
- Filing Date
- 2025-07-11
- Publication Date
- 2026-07-10
AI Technical Summary
Existing RAG systems suffer from semantic ambiguity, low recall accuracy, missing context, and loss of topic information when the text block length is unreasonable, which affects the quality of answers generated by large models.
A multi-level knowledge base construction method is adopted. Documents are segmented and incorporated into different levels of knowledge bases through a recursive character text segmenter. Vectorized retrieval is performed in combination with the BERT model. Similarity is calculated at each level, thresholds are configured and optimized to improve retrieval accuracy, and data is updated and synchronized.
It improves retrieval efficiency and system adaptability, generates more accurate and complete answers, ensures data consistency, and enhances the performance and user satisfaction of the RAG system.
Smart Images
Figure CN120874989B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and artificial intelligence, and in particular to a method for constructing a multi-level knowledge base for RAG. Background Technology
[0002] In the field of natural language processing, RAG (Retrieval-Augmented Generation) systems have been widely used in recent years. They combine the generative capabilities of language models with the retrieval capabilities of external knowledge, using relevant knowledge retrieval to assist large models in generating high-quality answers. However, text segmentation is a crucial step in building a RAG system.
[0003] When text blocks are too long, semantic ambiguity issues arise. During vectorization, compressing a large amount of lexical information into fixed-length vectors can lead to the averaging or dilution of detailed semantic information, making it difficult for the generated vectors to accurately represent the core content of the text. Furthermore, during the retrieval phase, because long text blocks cover multiple topics and have complex semantics, they struggle to accurately match user query intent after vectorization, resulting in decreased relevance of the retrieved text and impacting the quality of answers generated by large models.
[0004] On the other hand, excessively short text chunks also present problems. Short text chunks lack necessary contextual information, making it difficult for the model to accurately understand the text's meaning, resulting in incomplete or off-topic responses. Furthermore, paragraph or chapter-level thematic information requires a certain text length to express; excessively short text chunks may only contain fragmented information, failing to fully convey the main points and affecting the model's grasp of the overall content. Therefore, a reasonable text chunking and knowledge base construction strategy is needed to improve the performance of the RAG system and the quality of responses from large models.
[0005] To address a series of problems caused by unreasonable text block lengths during the construction of RAG systems, including semantic ambiguity, low recall accuracy, missing context, and loss of topic information, this invention proposes a method for constructing a multi-level knowledge base for RAG. Summary of the Invention
[0006] To overcome the shortcomings of existing technologies, this invention provides a simple and efficient method for constructing a multi-level knowledge base for RAG.
[0007] This invention is achieved through the following technical solution:
[0008] A method for constructing a multi-level knowledge base for RAG includes the following steps:
[0009] Step S1: Document Segmentation and Storage
[0010] Step S1.1: Document Segmentation
[0011] The document is processed recursively at the character level, and the initial segmentation is performed based on the pre-set initial segment length.
[0012] In step S3.1, the recursive character text splitter segments the document.
[0013] Step S1.2, Graded warehousing
[0014] After the initial segmentation is completed, the first-level fragments are stored in the first-level knowledge base; then, the first-level fragments are merged, and the length of the merged fragments is doubled to obtain the second-level fragments, which are then stored in the second-level knowledge base.
[0015] And so on, each time the fragment length doubles, the new fragment is stored in the knowledge base of the corresponding level, until it can no longer be merged or all levels of knowledge base are stored.
[0016] Step S2, Information Retrieval
[0017] Step S2.1, Initial Search
[0018] When a user enters a query question, it is vectorized into a query vector using the BERT model. First, a search is performed in the primary knowledge base. Using vector retrieval technology, the similarity between the query vector and all fragment vectors in the primary knowledge base is calculated, and they are sorted from high to low similarity. The fragments with the highest similarity to the query vector are then identified, and the relevant files containing the knowledge related to the query question are determined.
[0019] Step S2.2, step-by-step retrieval
[0020] After identifying the relevant files, the corresponding fragments of the relevant files are retrieved in the secondary knowledge base. Vector retrieval technology is then used to calculate the similarity between the query vector and the corresponding fragment vector, and the fragments are sorted from high to low similarity to find the fragments with the highest similarity to the query vector. This helps to determine the relevant files containing the knowledge related to the query question.
[0021] The search and calculation are performed on each level of the support library until the distance between the fragment vector in the x-level knowledge base and the query vector exceeds a custom threshold. Then the search and calculation are stopped, and the fragment found in the x-1 level knowledge base is determined to be the largest segment with the most semantic relevance.
[0022] In step S2, cosine similarity is used to calculate the similarity between the query vector and all fragment vectors in each level of the knowledge base.
[0023] Step S3: Threshold Configuration and Optimization
[0024] Step S3.1: Configure threshold
[0025] Users can customize the vector distance threshold for retrieval based on their actual needs and data characteristics.
[0026] In step S3.1, the higher the accuracy requirement, the smaller the configured threshold value.
[0027] Step S3.2, Dynamic Optimization
[0028] The threshold is dynamically adjusted based on search results and user feedback. User feedback includes whether the recalled fragments are complete and whether the recalled fragments are highly relevant.
[0029] If users report that the recalled fragments are incomplete, increase the threshold.
[0030] If users report that the retrieved fragments are not highly relevant after retrieval, then lower the threshold.
[0031] In step S3.2, if users frequently request supplementary information for the answer, the recalled fragment is considered incomplete, and the threshold is increased.
[0032] Step S4: Data Update and Synchronization
[0033] Step S4.1, Incremental Update
[0034] When a new document is added, it is processed according to the sharding and storage rules, and the shards are stored in the knowledge base at the corresponding level.
[0035] Step S4.2, Data Synchronization
[0036] After modifying or deleting a document, the relevant fragments in each level of the knowledge base are updated synchronously to ensure data consistency and avoid errors in search results due to data inconsistency.
[0037] A multi-level knowledge base construction apparatus for RAG, used to implement the above method, includes:
[0038] The document segmentation and storage module is responsible for segmenting documents and storing the segments into knowledge bases at different levels.
[0039] The retrieval module is responsible for searching and calculating the support base at each level, starting from the smallest fragment knowledge base, based on the query input by the user. This process continues until the distance between the fragment vector in the x-level knowledge base and the query vector exceeds a custom threshold. At this point, the retrieval and calculation stop, and the fragment found in the x-1 level knowledge base is determined to be the most semantically relevant largest segment.
[0040] The threshold configuration and optimization module is responsible for customizing the vector distance threshold during retrieval based on the user's actual needs and data characteristics, and dynamically adjusting the threshold based on retrieval results and user feedback.
[0041] The data update and synchronization module is responsible for processing new documents according to the sharding and database entry rules when new documents are added, and storing the shards in the corresponding level of the knowledge base. At the same time, after a document is modified or deleted, the module synchronously updates the relevant shards in the knowledge base at each level to ensure data consistency in the knowledge base and avoid errors in search results due to data inconsistency.
[0042] A multi-level knowledge base construction device for RAG, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-described method steps when executing the computer program.
[0043] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program, when executed by a processor, implements the above-described method steps.
[0044] The beneficial effects of this invention are as follows: This multi-level knowledge base construction method for RAG improves retrieval efficiency, enhances system adaptability, stability and reliability, and enables large models to generate more accurate, complete and relevant answers, meeting the needs of users in various natural language processing tasks. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Appendix Figure 1 This is a schematic diagram of the document fragmentation and database storage method of the present invention.
[0047] Appendix Figure 2 This is a schematic diagram of the primary retrieval method for a multi-level knowledge base according to the present invention.
[0048] Appendix Figure 3 This is a schematic diagram of the multi-level knowledge base retrieval method of the present invention. Detailed Implementation
[0049] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0050] The method for constructing a multi-level knowledge base for RAG includes the following steps:
[0051] Step S1: Document Segmentation and Storage
[0052] Step S1.1: Document Segmentation
[0053] The document is processed recursively at the character level, and the first segment is performed based on the pre-set initial segment length (e.g., 100 characters).
[0054] For example, for a long document, it can be divided into first-level segments by dividing it into 100-word segments starting from the beginning of the document.
[0055] In step S3.1, the recursive character text splitter segments the document.
[0056] Step S1.2, Graded warehousing
[0057] After the initial segmentation is completed, the first-level segments are stored in the first-level knowledge base; then, the first-level segments are merged, and the length of the merged segments is doubled (to 200 characters) to obtain the second-level segments, which are then stored in the second-level knowledge base.
[0058] And so on, each time the fragment length doubles, the new fragment is stored in the knowledge base of the corresponding level, until it can no longer be merged (the number of fragments is 1) or all levels of knowledge base are stored.
[0059] The length of the fragments stored in different levels of the knowledge base varies, with a maximum of m-level knowledge bases. For example, a document containing 1000 words, after the above process, might have 10 fragments of 100 words each in the first-level knowledge base, 5 fragments of 200 words each in the second-level knowledge base, and so on.
[0060] Step S2, Information Retrieval
[0061] Step S2.1, Initial Search
[0062] When a user enters a query question, it is vectorized into a query vector using the BERT model. First, a search is performed in the first-level knowledge base (the knowledge base with the smallest fragment). Using vector retrieval technology, the similarity between the query vector and all fragment vectors in the first-level knowledge base is calculated, and they are sorted from high to low similarity to find the fragments with the highest similarity to the query vector. Then, the relevant files containing the knowledge related to the query question are determined.
[0063] By identifying the fragments with the highest similarity to the query vector, we can determine the documents containing the relevant knowledge. For example, if a user queries "the nutritional value of apples," the fragments retrieved in the primary knowledge base might come from a document about fruit nutrition.
[0064] Step S2.2, step-by-step retrieval
[0065] After identifying the relevant files, the corresponding fragments are retrieved from the secondary knowledge base. Vector retrieval techniques are then used to calculate the similarity between the query vector and the corresponding fragment vectors. As the knowledge base level increases, the fragment length increases, and the contextual information contained becomes richer. The fragments are then sorted from highest to lowest similarity to identify the fragments with the highest similarity to the query vector, thereby determining the relevant files containing the knowledge related to the query question.
[0066] The search and calculation are performed on each level of the support library until the distance between the fragment vector in the x-level knowledge base and the query vector exceeds a custom threshold (e.g., the cosine similarity threshold is set to 0.8). Then the search and calculation are stopped, and the fragment found in the x-1 level knowledge base is determined to be the largest segment with the most semantic relevance.
[0067] For example, if the similarity of a segment found in the first-level knowledge base is 0.9, in the second-level knowledge base it is 0.85, and in the third-level knowledge base it is 0.79 (exceeding the threshold), then the segment found in the second-level knowledge base is determined to be the semantically most relevant largest segment. This ensures the completeness and coherence of the recalled knowledge, providing more comprehensive and accurate information for the large model to generate answers.
[0068] In step S2, cosine similarity is used to calculate the similarity between the query vector and all fragment vectors in each level of the knowledge base.
[0069] Step S3: Threshold Configuration and Optimization
[0070] Step S3.1: Configure threshold
[0071] Users can customize the vector distance threshold for retrieval based on their actual needs and data characteristics.
[0072] In step S3.1, the higher the accuracy requirement, the smaller the configured threshold value.
[0073] For scenarios where high accuracy is required, a smaller threshold can be set to ensure that the recalled text fragments are highly relevant to the query; for scenarios that require obtaining broader information, the threshold can be appropriately increased.
[0074] For example, in a medical knowledge Q&A scenario, the threshold can be set to 0.85 to ensure the accuracy of the answers; in a general knowledge popularization scenario, the threshold can be set to 0.8.
[0075] Step S3.2, Dynamic Optimization
[0076] The threshold is dynamically adjusted based on search results and user feedback. User feedback includes whether the recalled fragments are complete and whether the recalled fragments are highly relevant.
[0077] If the user reports that the retrieved fragment is incomplete, the threshold will be increased (the higher the threshold, the higher the level of knowledge base will be searched, the longer the fragment will be, and the more comprehensive the information will be).
[0078] If users report that the retrieved fragments are not highly relevant after retrieval, then lower the threshold.
[0079] In step S3.2, if users frequently request supplementary information for the answer, the recalled fragment is considered incomplete, and the threshold is increased.
[0080] Step S4: Data Update and Synchronization
[0081] Step S4.1, Incremental Update
[0082] When a new document is added, it is processed according to the sharding and storage rules, and the shards are stored in the knowledge base at the corresponding level.
[0083] For example, when a new document about the development trend of artificial intelligence is uploaded, it is segmented and stored in different levels of knowledge bases to ensure the real-time nature and completeness of the knowledge base.
[0084] Step S4.2, Data Synchronization
[0085] After modifying or deleting a document, the relevant fragments in each level of the knowledge base are updated synchronously to ensure data consistency and avoid errors in search results due to data inconsistency.
[0086] For example, if part of the content in a technical document is modified, the system will automatically update the corresponding fragments of that document in the knowledge base at all levels to ensure that the retrieved information is up-to-date.
[0087] The multi-level knowledge base construction apparatus for RAG, used to implement the above method, includes:
[0088] The document segmentation and storage module is responsible for segmenting documents and storing the segments into knowledge bases at different levels.
[0089] The retrieval module is responsible for searching and calculating the support base at each level, starting from the smallest fragment knowledge base, based on the query input by the user. This process continues until the distance between the fragment vector in the x-level knowledge base and the query vector exceeds a custom threshold. At this point, the retrieval and calculation stop, and the fragment found in the x-1 level knowledge base is determined to be the most semantically relevant largest segment.
[0090] The threshold configuration and optimization module is responsible for customizing the vector distance threshold during retrieval based on the user's actual needs and data characteristics, and dynamically adjusting the threshold based on retrieval results and user feedback.
[0091] The data update and synchronization module is responsible for processing new documents according to the sharding and database entry rules when new documents are added, and storing the shards in the corresponding level of the knowledge base. At the same time, after a document is modified or deleted, the module synchronously updates the relevant shards in the knowledge base at each level to ensure data consistency in the knowledge base and avoid errors in search results due to data inconsistency.
[0092] The multi-level knowledge base construction device for RAG includes a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method steps.
[0093] The readable storage medium stores a computer program that, when executed by a processor, implements the above-described method steps.
[0094] Compared with existing technologies, this method for constructing a multi-level knowledge base for RAG has the following characteristics:
[0095] First, it improved the quality of responses: Through a reasonable segmentation strategy and hierarchical retrieval method, it effectively solved problems such as semantic ambiguity and missing context, enabling the large model to acquire more accurate and complete knowledge, and significantly improving the quality of the generated responses. For example, in knowledge question answering tasks, the accuracy and completeness of the responses were greatly improved.
[0096] Secondly, it improves retrieval efficiency: the initial search starts from the smallest knowledge base segment, reducing the search scope and speeding up the search. During the progressive retrieval process, as the segment length increases, the number of searches decreases, resulting in improved overall retrieval efficiency. For example, when processing a large number of documents, the retrieval time is significantly shortened compared to traditional retrieval methods.
[0097] Third, it enhances system adaptability: Threshold configuration and dynamic optimization functions allow users to flexibly adjust the system according to different scenarios and needs, improving the system's adaptability and flexibility. Whether in highly specialized fields or general knowledge processing scenarios, good results can be achieved through reasonable threshold configuration.
[0098] Fourth, it ensures data consistency: The data update and synchronization module ensures data consistency at all levels of the knowledge base, avoids retrieval errors caused by data inconsistency, and improves the stability and reliability of the system.
[0099] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for constructing a multi-level knowledge base for RAG, characterized in that: Includes the following steps: Step S1: Document Segmentation and Storage Step S1.1: Document Segmentation The document is processed recursively at the character level, and the initial segmentation is performed based on the pre-set initial segment length. Step S1.2, Graded warehousing After the initial segmentation is completed, the first-level fragments are stored in the first-level knowledge base; then, the first-level fragments are merged, and the length of the merged fragments is doubled to obtain the second-level fragments, which are then stored in the second-level knowledge base. And so on, each time the fragment length doubles, the new fragment is stored in the knowledge base of the corresponding level, until it can no longer be merged or all levels of knowledge base are stored. Step S2, Information Retrieval Step S2.1, Initial Search When a user enters a query question, it is vectorized into a query vector by the BERT model and then searched in the first-level knowledge base. Using vector retrieval technology, the similarity between the query vector and all fragment vectors in the primary knowledge base is calculated, and the fragments are sorted from high to low similarity to find the fragments with the highest similarity to the query vector, thereby determining the relevant files where the knowledge related to the query question is located. Step S2.2, step-by-step retrieval After identifying the relevant files, the corresponding fragments of the relevant files are retrieved in the secondary knowledge base. Vector retrieval technology is then used to calculate the similarity between the query vector and the corresponding fragment vector, and the fragments are sorted from high to low similarity to find the fragments with the highest similarity to the query vector. This helps to determine the relevant files containing the knowledge related to the query question. The search and calculation are performed on each level of the support library until the distance between the fragment vector in the x-level knowledge base and the query vector exceeds a custom threshold. Then the search and calculation are stopped, and the fragment found in the x-1 level knowledge base is determined to be the largest segment with the most semantic relevance. Step S3: Threshold Configuration and Optimization Step S3.1: Configure threshold Users can customize the vector distance threshold for retrieval based on their actual needs and data characteristics. Step S3.2, Dynamic Optimization The threshold is dynamically adjusted based on search results and user feedback. User feedback includes whether the recalled fragments are complete and whether the recalled fragments are highly relevant. If users report that the recalled fragments are incomplete, increase the threshold. If users report that the retrieved fragments are not highly relevant after retrieval, then lower the threshold. Step S4: Data Update and Synchronization Step S4.1, Incremental Update When a new document is added, it is processed according to the sharding and storage rules, and the shards are stored in the knowledge base at the corresponding level. Step S4.2, Data Synchronization After modifying or deleting a document, the relevant fragments in each level of the knowledge base are updated synchronously to ensure data consistency and avoid errors in search results due to data inconsistency.
2. The method for constructing a multi-level knowledge base for RAG according to claim 1, characterized in that: In step S3.1, the recursive character text splitter segments the document.
3. The method for constructing a multi-level knowledge base for RAG according to claim 1, characterized in that: In step S2, cosine similarity is used to calculate the similarity between the query vector and all fragment vectors in each level of the knowledge base.
4. The method for constructing a multi-level knowledge base for RAG according to claim 3, characterized in that: In step S3.1, the higher the accuracy requirement, the smaller the configured threshold value.
5. The method for constructing a multi-level knowledge base for RAG according to claim 1, characterized in that: In step S3.2, if users frequently request supplementary information for the answer, the recalled fragment is considered incomplete, and the threshold is increased.
6. A multi-level knowledge base construction device for RAG, characterized in that: To implement the method according to any one of claims 1 to 5, comprising: The document segmentation and storage module is responsible for segmenting documents and storing the segments into knowledge bases at different levels. The retrieval module is responsible for searching and calculating the support base at each level, starting from the smallest fragment knowledge base, based on the query input by the user. This process continues until the distance between the fragment vector in the x-level knowledge base and the query vector exceeds a custom threshold. At this point, the retrieval and calculation stop, and the fragment found in the x-1 level knowledge base is determined to be the most semantically relevant largest segment. The threshold configuration and optimization module is responsible for customizing the vector distance threshold during retrieval based on the user's actual needs and data characteristics, and dynamically adjusting the threshold based on retrieval results and user feedback. The data update and synchronization module is responsible for processing new documents according to the sharding and database entry rules when new documents are added, and storing the shards in the corresponding level of the knowledge base. At the same time, after a document is modified or deleted, the module synchronously updates the relevant shards in the knowledge base at each level to ensure data consistency in the knowledge base and avoid errors in search results due to data inconsistency.
7. A device for constructing a multi-level knowledge base for RAG, characterized in that: It includes a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method as described in any one of claims 1 to 5.
8. A readable storage medium, characterized in that: The readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.