Rag knowledge retrieval method based on b+ tree structure

Abstract

This invention discloses a RAG knowledge retrieval method based on a B+ tree structure, comprising: converting user queries into low-dimensional vectors obtained through principal component analysis; performing range queries using a pre-built B+ tree index to retrieve similar knowledge fragments, wherein the B+ tree index uses the low-dimensional vectors of knowledge documents as keys; and generating a final answer by combining the retrieval results. By constructing a hierarchical semantic index and a dynamic update mechanism, efficient knowledge retrieval, dynamic updates, and comprehensive optimization of storage costs are achieved.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and information retrieval technology, and in particular to a RAG knowledge retrieval method based on a B+ tree structure. Background Technology

[0002] Amidst the booming development of artificial intelligence, RAG technology, with its innovative architecture that deeply couples retrieval systems with generative models, has demonstrated significant advantages in knowledge-intensive tasks. By dynamically retrieving external knowledge bases to supplement the generative context, RAG effectively alleviates the "illusion" problem of traditional large models, achieving more factually grounded outputs in scenarios such as question-answering systems and content creation. How to efficiently retrieve external knowledge bases has become a research hotspot.

[0003] Currently, research on knowledge retrieval methods in RAG mainly falls into two categories:

[0004] Similarity retrieval transforms text into high-dimensional vectors, calculates the similarity score between the query vector and all stored vectors, and returns the records with the highest scores. Common similarity calculation methods include cosine similarity, Euclidean distance, and Manhattan distance. However, building vector database indexes is time-consuming, incremental insertions may trigger global index reconstruction, and the complexity of vector similarity calculation increases exponentially with dimensionality, making high-dimensional vector storage costly.

[0005] In full-text search, the inverted index is the core data structure, establishing a mapping relationship between "terms → document lists". When building an inverted index, the document is first segmented into individual terms. Then, for each term, a list of documents containing that term and the term's position within the document is recorded. For example, for the document set {D1:"Helloworld",D2:"HelloPython"}, the inverted index might be {Hello:[D1,D2],world:[D1],Python:[D2]}. During a query, the inverted index can quickly locate documents containing the query term. However, inverted indexes struggle with semantic similarity retrieval and lack hierarchical knowledge organization capabilities. Summary of the Invention

[0006] To address the shortcomings and deficiencies of existing technologies, this invention provides a RAG knowledge retrieval method and system based on a B+ tree hierarchical semantic index, whose innovative design includes:

[0007] Low-dimensional semantic index construction mechanism: By fusing TF-IDF statistical features with semantic vectors from pre-trained models (such as Sentence-BERT), and reducing the dimensionality to a low-dimensional space using PCA (cumulative variance contribution rate ≥85%), high-dimensional semantic retrieval is transformed into efficient range queries using low-dimensional vectors;

[0008] A B+ tree index is constructed using the low-dimensional vector as the key. Its ordered structure and range query capability replace the traditional high-dimensional vector similarity calculation, significantly reducing storage costs and retrieval complexity.

[0009] Dynamic incremental update design: New knowledge blocks generate low-dimensional key values ​​in real time and insert them into the B+ tree. The index structure is dynamically adjusted through node splitting algorithms (such as splitting a leaf node into two nodes when it is oversized and promoting the middle key value to the parent node) to avoid global reconstruction.

[0010] Periodically merge adjacent leaf nodes to optimize index balance and ensure long-term retrieval efficiency.

[0011] Multi-level search enhancement strategy: Query optimization: Weight the dimensions corresponding to keywords in the user query (preset weight greater than 1) to improve semantic matching accuracy;

[0012] Two-stage screening: First, the candidate set is quickly located in the B+ tree by Euclidean distance (≤ preset threshold δ), and then the Top-k results are finely screened by cosine similarity (based on max heap).

[0013] Generative enhancement: The search results are concatenated with the user's question to form prompts, which are then input into a large model to generate streaming answers.

[0014] Knowledge segmentation and feature fusion:

[0015] Semantic continuity is ensured by using overlapping blocks (pre-defined length tokens) and syntactic boundary detection;

[0016] It integrates TF-IDF and deep semantic features, taking into account both local keywords and global contextual information.

[0017] The present invention specifically adopts the following technical solution:

[0018] A RAG knowledge retrieval method based on B+ tree structure:

[0019] Transform user queries into low-dimensional vectors obtained through principal component analysis;

[0020] Range queries are performed using a pre-built B+ tree index to retrieve similar knowledge fragments, where the B+ tree index uses a low-dimensional vector of the knowledge document as the key.

[0021] The final answer is generated by combining the search results.

[0022] Furthermore, the construction of the B+ tree index includes:

[0023] The knowledge document is preprocessed to generate a low-dimensional vector as a key value. The preprocessing includes document segmentation and feature extraction.

[0024] Construct a B+ tree to store the key-value pairs and associated pointers.

[0025] Furthermore, the document segmentation process includes:

[0026] The document is divided into fixed-length text blocks, with adjacent blocks overlapping by a preset length of token.

[0027] Syntactic boundary detection is used to adjust block boundaries.

[0028] Furthermore, the document segmentation process includes:

[0029] Extract TF-IDF features;

[0030] Generate semantic vectors for the pre-trained model;

[0031] By fusing the TF-IDF features and semantic vectors, and reducing the dimensionality to a lower level through principal component analysis, the cumulative variance contribution rate is no less than 85%.

[0032] Furthermore, in the process of retrieving similar knowledge fragments through range queries using a pre-built B+ tree index:

[0033] The dimensions corresponding to the keywords in the query are weighted, with a preset weight greater than 1;

[0034] Candidate sets are filtered using Euclidean distance, with the distance not exceeding a preset threshold;

[0035] Filter Top-k results using cosine similarity.

[0036] Furthermore, it also includes a dynamic update step:

[0037] The newly added knowledge blocks are preprocessed to generate low-dimensional vectors as key-value pairs;

[0038] Insert the key value into the B+ tree index. If the leaf node capacity exceeds the threshold, trigger a splitting algorithm to adjust the tree structure.

[0039] And, a RAG knowledge retrieval system based on a B+ tree structure, comprising:

[0040] The query transformation module is used to convert user queries into low-dimensional vectors obtained through principal component analysis dimensionality reduction.

[0041] The retrieval module is used to perform range queries through a pre-built B+ tree index to retrieve similar knowledge fragments, wherein the B+ tree index uses a low-dimensional vector of the knowledge document as the key.

[0042] The generation module is used to generate the final answer by combining the search results.

[0043] Furthermore, it also includes an index building module for:

[0044] The knowledge document is preprocessed to generate a low-dimensional vector as a key value. The preprocessing includes document segmentation and feature extraction.

[0045] Construct a B+ tree to store the key-value pairs and associated pointers.

[0046] And, a computer device, characterized in that it includes a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the RAG knowledge retrieval method based on the B+ tree structure as described above.

[0047] A non-transitory computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the RAG knowledge retrieval method based on a B+ tree structure as described above.

[0048] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0049] Significantly improves retrieval efficiency and storage performance:

[0050] By constructing a retrieval mechanism based on a B+ tree hierarchical semantic index, high-dimensional semantic similarity calculation is transformed into a range query in a low-dimensional space (complexity O(log n)), significantly reducing retrieval time.

[0051] Combining PCA dimensionality reduction (cumulative variance ≥ 85%) significantly compresses vector dimensions, reduces storage overhead, and avoids the storage bottleneck of traditional high-dimensional vector databases.

[0052] Supports dynamic incremental updates and long-term stability:

[0053] The B+ tree splitting algorithm enables real-time insertion of new knowledge blocks without the need for global index reconstruction, ensuring the real-time update capability of the knowledge base.

[0054] By periodically merging low-capacity leaf nodes, the index structure is optimized, thus maintaining the stability of long-term retrieval performance.

[0055] Enhance semantic matching accuracy and generation quality:

[0056] By integrating TF-IDF statistical features with deep semantic vectors from pre-trained models, and taking into account both keyword weights and contextual relevance, the accuracy of semantic understanding is improved.

[0057] A two-stage screening mechanism (Euclidean distance for rapid initial screening + cosine similarity for fine screening) is adopted to ensure the accuracy of Top-k results;

[0058] By intelligently combining search results with user questions, the reliability and fluency of content generated by large models are optimized.

[0059] Ensuring knowledge continuity and system flexibility:

[0060] Overlapping segmentation and syntactic boundary detection techniques avoid semantic fragmentation and ensure the integrity of knowledge segments;

[0061] The configurable parameters (such as text block length, distance threshold, etc.) can be adapted to the needs of multiple scenarios, enhancing the system's versatility. Attached Figure Description

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0063] Figure 1 This is a flowchart of the RAG knowledge retrieval implementation based on the B+ tree structure in an embodiment of the present invention, wherein the left figure shows the scheme construction process and the right figure shows the retrieval implementation process;

[0064] Figure 2 This is a schematic diagram of the B+ tree index structure in an embodiment of the present invention. Detailed Implementation

[0065] In the following, specific embodiments of this application will be described in detail with reference to the accompanying drawings. Based on these detailed descriptions, those skilled in the art will be able to clearly understand and implement this application. Without departing from the principles of this application, features from various embodiments can be combined to obtain new implementations, or certain features from some embodiments can be substituted to obtain other preferred implementations.

[0066] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings:

[0067] This invention provides a RAG knowledge retrieval scheme based on a B+ tree structure, applicable to scenarios such as enterprise knowledge base management, intelligent question-answering systems, and professional domain document retrieval. Addressing the problems of high storage costs for high-dimensional vector databases, time-consuming index reconstruction, and lack of hierarchical knowledge organization capabilities in existing RAG systems, the scheme includes: knowledge preprocessing and feature extraction; B+ tree index construction; retrieval enhancement generation; a dynamic update mechanism to reflect the latest knowledge in real time, improving retrieval efficiency; and periodic optimization of the B+ tree structure. By constructing a hierarchical semantic index and a dynamic update mechanism, efficient knowledge retrieval, dynamic updates, and comprehensive optimization of storage costs are achieved. Figure 1 As shown, the specific implementation process includes the following steps:

[0068] Step S1: Knowledge preprocessing and feature extraction.

[0069] As a preferred embodiment, step S1 specifically includes the following steps:

[0070] Step S11: Divide the original knowledge document into fixed-length text blocks, with adjacent blocks overlapping by 100 tokens to ensure semantic continuity across blocks. Use syntactic boundary detection to adjust block boundaries and avoid splitting complete sentences into different blocks.

[0071] Step S12: TF-IDF feature calculation. The calculation formula is as follows:

[0072]

[0073]

[0074] Here, TF stands for Term Frequency, IDF stands for Inverse Document Frequency, the subscript t represents a word, and d represents a document block; Represents local frequency ( ) and global discrimination ( The product of )

[0075] The number of times word t appears in document block d; Let k be the total number of words in the block. In this formula, the subscript k is the traversal subscript, which means summing over all terms.

[0076]

[0077] Where N is the total number of documents, This represents the number of document blocks containing the word t.

[0078] Step S13: Use models such as Sentence-BERT to generate 768-dimensional semantic vectors to capture contextual semantics.

[0079] Step S14: After concatenating the TF-IDF vector with the semantic vector, reduce the dimensionality to k dimensions using principal component analysis. The calculation formula is as follows:

[0080]

[0081] in, Let represent the eigenvectors after dimensionality reduction using PCA. PCA(k) is a dimensionality reduction operator that performs principal component analysis on the input data. Specifically, it projects the data onto the principal component directions using an orthogonal transformation on the k eigenvectors with the largest eigenvalues. This represents the TF-IDF vector of document block d. This refers to semantic vectors generated by pre-trained models (such as Sentence-BERT). This indicates a splicing operation.

[0082] The cumulative variance contribution rate is required to be ≥85% to ensure that the main semantic information is preserved after dimensionality reduction.

[0083] Step S2: B+ tree index construction

[0084] As a preferred embodiment, step S2 specifically includes the following steps:

[0085] Step S21: Define the B+ tree node structure, such as... Figure 2 As shown: The internal nodes of a B+ tree contain pointers and keys, while the leaf nodes contain vector keys for knowledge document fragments, pointers to their storage locations, and pointers to the next leaf node. Internal nodes are used for fast location of information in index range queries, while leaf nodes store links to the actual knowledge fragment information and links to adjacent leaf nodes, facilitating sequential traversal.

[0086] Step S22: Take the k-dimensional principal component vector after dimensionality reduction as the key value of the B+ tree.

[0087] Step S23, Node Splitting and Merging: When the key values ​​stored in a node exceed the threshold, the B+ tree splitting algorithm is used to maintain the balance of the tree.

[0088] Step S3: Retrieval Enhancement Generation

[0089] As a preferred embodiment, step S3 specifically includes the following steps:

[0090] Step S31: Convert the user question into a 768-dimensional semantic vector using the same pre-trained model, and then reduce the dimensionality using PCA to obtain a k-dimensional query key. .

[0091] Step S32: Weight the dimensions corresponding to the keywords in the query. The calculation formula is as follows:

[0092] ( =1.5 when the word is a keyword)

[0093] Step S33: Starting from the root node, query the key-value pairs by comparison. For each key-value pair of internal nodes, recursively traverse the nodes, selecting the closest child node, until a leaf node is reached. Collect all key-value pairs. Euclidean distance The leaf nodes form the candidate set C.

[0094] Step S34: For each knowledge block vector b in the candidate set C, calculate the cosine similarity with the original query vector q. The calculation formula is:

[0095]

[0096] CosineSimilarity is the standard cosine similarity function.

[0097] Use a max-heap to filter out the top-k results.

[0098] Step S35: Concatenate the retrieved knowledge block text with the user's question to form prompt words, input the prompt words into the large model, generate an answer, and support streaming output to improve the interactive experience.

[0099] Step S4: Preprocess the newly added knowledge blocks to generate k-dimensional principal component key values. Starting from the root node, find a suitable leaf node to insert the key and store the pointer. If the capacity of a leaf node exceeds max_size, trigger the splitting algorithm to adjust the tree structure.

[0100] As a preferred embodiment, in the dynamic update mechanism:

[0101] When the number of key values ​​in a leaf node exceeds a preset threshold, the B+ tree splitting algorithm is executed:

[0102] Split the current node into two new nodes;

[0103] Extract the intermediate key value and promote it to the parent node;

[0104] Recursively check the capacity of the parent node until the root node is adjusted.

[0105] As a preferred embodiment, the B+ tree structure is periodically optimized:

[0106] Periodically traverse the B+ tree, merge adjacent leaf nodes whose number of key values ​​is below a threshold, and rebuild the index balance.

[0107] The following parameters are preset values:

[0108] Fixed length of text block: L (unit: token)

[0109] Node capacity threshold: max_size

[0110] Euclidean distance threshold: δ

[0111] Number of Top-k results: k.

[0112] The technology provided in this embodiment has the following advantages and effects:

[0113] (1) Hierarchical semantic index: By using the ordered structure of B+ tree and PCA dimensionality reduction, high-dimensional semantic retrieval is transformed into range query in low-dimensional space, thereby improving retrieval efficiency.

[0114] (2) Dynamic update optimization: The splitting and merging mechanism of B+ tree is used to support incremental insertion without rebuilding the global index, thus improving the performance of real-time updates.

[0115] Multi-dimensional retrieval enhancement: Improve the accuracy of semantic matching by fusing features from TF-IDF and pre-trained models.

[0116] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0117] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0119] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

[0120] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of RAG knowledge retrieval methods based on B+ tree structures. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A RAG knowledge retrieval method based on B+ tree structure, characterized in that: Transform user queries into low-dimensional vectors obtained through principal component analysis; Range queries are performed using a pre-built B+ tree index to retrieve similar knowledge fragments. The B+ tree index uses the low-dimensional vector of the knowledge document as the key. The internal nodes of the B+ tree contain pointers and keys, while the leaf nodes contain the vector key of the knowledge document fragment, a pointer to the storage location, and a pointer to the next leaf node. The internal nodes are used for fast location in the index range query, and the leaf nodes are used to store links to the actual knowledge fragment information and links to adjacent leaf nodes, which facilitates sequential traversal. The final answer is generated by combining the search results; The construction of the B+ tree index includes: The knowledge document is preprocessed to generate a low-dimensional vector as a key value. The preprocessing includes document segmentation and feature extraction. The document segmentation process includes: dividing the original knowledge document into fixed-length text blocks, and using syntactic boundary detection to adjust the block boundaries to avoid splitting complete sentences into different blocks; Extract TF-IDF features; Generate semantic vectors for the pre-trained model; By fusing the TF-IDF features and semantic vectors, and reducing the dimensionality to a low level through principal component analysis, the cumulative variance contribution rate is no less than 85%. Construct a B+ tree to store the key-value pairs and associated pointers; It also includes a dynamic update step: The newly added knowledge blocks are preprocessed to generate low-dimensional vectors as key-value pairs; Insert the key value into the B+ tree index. If the leaf node capacity exceeds the threshold, trigger a splitting algorithm to adjust the tree structure.

2. The RAG knowledge retrieval method based on B+ tree structure according to claim 1, characterized in that: The process of retrieving similar knowledge fragments through range queries using a pre-built B+ tree index: The dimensions corresponding to the keywords in the query are weighted, with a preset weight greater than 1; Candidate sets are filtered using Euclidean distance, with the distance not exceeding a preset threshold; Filter Top-k results using cosine similarity.

3. A RAG knowledge retrieval system based on a B+ tree structure, characterized in that, For performing the knowledge retrieval method as described in claim 1 or 2, including: The query transformation module is used to convert user queries into low-dimensional vectors obtained through principal component analysis dimensionality reduction. The retrieval module is used to perform range queries through a pre-built B+ tree index to retrieve similar knowledge fragments, wherein the B+ tree index uses a low-dimensional vector of the knowledge document as the key. The generation module is used to generate the final answer by combining the search results; The index building module is used for: The knowledge document is preprocessed to generate a low-dimensional vector as a key value. The preprocessing includes document segmentation and feature extraction. Construct a B+ tree to store the key-value pairs and associated pointers.

4. A computer device, characterized in that, It includes a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the RAG knowledge retrieval method based on the B+ tree structure as described in claim 1 or 2.

5. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the RAG knowledge retrieval method based on a B+ tree structure as described in claim 1 or 2.

Images