An LSM-Tree key-value store system that uses underlying information to build query indexes

Abstract

This invention discloses an LSM-Tree key-value storage system that uses underlying information to build a query index, belonging to the field of data storage technology. The system utilizes the boundaries of the underlying SSTables of the LSM-Tree as index keys. Through a compression process, it traverses all key-value pairs in the generated SSTables and uses the file numbers of the upper-level SSTables that intersect with the range of the index keys as index values ​​to construct an index. This invention accelerates query operations in the LSM-Tree key-value storage system.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, specifically relating to an LSM-Tree key-value storage system that uses underlying information to build a query index. Background Technology

[0002] With the widespread adoption of the internet and the gradual shift from PC internet to mobile internet, the application scenarios for databases have become increasingly diverse. Furthermore, with the emergence of digital development projects such as the Industrial Internet, Energy Internet, Smart Cities, Smart Healthcare, Connected Vehicles, Fintech, Smart Education, and Data Villages, the scale of data has grown rapidly. This increasing market size presents significant challenges to data storage and analysis. Efficient and secure storage is fundamental to the analysis and processing of massive amounts of data. To address the challenges posed by the ever-increasing data volume, storage systems and hardware are constantly evolving. As data types become more diverse and content more varied, relational (SQL) databases have gradually shifted towards more scalable non-relational (NoSQL) databases in the 21st century, and these have gained widespread application.

[0003] NoSQL databases include key-value databases, time-series databases, graph databases, vector databases, document databases, and columnar databases. In CCSA TC601 statistics, there are 82 key-value databases, accounting for 23.8% of the 344 NoSQL databases, making it the most prevalent type of NoSQL database. Key-value databases are characterized by using key-value pairs to store data, resulting in a simple and easy-to-understand structure. Typical systems include BigTable, Redis, Memcached, RocksDB, LevelDB, and ForestDB. Among key-value storage databases, the Log Merge Tree (LSM-Tree) structure is the most widely used write-optimized hierarchical data structure. It writes data to an internal ordered string table (SSTable) using append-only log operations. Because it fully utilizes the disk's write capabilities, it greatly improves write (put) performance. Later, the Log Merge Tree (LSM-Tree) structure transforms delete and update operations into write operations, avoiding the query operations before each modification and deletion. This out-of-place update method further improves delete and update performance. However, these operations disrupt the relationships between data, making the query process challenging. Therefore, the LSM-Tree proposes a layered approach, dividing the overall data structure into multiple layers with progressively increasing storage requirements, storing data layer by layer from top to bottom. Then, through compaction, the data in each layer becomes ordered. This allows for faster point queries within each layer using binary search, while the ordered data also facilitates range queries. To further improve query performance, the Log Structure Merged Tree (LSM-Tree) also incorporates data ranges and Bloom filters into the ordered string table (SSTable) to enhance the speed of filtering data blocks during the query process. These features make the LSM-Tree an important component of key-value storage systems.

[0004] However, the design of the log structure merge tree (LSM-Tree) also led to some new problems, namely read / write amplification and space amplification.

[0005] 1. Read Amplification:

[0006] During a point query, because the ordered string tables storing data in each layer are ordered, and the data within each ordered string table is also ordered, the Log Structure Merge Tree (LSM-Tree) reads the header files of the ordered string tables from top to bottom during a point query. It then filters the ordered string tables based on the file range and Bloom filter recorded in the header files. Since data is written to the LSM-Tree in an append-only manner, the first value found from top to bottom is the latest value. This method avoids the multi-versioning problem of values ​​stored in the LSM-Tree.

[0007] When performing a range query, since the ordered string tables storing data in each layer are ordered, and the data within each ordered string table is also ordered, the Log Structure Merging Tree (LSM-Tree) selects data blocks within the range of each layer from top to bottom through the header file of the ordered string table. Finally, it reads all the ordered string tables that meet the requirements into memory, arranges and integrates them, and finally outputs the query result.

[0008] 2. Write Amplification:

[0009] Write amplification refers to the ratio of the actual amount of data written to a traditional hard disk drive (HDD) / solid state drive (SSD) to the amount of data required to be written by the program. During compression operations, data is continuously read out, merged, and sorted before being written, resulting in the actual amount of data written being much greater than the amount required by the program.

[0010] 3. Space Amplification:

[0011] Because all writes are sequential, and deletions and modifications are performed using out-of-place updates rather than in-place updates, expired data is not immediately cleared and continues to occupy space.

[0012] In conclusion, it is both necessary and worthwhile to conduct research in the field of key-value storage based on log structure merged tree (LSM-Tree), to study how to optimize existing key-value storage systems while retaining their excellent performance and minimizing the impact of read amplification, write amplification, and space amplification, and to research and implement a key-value storage system with better performance. Summary of the Invention

[0013] To address the aforementioned issues, this invention discloses an LSM-Tree key-value storage system (BIVXDB) that uses underlying information to build a query index. This system uses the underlying information to build a query index of the underlying information inverted index (BottomInformationInVertIndeX, abbreviated as BIVX), which can accelerate the query operation of the LSM-Tree key-value storage system.

[0014] BIVX utilizes the boundaries of the underlying storage unit files of the LSM-Tree—ordered string tables—as index keys. Through the compression process within the LSM-Tree key-value storage system, it traverses all key-value pairs within the generated ordered string tables, constructing an index using the file numbers of the ordered string tables whose ranges intersect with the index key's range. During the search process, using the index, it first identifies the range of the target key-value pair from the underlying LSM-Tree. Then, it uses the index to find the file numbers of all ordered string tables that might contain the target value. Finally, it uses the string table boundary information and its own Bloom filter to determine if the target value exists in the table. Using BIVX, this invention can fully leverage the write advantages of the LSM-Tree without sacrificing its query performance.

[0015] This invention constructs BIVXDB, a single LSM-Tree key-value store system that incorporates BIVX indexes. By integrating the compression phase and incidentally constructing BIVX, BIVXDB achieves both low write amplification and fast search.

[0016] The technical solution of the present invention includes the following technical contents.

[0017] An LSM-Tree key-value storage system that uses underlying information to build a query index, the system comprising: a memory data storage module, an LSM-Tree structured disk data storage module, and an index module;

[0018] The memory data storage module is used to generate a Memtable data structure for storing data, and after the Memtable data structure is filled with data, it converts the Memtable data structure into an Immutable Memtable data structure and generates a new Memtable data structure to continue to receive the written data.

[0019] The disk data storage module is used to convert the Immutable Memtable data structure into an ordered string table file and store it in the L0 layer of the LSM-Tree structure, and then store it in the L0 layer of the LSM-Tree structure. i Once the number of files in a layer reaches a threshold, compression is performed, and the resulting ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number;

[0020] An index module is used to construct an inverted index for underlying information; wherein the index key of the inverted index for underlying information is the boundary value of the ordered string table file at the bottom layer of the LSM-Tree structure, and the index value of the inverted index for underlying information is the file number of the ordered string table file at the bottom layer of the LSM-Tree structure.

[0021] Furthermore, when the input file for the compression operation is a non-low-level ordered string table file and the output file is an ordered string table file written to the low-level structure, the process of updating the low-level information inverted index includes:

[0022] Get the index key of the output file;

[0023] While inserting the index key of the output file into the underlying information inverted index, delete the file number corresponding to the input file from all index values ​​of the underlying information inverted index to obtain the updated underlying information inverted index.

[0024] Furthermore, when the input file for the compression operation is a low-level ordered string table file and the output file is a low-level ordered string table file written to it, the process of updating the low-level information inverted index includes:

[0025] Step S31: Obtain the index key of the output file;

[0026] Step S32: Determine whether there is an intersection between the range of the index keys in the underlying inverted index and the range of the index keys in the output file;

[0027] Step S33: If there is an intersection between the range of the index key of the output file and the inverted index of the underlying information, then proceed to step S34; otherwise, proceed to step S38.

[0028] Step S34: Back up the original index key in the underlying information inverted index and delete the original index key from the underlying information inverted index;

[0029] Step S35: Insert the index key of the output file into the underlying information inverted index;

[0030] Step S36: Compare the intersection of the index values ​​of the original index key and the new index key inserted into the bottom-level information inverted index, and copy the index values ​​that have the intersection into the new index key to obtain the preliminary updated bottom-level information inverted index;

[0031] Step S37: Delete the file number corresponding to the input file from all index values ​​of the initially updated underlying information inverted index to obtain the updated underlying information inverted index;

[0032] Step S38: While inserting the index key of the output file into the underlying information inverted index, delete the index key of the input file in the underlying information inverted index to obtain the updated underlying information inverted index.

[0033] Furthermore, when the input file for the compression operation is a non-low-level file and the output file is a file written to a non-low-level file, the process of updating the low-level information inverted index includes:

[0034] A list of tags is constructed based on the index keys in the inverted index of the underlying information;

[0035] If the key-value pairs output by the tagging compression process intersect with the data regions represented by each index key, then the data region is tagged in the tagging list.

[0036] Update the underlying information inverted index according to the tag list, and delete specific index values ​​in the underlying information inverted index; wherein, the specific index values ​​include: expired index values, index values ​​that do not intersect with boundary values, and index values ​​that do not intersect with the data region.

[0037] Furthermore, when the input file for the compression operation is a low-level file and the output file is a file written to a non-low-level file, the process of updating the low-level information inverted index includes:

[0038] Iterate through all key-value pairs output during the compression process, where the input file is a non-low-level file and the output file is a file written to a non-low-level file.

[0039] Insert the key-value pair information into the current underlying inverted index;

[0040] Delete the information in the input file and back it up to the first region larger than the index key to be deleted;

[0041] A maximum value marker is stored in the back end of the inverted index of the underlying information to buffer the maximum value marker. This maximum value marker is then removed from the nth level of the LSM-Tree and gradually stored in the MAX region before being transferred to the (n+1)th level of the LSM-Tree.

[0042] Furthermore, the system also includes an information synchronization module, which is used to synchronize and store the index information in the underlying inverted index to the disk.

[0043] A range query method based on the LSM-Tree key-value storage system according to any one of the above claims, comprising:

[0044] Step S71: Construct an iterator;

[0045] Step S72: Use the iterator to perform a range query on the data in the Memtable data structure, and add the filtered file obtained based on the Memtable data structure to the iterator, then jump to step S73;

[0046] Step S73: Use the iterator to perform a range query on the data in the Immutable Memtable data structure, and add the filtered file obtained based on the Immutable Memtable data structure to the iterator, then jump to step S74;

[0047] Step S74: Use the iterator to perform a range query on the ordered string table file in the L0 layer of the LSM-Tree structure, and add the filtered file obtained based on the L0 layer of the LSM-Tree structure to the iterator, then jump to step S75.

[0048] Step S75: Obtain the file number of the ordered string table file that meets the query conditions from the inverted index of the underlying information, add the filtered file obtained based on the inverted index of the underlying information to the iterator, and then jump to step S76.

[0049] Step S76: Combine the filtered files obtained based on the Memtable data structure, the filtered files obtained based on the Immutable Memtable data structure, the filtered files obtained based on the L0 layer of the LSM-Tree structure, and the filtered files obtained based on the inverted index of the underlying information to obtain the range query results that meet the query conditions.

[0050] Furthermore, the processing of the ordered string table file includes:

[0051] Filter data within a target range from an ordered string table file;

[0052] The filtered data is merged and / or updated to generate preliminary query results;

[0053] Sort the preliminary query results to obtain the range of query results that meet the query criteria.

[0054] A point query method based on any of the LSM-Tree key-value storage systems described above, comprising:

[0055] Step S91: Obtain the target key that matches the query conditions;

[0056] Step S92: Perform a target key query in the Memtable data structure, and if the target key is obtained, proceed to step S97; otherwise, proceed to step S93.

[0057] Step S93: Perform a target key query in the Immutable Memtable data structure, and if the target key is obtained, proceed to step S97; otherwise, proceed to step S94.

[0058] Step S94: Query the target value corresponding to the target key in the L0 layer of the LSM-Tree structure. If the target key is obtained based on the target value, proceed to step S96; otherwise, proceed to step S95.

[0059] Step S95: Locate the range of key values ​​that may contain the target key based on the inverted index of the underlying information, and filter the range of key values ​​that may contain the target key through a Bloom filter, then proceed to step S96.

[0060] Step S96: Output the query results of points that meet the query conditions.

[0061] A data writing method based on any of the above-described LSM-Tree key-value storage systems includes:

[0062] Write the original data into the Memtable data structure;

[0063] When the Memtable data structure reaches its capacity limit, the Memtable data structure is converted into an Immutable Memtable data structure, and a new Memtable data structure is generated to continue to accept the written data.

[0064] After converting the Immutable Memtable data structure into an ordered string table file, it is stored in the L0 layer of the LSM-Tree structure;

[0065] In the LSM-Tree structure i Once the number of files in a layer reaches a threshold, compression is performed, and the resulting ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number;

[0066] The data space is divided into several data fields based on the boundary values ​​of the ordered string table files at the bottom layer of the LSM-Tree structure, and the file numbers of the ordered string table files that intersect with each data field are used to fill the inverted index of the underlying information.

[0067] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0068] 1. High-performance point query and range query capabilities.

[0069] This invention uses the boundary information of the ordered string table file stored in the underlying layer of the LSM-Tree as the index key to divide the entire data domain into multiple regions. It then compares the key-value pairs in the upper-level ordered string table with the divided data domains, recording the file number of the ordered string table that has a clear intersection with the underlying data domain as the index value. In both point and range queries, this invention can directly filter out the ordered string table file to be queried and read using the range information stored in the index. This skips the original layer-by-layer traversal process of the LSM-Tree, greatly saving time during the query phase and significantly improving the system's point and range query capabilities.

[0070] 2. Low-overhead index creation process.

[0071] This invention utilizes the LSM-Tree compression process for index building. The reasons for index building during this process are twofold: First, it deletes expired files and inserts new ones, resulting in concentrated modifications to existing files in the database, necessitating synchronous index updates. Second, during this process, the ordered string table is read from disk into memory, and its key-value pairs are traversed. The window period during which all key-value pairs in this ordered string table are traversed allows this invention to accurately record information in the index, avoiding the misreading problems caused by the boundary values ​​of the ordered string table and Bloom filters in traditional methods. Based on these two reasons, this invention builds the index during the compression phase. Since the index building process is integrated into the compression phase, it does not introduce significant new file I / O overhead. Therefore, this system achieves a low-cost, high-return index building process. Attached Figure Description

[0072] Figure 1 This is the overall structure of BIVXDB.

[0073] Figure 2 This is a compression flowchart in BIVXDB under the BIBO (Bottom-In-Bottom-Out) configuration.

[0074] Figure 3This is a compression flowchart in BIVXDB under the UIUO (Unbottom-In-Unbottom-Out) case (i.e., not a bottom-level file input and not a bottom-level file output).

[0075] Figure 4 This section describes the occurrence of BIUO (Bottom-In-Unbottom-Out) in BIVXDB.

[0076] Figure 5 This is a compression flowchart in BIVXDB under the BIUO (Bottom-In-Unbottom-Out) case, where the input is a low-level file and the output is a non-low-level file.

[0077] Figure 6 This is the BIVXDB range query process.

[0078] Figure 7 This is the BIVXDB point query process.

[0079] Figure 8 This describes how write performance varies across different systems depending on the amount of data.

[0080] Figure 9 This describes how the performance of point queries varies with the amount of data in different systems.

[0081] Figure 10 This describes how the performance of range queries varies with the range query length across different systems.

[0082] Figure 11 This shows how different systems perform under the default YCSB load. Detailed Implementation

[0083] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0084] The overall architecture of the present invention is as follows Figure 1As shown, this invention retains most of the RocksDB structure. Therefore, when data is written, it is first written to the Memtable. When the Memtable is full, it is converted into an ImmutableMemtable, and a new Memtable is created to continue handling the written data. The Immutable Memtable is flushed to the L0 layer on disk in the background, converted into ordered string table files for storage. In RocksDB, multiple ordered string tables exist in the L0 layer; these files overlap and are unordered. Then, when the number of files in the L0 layer reaches a threshold, a compaction process is triggered, re-sorting and merging these unordered ordered string table files, and storing them in the L1 layer. Subsequent layers undergo compaction when their capacity reaches a threshold. Therefore, the ordered string table files in each layer from L1 onwards are ordered and have no overlap. Because of this characteristic of RocksDB, this invention ignores the L0 layer when creating indexes, and creates indexes for all other layers. This study stores BIVX in DRAM and modifies the original Rocksdb compression process. Utilizing the window during which all data in the ordered string table file is read into memory, all data in the ordered string table file involved in the compression operation is partitioned according to the underlying index and stored separately in the corresponding indexes of the underlying data regions. This method establishes a global index for the Log Structure Merged Tree (LSM-Tree). Since BIVX is updated during compression, it is updated continuously as the size and number of levels of the LSM-Tree change.

[0085] 1. BIVX Design

[0086] BIVX is an abbreviation for Bottom Information Inverted Index. BIVX is implemented using a Map data structure in C++. When a BIVX is created for an LSM-Tree, the correspondence between the two is as follows: Figure 1As shown, this invention divides the entire data space into 2n+1 regions based on the boundary values ​​of n ordered string table files stored at the bottom layer of the LSM-Tree. BIVX then uses the file numbers of the ordered string table files that intersect with each data field to populate the index, achieving fast lookup. The process of determining whether there is intersection and updating BIVX is performed during the compression stage when traversing all key-value pairs of the generated ordered string table files. Using the traversal of all key-value pairs during compression is to avoid potential misjudgments that might occur when judging solely based on the boundary values ​​of the generated ordered string table files. In this study, the data regions used for indexing are referred to as index keys, and the stored file numbers are referred to as index values. An index key and its corresponding index value constitute an index item.

[0087] In BIVXDB, an empty BIVX is created simultaneously when an LSM-Tree is created; subsequently, the BIVX is updated during the compression phase as LSM-Tree data is continuously written. Since the presence of input and output files at the lowest level during compression affects whether the index keys in the BIVX need to be modified, this invention divides compression into four cases based on whether the input file (SSTables to be compressed in the RocksDB system) and the output file are at the lowest level:

[0088] a) The input is a non-bottom file, and the generated file is written to the bottom layer (UIBO).

[0089] When the input file in the compression process is a high-level file and the output file is a low-level file, it means that there is no overlap between the newly generated file and the existing low-level file, so it will not affect the index keys that have already been written. Therefore, in this case, it is only necessary to directly insert the boundary values ​​of the newly written low-level file into BIVX, and at the same time delete the file numbers of the files involved in the compression process from each index entry.

[0090] b) The input is a low-level file, and the generated file is also written to the low-level file (BIBO).

[0091] When the input file contains underlying files during the compression process, and the output file also needs to be written to the underlying files, BIVX requires two steps: deleting old index entries and inserting new index entries. One key issue is how to transfer the information from the old index entries to the new ones. Figure 2 The process of compressing S4, located at the bottom layer (nth layer) of an LSM-Tree, and S10, located at the (n-1th)th layer, and storing the output files S21 and S22 back at the bottom layer of the LSM-Tree. This process is... Figure 2 The section shown by the dotted lines is mainly divided into three processes: First, the data from S4 and S10 are read into memory, then merged and sorted; next, the data is stored in S21 and S22 respectively; finally, the information from S21 and S22 is updated in the LSM-Tree. Figure 2 The process is shown in detail below the dotted line.

[0092] Step b1: Back up the index key to be deleted and delete the expired data in the index key.

[0093] Since S4 is located at the bottom layer of the LSM-Tree, any changes to it will affect the data area division in BIVX. Therefore, the system first selects the index items (1: S4 & 21: S20, S10, S4) that need to be deleted from BIVX, backs them up, and then deletes them from BIVX.

[0094] Step b2: Insert a new index key.

[0095] Since the newly generated files S21 and S22 will be output to the lowest level, they will also affect the partitioning of the data region. Therefore, the system should also insert their boundary values ​​into BIVX sequentially. The index entries within the black box in the diagram are all newly inserted.

[0096] Step b3: Compare the intersection between the new index region and the old region, and copy the index values ​​that have intersection to the new index.

[0097] The system will iterate through the backed-up entries and insert all the index data into the entries that intersect in the new index. In this way, the information from the original BIVX will be completely inherited.

[0098] Step b4: Delete expired index values.

[0099] The system needs to compare whether the files corresponding to all index values ​​are within the range corresponding to the index keys, and delete expired and redundant data to facilitate the subsequent search process and reduce the space overhead of BIVX.

[0100] c) The input is a non-low-level file, and the generated file is also output to a non-low-level file (UIUO).

[0101] When compressing ordered string table files across all layers except the lowest layer, updates to BIVX are limited to index values. Therefore, its update process mainly focuses on the traversal and output processes during the compression phase.

[0102] Figure 3An example is used to illustrate the above process. When S20 and S10, which are not at the bottom layer in the LSM-Tree, are merged, the corresponding BIVX update steps are as follows: Figure 3 The lower half of the dashed line is shown.

[0103] Step c1: Store a list of tags according to BIVX to mark whether there is an intersection with each range when the output of the compression process is used.

[0104] The system needs to build a tag list based on the index keys in BIVX. This tag list is used to mark whether there is any overlap between the key-value pairs output by the compression process and the data areas represented by each index key.

[0105] Step c2: When the compression process outputs key-value pairs sequentially, determine whether they intersect with the ranges in BIVX. If they do, mark them in the list.

[0106] During LSM-Tree compression, the traversal and output phase determines whether each key-value pair intersects with the data area represented by each index key. If an intersection exists, the corresponding area is marked. To reduce the overhead of this process, the system first finds the first value (21) not less than the first key output (4 in the example) and marks it. Subsequently, the system will not modify the list until it encounters the first value (26) greater than 21 in the output, and then finds the first value (38) not less than it in the list and marks it. The above steps are repeated until all output keys have been traversed.

[0107] Step c3: Update BIVX based on the tag list and delete expired index values ​​and index values ​​that do not intersect with boundary values ​​and ranges.

[0108] Based on the records in the list, filter out the information corresponding to the data range, and update the number of the ordered string table to the corresponding index item.

[0109] d) The input is a low-level file, but the output file is not written to the low-level file (BIUO).

[0110] As the data volume increases and the number of layers in the LSM-Tree changes from n to n+1, the "bottom layer" gradually transitions from the original nth layer to the (n+1th)th layer. During this process, BIVX continuously updates with compression, gradually shifting its scope from the nth layer to the (n+1th)th layer. In this process, BIVX simultaneously contains data from both the (n+1th)th layer (the new bottom layer) and the nth layer (the original bottom layer). Because each layer of the LSM-Tree is ordered and non-overlapping, but this principle cannot be guaranteed between different layers, improper handling of this process can corrupt the information at the bottom layer, making effective indexing impossible. This process is as follows... Figure 4As shown.

[0111] As the range of layer n+1 gradually expands during this process, its intersection with layer n also increases. What this invention needs to do is to gradually inherit the information of layer n stored in BIVX during this process, and gradually update and assign it to layer n+1.

[0112] A fourth scenario can occur during compression in this transitional phase: the input file for compression contains a low-level file located at layer n, but the output file is simultaneously stored back in this area. In this case, BIVX still uses the information from the input file to construct the index keys, but since the target layer of the output file is no longer the lowest layer in the current situation, updating the BIVX index keys using the update method employed in the UIUO case would result in the loss of existing data. Therefore, this invention proposes a novel composite update method for updating BIVX.

[0113] First, this invention iterates through all key-value pairs output during the compression process according to the UIUO method, inserting their information into the current BIVX. Then, to gradually reduce the proportion of the nth layer in the current BIVX index key, this invention chooses to delete information from the input file and back it up to the first region larger than the index key to be deleted. During this process, since BIVX selects regions less than or equal to the index key during searches, backing up information to a region with a larger index key does not affect the query. By combining these two update methods, this invention can simultaneously retain old information, update the information, and reduce the proportion of the nth layer file in the BIVX index key. Furthermore, because this invention stores a maximum value marker (MAX) in the BIVX backend for buffering, information is continuously deleted from the nth layer and gradually stored in the MAX region, then slowly transitions to the (n+1)th layer. The process is roughly as follows. Figure 5 As shown.

[0114] Figure 5 This demonstrates the entire process of the system compressing S23 and S20 and placing the output file in the nth layer to update BIVX, as follows: Figure 5 As shown, the system first reads all the data from both indices and iterates through them sequentially. Based on the traversal information, a new file number is inserted into index entry 97, making its index value S23, S28, and other existing indices. The expired record for S23 is then deleted from the index. However, it is then discovered that the output file has reached level n. Therefore, the information from index entries 88 and 97 needs to be backed up to the first region larger than the index key to be deleted, i.e., the ∞ index entry (MAX region), and then both index entries 88 and 97 are deleted. Only then is the compression process in the BIUO case completely finished.

[0115] 2. BIVXDB Operation Process Analysis

[0116] The core operations of the BIVXDB system include write (Put()), range query (Scan()), and point query (Get()) processes, involving the management and retrieval of key-value pairs in the database. In BIVXDB, data update and deletion operations do not directly modify or remove old data items. Instead, they employ a version control and marker-based deletion approach to handle these changes. Specifically, when performing an update operation, the system writes a new version of the data, containing the updated value while retaining the old version data with the same key. This approach allows BIVXDB to provide historical versions of data while also supporting efficient data updates. Each version of a data item is assigned a unique timestamp or version number to distinguish the chronological order of different versions. Similarly, for deletion operations, BIVXDB does not immediately remove the corresponding data item from physical storage. Instead, it writes a special deletion marker to indicate that the key-value pair has been deleted. When performing a read operation, the system checks these deletion markers and filters out data items marked as deleted before returning the results. This method simplifies the deletion process and makes future data recovery possible. By employing this design, BIVXDB enhances the flexibility and efficiency of data operations while maintaining the traceability of historical data versions, meeting the data management needs of complex application scenarios. Furthermore, by utilizing version control and deletion tags, BIVXDB can reduce data redundancy and wasted storage space while ensuring operational efficiency, further optimizing the overall system performance.

[0117] a) Range query process

[0118] The introduction of BIVX significantly improves the efficiency of LSM-Tree-based database systems when performing range queries. Traditionally, range searches require traversing all layers of the LSM-Tree, performing a global search on the ordered string table files of each layer, and then checking each ordered string table file to see if it contains data within the target range. This process is not only inefficient but also becomes significantly slower as the data volume increases. In contrast, BIVX's design allows the system to skip this layer-by-layer global search and directly utilize BIVX for efficient range queries.

[0119] The range query process in BIVXDB is as follows: Figure 6As shown. In its specific implementation, the range query process begins by constructing an iterator, which first includes the in-memory Memtable and the ImmutableMemtable converted to a read-only state. This step ensures that the latest written data can be queried immediately, maintaining data real-time performance. Next, considering that BIVX does not include information about files in the L0 layer (where files are unordered and overlap), the system needs to perform special processing on the L0 layer, that is, to include all ordered string table files in the L0 layer in the iterator's search range.

[0120] Furthermore, BIVX provides direct indexing information for each fine-grained data region, including the file number storing that data range. With BIVX, the system can directly locate the specific file containing the target range of data without traversing other unrelated files, significantly reducing search time. Subsequently, the system reads the corresponding ordered string table file directly from the disk based on these file numbers and adds them to the iterator.

[0121] The final step involves the system comprehensively processing the data from all related ordered string table files in the iterator. This includes filtering data within the target range, merging duplicate or updated records, and sorting the results. This not only ensures the accuracy of the query results but also significantly improves the speed of range queries.

[0122] In this way, BIVX provides a more efficient range query solution for LSM-Tree-based key-value storage systems, significantly reducing query latency and improving the system's ability to process large-scale data.

[0123] b) Point query process

[0124] In BIVXDB, the point query process achieves efficient and accurate key-value pair lookup through a series of refined steps, as follows: Figure 7As shown, this process begins with the Memtable in memory and its read-only version, ImmutableMemtable, which stores the latest written data. Prioritizing queries here allows for quick access to recent writes, updates, and deletions. If the target key is not found in either of these structures, the search proceeds to the first level of storage on disk, the ordered string table files at level L0. Since files at level L0 may overlap, this step requires checking each file in that level individually. If the target value is still not found in level L0, the search continues in other levels. Because the ordered string table files in other levels are recorded in the BIVX index, the query operation can leverage the BIVX index to quickly locate the range of key values ​​that might contain the target key, narrowing the search to a few files. To further improve accuracy, the query process retains a Bloom filter and uses it to further filter the files identified by the BIVX index, quickly eliminating files that are unlikely to contain the target key. Finally, the target key is found in the filtered files, and the query results are output. If the key exists in the database, the corresponding value is returned; otherwise, a "not found" indication is returned. This efficient query process avoids the previous LSM-Tree layer-by-layer search process, ensuring that BIVXDB can respond to point query requests quickly and accurately, even when dealing with large amounts of data.

[0125] c) Writing process

[0126] BIVXDB employs RocksDB's data writing strategy. Initially, data is written to an in-memory Memtable, a sorted data structure to ensure fast write speeds. When the Memtable reaches its capacity limit, it is converted to a read-only Immutable Memtable, and a new Memtable is created to continue receiving new data, preventing interruptions to write operations. Next, the Immutable Memtable is asynchronously flushed to the L0 level on disk, with data stored as an ordered string table file—a file format that supports efficient range queries. After writing is complete, BIVXDB performs a compression operation, a background process designed to organize and merge the ordered string table files to reduce storage footprint and improve query efficiency. Furthermore, during compression, BIVXDB updates the BIVX index, its core component responsible for providing efficient data retrieval capabilities. This approach combines the speed advantages of in-memory computing with the large capacity of disk storage, and by timely updating the BIVX index, it maintains high query performance even during frequent data updates and deletions, thereby optimizing the data writing and querying process and improving the overall performance and scalability of the database.

[0127] d) Crash consistency

[0128] BIVXDB achieves crash consistency through a mechanism similar to RocksDB, primarily employing a Write-Ahead Log (WAL) strategy. This strategy ensures that any unfinished data operations can be recovered via the log in the event of a crash, thus guaranteeing data integrity and consistency. However, for BIVX indexes in BIVXDB, since their update operations are not incremental, the traditional WAL mechanism cannot be directly applied. Therefore, BIVXDB adopts a special strategy: after each BIVX index update, the updated index information is immediately synchronized and stored to disk. This step ensures that BIVXDB maintains index consistency and integrity even in the event of a system crash, further enhancing system reliability.

[0129] In summary, this invention proposes a "BIVX" index built using underlying information to improve the query performance of LSM-Trees. By leveraging the ordered, large-scale, and densely distributed characteristics of the underlying files, a dynamic underlying information index is cleverly constructed, eliminating the need for layer-by-layer searches during LSM-Tree queries. This significantly improves the query performance of the storage system, especially for range queries.

[0130] This invention achieves data traversal through the reading process of stored files during the compression stage of LSM-Tree, accurately and efficiently recording information in the index, and realizing index creation without introducing additional read / write overhead.

[0131] The following is a specific experiment illustrating the LSM-Tree key-value storage system provided by this invention, which uses underlying information to build a query index.

[0132] 1. Dataset

[0133] When evaluating write performance and point query performance, engineers primarily used the benchmark tools included with RocksDB and LevelDB for testing, and used YCSB as an auxiliary tool for verification. However, when testing range query performance, engineers faced some challenges. The range query interfaces provided by RocksDB and LevelDB are based on iterators, while BIVXDB implements its own proprietary scan interface, making it impossible to directly use the iterator interface in the benchmark tools to test range query performance. To address this challenge, this invention adopts a testing approach using YCSB.

[0134] YCSB is a general-purpose cloud service benchmarking tool that provides a rich set of workload types, including range query operations. The table below shows the default workload of YCSB. Using YCSB, this invention allows users to customize the workload of range query operations to simulate different query scenarios and measure the performance of BIVXDB in these scenarios, as shown in Table 1.

[0135]

[0136] Table 1

[0137] 2. Evaluation Indicators

[0138] First, dbbench was used to test the write performance and point query performance of the present invention (BIVXDB) and some existing solutions (LevelDB and RocksDB); then YCSB was used to test the range query performance of each solution; finally, YCSB was used to test the experimental performance of each solution under the default workload.

[0139] 3. Experimental Results

[0140] from Figure 8 The study observed that as the data volume increased from 20GB to 80GB, the write performance of BIVXDB, RocksDB, and LevelDB all showed a certain degree of decline. This phenomenon is common in storage systems because as the data volume increases, the system must handle more data files and indexes, which in some cases increases the complexity of the data write path, especially when compaction operations occur frequently. Despite this, BIVXDB consistently maintained superior write performance compared to RocksDB and LevelDB throughout the entire data volume growth process. This result indicates that although BIVXDB introduces additional operations during the compaction phase to optimize query performance, these improvements did not have a significant negative impact on overall write performance. On the contrary, BIVXDB, through effective indexing strategies, was able to maintain or even improve write performance while achieving a significant improvement in query performance.

[0141] according to Figure 9BIVXDB consistently outperforms RocksDB in read performance, especially as data volume increases, this performance difference becomes increasingly significant. LevelDB, on the other hand, demonstrates better read performance with smaller datasets, outperforming the other two database systems. This is primarily due to its concise and efficient design and optimized handling of small datasets, enabling rapid location and retrieval of required information when processing limited data. However, as data volume increases, LevelDB's performance advantage begins to diminish. This is because LevelDB's Log Merge Tree (LSM-Tree) mechanism faces more frequent merge operations and significant read amplification issues when handling large-scale data, gradually reducing query efficiency. Therefore, although LevelDB initially exhibits strong read performance, BIVXDB's performance improvements become increasingly apparent as data volume grows, ultimately demonstrating superior query performance in large-scale data processing. This performance is attributed to the unique design of BIVXDB's index structure. Because files at level 0 in the LSM-Tree are unordered and overlap, BIVX starts storing information from level 1. When the data volume is small, a large amount of data is concentrated at level 0. At this stage, BIVXDB's efficient indexing mechanism is not fully utilized, resulting in a small performance difference compared to RocksDB in the initial comparison, and relatively poor performance compared to LevelDB. However, as the data volume grows, more data gradually migrates to lower levels, and BIVXDB's indexing mechanism begins to play a crucial role, significantly improving query speed. Meanwhile, LevelDB and RocksDB employ a top-down, layer-by-layer search strategy. In contrast, BIVXDB's bottom-up indexing querying is more efficient, especially when handling large-scale datasets. Its performance advantage is even more pronounced, fully demonstrating BIVXDB's advanced design and efficient response to query demands in big data environments.

[0142] like Figure 10As shown, BIVXDB demonstrates an overwhelming performance advantage over the other two key-value stores, with its range query throughput being almost 2 to 6 times that of the other two. RocksDB shows a trend of decreasing query performance as query length increases. This is because RocksDB relies on a global iterator when performing range queries, and the overhead of the iterator increases with the query length, leading to a decrease in range query performance. LevelDB performs better than RocksDB, but the specific trend is basically the same as RocksDB. BIVXDB's performance is more complex, exhibiting a fluctuating pattern. The reason behind this fluctuating performance is that BIVXDB uses an inverted index mechanism based on the underlying information. This inverted index is built based on the maximum and minimum values ​​of the underlying files, allowing BIVXDB to quickly exclude a large number of irrelevant files when performing range queries, accessing and reading only files that may contain the target data, greatly improving query efficiency. However, BIVXDB's inverted index strategy also brings certain limitations. When the length of the query range is not an integer multiple of the length of the underlying file, the indexing mechanism may not be able to fully realize its performance advantages, resulting in a slight decrease in query efficiency. This explains the wave-like performance fluctuations exhibited by BIVXDB in the experiment.

[0143] From the experimental results Figure 11As observed, BIVXDB exhibits a significant performance improvement over RocksDB in most test scenarios, especially in non-update-intensive workloads. This performance advantage is primarily attributed to the significant improvement in point query performance brought about by the BIVXDB architecture. By building an underlying information index structure and utilizing it in the point query process, BIVXDB significantly improves query efficiency, especially when handling a large number of point query operations, enabling rapid data location and reducing data access latency. However, in the data loading phase and the update-heavy Workload D test scenario, BIVXDB's performance slightly decreases compared to RocksDB. This is mainly because BIVXDB requires additional comparison and record operations during the compaction phase to maintain the index when building and updating BIVX, which sacrifices write performance to some extent. Especially in update-intensive environments, frequent update operations lead to frequent index changes, increasing the system's maintenance burden and thus affecting overall write efficiency. In terms of range queries, BIVXDB's performance advantage over RocksDB is particularly pronounced; in the Workload E test, BIVXDB's performance is more than six times that of RocksDB. This significant improvement is attributed to BIVXDB's optimized design for range queries. By establishing an index structure and directly reading files using this structure, BIVXDB can handle range query requests more efficiently, greatly reducing the overhead of RocksDB's iterator-based file traversal, thus improving query efficiency. In summary, although BIVXDB's write performance may be affected in certain specific scenarios, the performance improvements demonstrated in key operations such as point and range queries fully demonstrate the effectiveness of BIVXDB's improvements.

[0144] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. An LSM-Tree key-value storage system that uses underlying information to build a query index, characterized in that, The system includes: a memory data storage module, an LSM-Tree structured disk data storage module, and an index module; The memory data storage module is used to generate a Memtable data structure for storing data, and after the Memtable data structure is filled with data, it converts the Memtable data structure into an Immutable Memtable data structure and generates a new Memtable data structure to continue to receive the written data. The disk data storage module is used to convert the Immutable Memtable data structure into an ordered string table file and store it in the L0 layer of the LSM-Tree structure, and then store it in the L0 layer of the LSM-Tree structure. i Once the number of files in a layer reaches a threshold, compression is performed, and the resulting ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number; An index module is used to construct an inverted index for underlying information; wherein the index key of the inverted index for underlying information is the boundary value of the ordered string table file at the bottom layer of the LSM-Tree structure, and the index value of the inverted index for underlying information is the file number of the ordered string table file at the bottom layer of the LSM-Tree structure.

2. The system according to claim 1, characterized in that, When the input file for the compression operation is a non-low-level ordered string table file, and the output file is an ordered string table file written to the low-level structure, the process of updating the low-level inverted index includes: Get the index key of the output file; While inserting the index key of the output file into the underlying information inverted index, delete the file number corresponding to the input file from all index values ​​of the underlying information inverted index to obtain the updated underlying information inverted index.

3. The system according to claim 1, characterized in that, When the input file for the compression operation is a low-level ordered string table file, and the output file is a low-level ordered string table file written to the low-level file, the process of updating the low-level information inverted index includes: Step S31: Obtain the index key of the output file; Step S32: Determine whether there is an intersection between the range of the index keys in the underlying inverted index and the range of the index keys in the output file; Step S33: If there is an intersection between the range of the index key of the output file and the inverted index of the underlying information, then proceed to step S34; otherwise, proceed to step S38. Step S34: Back up the original index key in the underlying information inverted index and delete the original index key from the underlying information inverted index; Step S35: Insert the index key of the output file into the underlying information inverted index; Step S36: Compare the intersection of the index values ​​of the original index key and the new index key inserted into the bottom-level information inverted index, and copy the index values ​​that have the intersection into the new index key to obtain the preliminary updated bottom-level information inverted index; Step S37: Delete the file number corresponding to the input file from all index values ​​of the initially updated underlying information inverted index to obtain the updated underlying information inverted index; Step S38: While inserting the index key of the output file into the underlying information inverted index, delete the index key of the input file in the underlying information inverted index to obtain the updated underlying information inverted index.

4. The system according to claim 1, characterized in that, When the input file for the compression operation is a non-low-level file and the output file is written to a non-low-level file, the process of updating the low-level information inverted index includes: A list of tags is constructed based on the index keys in the inverted index of the underlying information; If the key-value pairs output by the tagging compression process intersect with the data regions represented by each index key, then the data region is tagged in the tagging list. Update the underlying information inverted index according to the tag list, and delete specific index values ​​in the underlying information inverted index; wherein, the specific index values ​​include: expired index values, index values ​​that do not intersect with boundary values, and index values ​​that do not intersect with the data region.

5. The system according to claim 4, characterized in that, When the input file for the compression operation is a low-level file and the output file is a file written to a non-low-level file, the process of updating the low-level information inverted index includes: Iterate through all key-value pairs output during the compression process, where the input file is a non-low-level file and the output file is a file written to a non-low-level file. Insert the key-value pair information into the current underlying inverted index; Delete the information in the input file and back it up to the first region larger than the index key to be deleted; A maximum value marker is stored in the back end of the inverted index of the underlying information to buffer the maximum value marker. This maximum value marker is then removed from the nth level of the LSM-Tree and gradually stored in the MAX region before being transferred to the (n+1)th level of the LSM-Tree.

6. The system according to any one of claims 1 to 5, characterized in that, The system also includes an information synchronization module, which is used to synchronize and store the index information in the underlying inverted index to the disk.

7. A range query method based on the LSM-Tree key-value storage system according to any one of claims 1 to 6, characterized in that, The range query method includes: Step S71: Construct an iterator; Step S72: Use the iterator to perform a range query on the data in the Memtable data structure, and add the filtered file obtained based on the Memtable data structure to the iterator, then jump to step S73; Step S73: Use the iterator to perform a range query on the data in the Immutable Memtable data structure, and add the filtered file obtained based on the Immutable Memtable data structure to the iterator, then jump to step S74; Step S74: Use the iterator to perform a range query on the ordered string table file in the L0 layer of the LSM-Tree structure, and add the filtered file obtained based on the L0 layer of the LSM-Tree structure to the iterator, then jump to step S75. Step S75: Obtain the file number of the ordered string table file that meets the query conditions from the inverted index of the underlying information, add the filtered file obtained based on the inverted index of the underlying information to the iterator, and then jump to step S76. Step S76: Combine the filtered files obtained based on the Memtable data structure, the filtered files obtained based on the Immutable Memtable data structure, the filtered files obtained based on the L0 layer of the LSM-Tree structure, and the filtered files obtained based on the inverted index of the underlying information to obtain the range query results that meet the query conditions.

8. The range query method according to claim 7, characterized in that, The processing of the ordered string table file includes: Filter data within a target range from an ordered string table file; The filtered data is merged and / or updated to generate preliminary query results; Sort the preliminary query results to obtain the range of query results that meet the query criteria.

9. A point query method based on the LSM-Tree key-value storage system according to any one of claims 1 to 6, characterized in that, The point query method includes: Step S91: Obtain the target key that matches the query conditions; Step S92: Perform a target key query in the Memtable data structure, and if the target key is obtained, proceed to step S97; otherwise, proceed to step S93. Step S93: Perform a target key query in the Immutable Memtable data structure, and if the target key is obtained, proceed to step S97; otherwise, proceed to step S94. Step S94: Query the target value corresponding to the target key in the L0 layer of the LSM-Tree structure. If the target key is obtained based on the target value, proceed to step S96; otherwise, proceed to step S95. Step S95: Locate the range of key values ​​that may contain the target key based on the inverted index of the underlying information, and filter the range of key values ​​that may contain the target key through a Bloom filter, then proceed to step S96. Step S96: Output the query results of points that meet the query conditions.

10. A data writing method based on the LSM-Tree key-value storage system according to any one of claims 1 to 6, characterized in that, The data writing method includes: Write the original data into the Memtable data structure; When the Memtable data structure reaches its capacity limit, the Memtable data structure is converted into an ImmutableMemtable data structure, and a new Memtable data structure is generated to continue to accept the written data. After converting the Immutable Memtable data structure into an ordered string table file, it is stored in the L0 layer of the LSM-Tree structure; In the LSM-Tree structure i Once the number of files in a layer reaches a threshold, compression is performed, and the resulting ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number; The data space is divided into several data fields based on the boundary values ​​of the ordered string table files at the bottom layer of the LSM-Tree structure, and the file numbers of the ordered string table files that intersect with each data field are used to fill the inverted index of the underlying information.

Images