LSM-Tree key value storage system for establishing query index by using underlying information

Abstract

The invention discloses an LSM-Tree key value storage system for establishing a query index by using underlying information, and belongs to the technical field of data storage. According to the system, the boundary of a bottom-layer SSTable of an LSM-Tree is used as an index key, all key value pairs in generated SSTable are traversed through a compression process, and an upper-layer SSTable file number intersecting with the range of the index key is used as an index value to construct an index. According to the invention, the query operation of the LSM-Tree key value storage system is accelerated.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular relates to an LSM-Tree key-value storage system that uses underlying information to establish a query index. Background Art

[0002] With the popularization of the Internet and the gradual development of personal computer (PC) Internet to mobile Internet, the application scenarios of databases are constantly enriched. With the introduction of digital development projects such as industrial Internet, energy Internet, smart city, smart medical care, Internet of Vehicles, financial technology, smart education, and data villages, the scale of data is growing rapidly. The market scale continues to increase, which brings huge challenges to the storage and analysis of data. Efficient and secure storage is the basis for the analysis and processing of massive data. In order to cope with the challenges brought by the increasing scale of data, storage systems and storage hardware are also constantly evolving. As the types of data are gradually enriched and the content is gradually diversified, relational (SQL) databases have gradually developed into non-relational (NoSQL) databases with better scalability since entering the 21st century and have been widely used.

[0003] Non-relational databases include key-value databases, time series databases, graph databases, vector databases, document databases, columnar databases, etc. According to the statistics of CCSA TC601, there are 82 key-value databases, accounting for 23.8% of the 344 non-relational databases, and the highest proportion among non-relational (NoSQL) databases. The characteristic of key-value databases is that they use key-value pairs to store data, and the structure is simple and easy to understand. Typical systems include BigTable, Redis, Memcached, RocksDB, LevelDB, and ForestDB. Among key-value storage databases, the log structure merge tree (LSM-Tree) is the most widely used write-optimized hierarchical data structure in key-value storage systems. It writes data into the internal ordered data block ordered string table (Sorted String Table, referred to as SSTable) in the form of append only logs. Since it fully utilizes the write characteristics of the disk, it greatly improves the write (Put) performance; then, the log structure merge tree (LSM-Tree) converts the deletion and modification operations into write operations, avoiding the query operation before each modification and deletion, and improves the deletion (Delete) and modification (Update) performance in this update method called out-of-place. However, these operations will destroy the relationship between data, making the query (Query) process a difficult problem. Therefore, the log structure merge tree (LSM-Tree) proposes the idea of ​​​​layering, dividing the overall structure of all data into a multi-layer structure with gradually increasing storage capacity, storing data layer by layer from top to bottom; and then through the compression (Compaction) operation, the data of each layer will become ordered, so that point query (Point Query) operations can be performed relatively quickly in each layer through binary search, and at the same time, ordered data is also conducive to range query (Range Query) operations. In order to further improve query performance, the log structure merge tree (LSM-Tree) also adds data ranges and bloom filters to the ordered string table (SSTable) to improve the speed of filtering data blocks during the query process. These features make the log structure merge tree (LSM-Tree) occupy an important position in the key-value storage system.

[0004] However, the design of the log structure merge tree (LSM-Tree) also leads to some new problems, namely read-write amplification and space amplification.

[0005] 1. Read Amplification:

[0006] When performing a point query, since the ordered string tables storing data in each layer are ordered, and the data inside the ordered string table is also in an ordered state, the log structure merge tree (LSM-Tree) will select and read the header file of the ordered string table from top to bottom, and filter the ordered string table according to the file range and Bloom filter recorded in the header file. Since data is written to the log structure merge tree (LSM-Tree) in an append-only manner, the first value found from top to bottom is the latest value. In this way, the problem of multiple versions of values ​​stored in the log structure merge tree (LSM-Tree) can be avoided.

[0007] When performing a range query, since the ordered string tables storing data in each layer are ordered, and the data inside the ordered string tables are also in an ordered state, during a range query, the log structure merge tree (LSM-Tree) will select data blocks within the range of each layer from top to bottom through the header files of the ordered string tables, and finally read all the ordered string tables that meet the requirements into the memory, arrange and integrate them, and finally output the query results.

[0008] 2. Write Amplification:

[0009] Write amplification refers to the ratio of the actual data size written to a traditional hard disk drive (HDD) / solid state drive (SSD) to the data size required by the program to be written. During the compaction operation, data is continuously read out, merged, sorted, and then written, resulting in the actual amount of data written being much larger than the amount required by the program.

[0010] 3. Space Amplification:

[0011] Because all writes are written sequentially and are deleted (Delete) and modified (Update) in an out-of-place update rather than an in-place update (In-Place Update) manner, expired data will not be cleaned up immediately and will continue to occupy space.

[0012] In summary, it is necessary and worthwhile to conduct research in the field of key-value storage based on log-structured merge tree (LSM-Tree) to study how to optimize the existing key-value storage system, retain its excellent performance, and minimize the impact of read amplification, write amplification, and space amplification. It is also necessary to study and implement a key-value storage system with better performance. Summary of the invention

[0013] In view of the above problems, the present invention discloses an LSM-Tree key-value storage system (BIVXDB) that uses underlying information to establish a query index. The system uses the underlying information to establish a bottom-level information inverted index (BottomInformationInVertIndeX, referred to as BIVX) of the query index, which can accelerate the query operation of the LSM-Tree key-value storage system.

[0014] BIVX uses the boundary of the ordered string table, the bottom storage unit file of LSM-Tree, as the index key, and traverses all the key-value pairs in the generated ordered string table through the compression process in the LSM-Tree key-value storage system, and uses the file number of the upper ordered string table that intersects with the range of the index key as the index value to build an index. In the search process, the index is used to first find the range where the target key-value pair exists from the bottom layer of the LSM-tree, and then the file numbers of all ordered string tables that may contain the target value are found through the index, and then the boundary value information of the string table and its own Bloom filter are used to judge whether the target value exists in the table. Using BIVX, the present invention can make full use of the writing advantage of LSM-Tree without sacrificing its query performance.

[0015] The present invention builds BIVXDB, which is a single LSM-Tree key-value storage system combined with BIVX index. By integrating the compression stage and building BIVX incidentally, BIVXDB achieves low write amplification and fast search at the same time.

[0016] The technical solution of the present invention includes the following technical contents.

[0017] An LSM-Tree key value storage system that uses underlying information to establish a query index, the system comprising: a memory data storage module, a disk data storage module of an LSM-Tree structure, and an index module;

[0018] A memory data storage module is used to generate a Memtable data structure for storing data, and when the Memtable data structure is full of data, convert the Memtable data structure into an Immutable Memtable data structure and generate a new Memtable data structure to continue to receive the written data;

[0019] The disk data storage module is used to convert the Immutable Memtable data structure into an ordered string table file and store it in the L0 layer of the LSM-Tree structure. i When the number of files in the layer reaches the threshold, the compression operation is performed and the generated ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number;

[0020] An index module is used to construct an inverted index of underlying information; wherein the index key of the inverted index of the underlying information is the boundary value of the ordered string table file at the bottom of the LSM-Tree structure, and the index value of the inverted index of the underlying information is the file number of the ordered string table file at the bottom of the LSM-Tree structure.

[0021] Furthermore, in the case where the input file of the compression operation is a non-bottom-layer ordered string table file and the output file is an ordered string table file written to the bottom layer, the process of updating the bottom-layer information inverted index includes:

[0022] Get the index key of the output file;

[0023] While inserting the index key of the output file into the underlying information inverted index, the file number corresponding to the input file is deleted from all index values ​​of the underlying information inverted index to obtain an updated underlying information inverted index.

[0024] Furthermore, in the case where the input file of the compression operation is an underlying ordered string table file and the output file is an ordered string table file written to the underlying layer, the process of updating the underlying information inverted index includes:

[0025] Step S31: Obtain the index key of the output file;

[0026] Step S32: determining whether the inverted index of the underlying information has an intersection with the range of the index key of the output file;

[0027] Step S33: If there is an intersection between the inverted index of the underlying information and the range of the index key of the output file, jump to step S34; otherwise, jump to step S38;

[0028] Step S34: backing up the original index key in the bottom-level information inverted index, and deleting the original index key from the bottom-level information inverted index;

[0029] Step S35: inserting the index key of the output file into the underlying information inverted index;

[0030] Step S36: comparing the intersection of the original index key and the index value of the new index key inserted into the bottom-level information inverted index, and copying the index value with the intersection to the new index key to obtain a preliminarily updated bottom-level information inverted index;

[0031] Step S37: Delete the file number corresponding to the input file from all index values ​​of the initially updated bottom-level information inverted index to obtain an updated bottom-level information inverted index;

[0032] Step S38: while inserting the index key of the output file into the bottom-level information inverted index, the index key of the input file is deleted from the bottom-level information inverted index to obtain an updated bottom-level information inverted index.

[0033] Furthermore, in the case where the input file of the compression operation is a non-bottom layer file and the output file is a file written to the non-bottom layer, the process of updating the bottom layer information inverted index includes:

[0034] Construct a tag list according to the index key in the inverted index of the underlying information;

[0035] When the key-value pairs output by the mark compression process intersect with the data areas represented by each index key, the data areas are marked in the mark list;

[0036] The underlying information inverted index is updated according to the mark list, and specific index values ​​in the underlying information inverted index are deleted; wherein the specific index values ​​include: expired index values, index values ​​that do not intersect with boundary values, and index values ​​that do not intersect with the data area.

[0037] Furthermore, in the case where the input file of the compression operation is a bottom-level file and the output file is a file written to a non-bottom-level file, the process of updating the bottom-level information inverted index includes:

[0038] Traversing all key-value pairs outputted during a compression process, wherein the compression process is a compression process in which the input file is a non-bottom layer file and the output file is a file written to the non-bottom layer;

[0039] Insert the key-value pair information into the current underlying information inverted index;

[0040] Delete the information of the input file and back up the information of the input file to the first area larger than the index key to be deleted;

[0041] A maximum value marking information is stored in the back end of the bottom information inverted index for buffering, so that the maximum value marking information is deleted from the nth layer of the LSM-Tree and gradually stored in the MAX area, and then transitioned to the n+1th layer of the LSM-Tree.

[0042] Furthermore, the system also includes: an information synchronization module, which is used to synchronously store index information in the bottom-level information inverted index into a disk.

[0043] A range query method based on the LSM-Tree key-value storage system according to any one of the above claims, comprising:

[0044] Step S71: construct an iterator;

[0045] Step S72: perform a range query on the data in the Memtable data structure using the iterator, and add the screening file obtained based on the Memtable data structure to the iterator, and then jump to step S73;

[0046] Step S73: Use the iterator to perform a range query on the data in the Immutable Memtable data structure, and add the screening file obtained based on the Immutable Memtable data structure to the iterator, and then jump to step S74;

[0047] Step S74: perform a range query on the ordered string table file in the L0 layer of the LSM-Tree structure using the iterator, and add the screening file obtained based on the L0 layer of the LSM-Tree structure to the iterator, and then jump to step S75;

[0048] Step S75: Obtain the file number of the ordered string table file that meets the query condition in the bottom-level information inverted index, and add the filtered file obtained based on the bottom-level information inverted index to the iterator, and then jump to step S76;

[0049] Step S76: Comprehensively obtain the filter file based on the Memtable data structure, the filter file based on the Immutable Memtable data structure, the filter file based on the L0 layer of the LSM-Tree structure, and the filter file based on the inverted index of the underlying information to obtain a range query result that meets the query condition.

[0050] Further, the processing of the ordered string table file includes:

[0051] Filter out the data within the target range in the ordered string table file;

[0052] Merging and / or updating the filtered data to generate preliminary query results;

[0053] The preliminary query results are sorted to obtain range query results that meet the query conditions.

[0054] A point query method based on any of the above-mentioned LSM-Tree key-value storage systems, comprising:

[0055] Step S91: Obtain the target key corresponding to the query condition;

[0056] Step S92: perform a target key query in the Memtable data structure, and when the target key is obtained, jump to step S97; otherwise, jump to step S93;

[0057] Step S93: perform a target key query in the Immutable Memtable data structure, and when the target key is obtained, jump to step S97; otherwise, jump to step S94;

[0058] Step S94: perform a query on the target value corresponding to the target key at the L0 layer of the LSM-Tree structure, and if the target key is obtained based on the target value, jump to step S96; otherwise, jump to step S95;

[0059] Step S95: Locate the key value range that may contain the target key based on the inverted index of the underlying information, and filter the key value range that may contain the target key through the Bloom filter, and then jump to step S96;

[0060] Step S96: Output the point query results that meet the query conditions.

[0061] A method for writing data based on any of the above-mentioned LSM-Tree key-value storage systems, comprising:

[0062] Write the original data into the Memtable data structure;

[0063] When the Memtable data structure reaches the upper capacity limit, the Memtable data structure is converted into an Immutable Memtable data structure, and a new Memtable data structure is generated to continue to receive the written data;

[0064] Convert the Immutable Memtable data structure into an ordered string table file and store it in the L0 layer of the LSM-Tree structure;

[0065] In the LSM-Tree structure, i When the number of files in the layer reaches the threshold, the compression operation is performed and the generated ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number;

[0066] The data space is divided into several data domains based on the boundary values ​​of the ordered string table files at the bottom of the LSM-Tree structure, and the file numbers of the ordered string table files that have intersections with each data domain are used to fill the bottom information inverted index.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. High-performance point query and range query capabilities.

[0069] The present invention divides the entire data domain into multiple areas based on the boundary information of the ordered string table file stored in the bottom layer of the LSM-Tree as the index key; and compares the key-value pairs in the upper ordered string table with the divided data domain, and records the file number of the ordered string table that has a clear intersection with the bottom data domain as the index value in the index. In the process of point query or range query, the present invention can directly filter out the ordered string table file that needs to be queried and read through the range information stored in the index, so that the original layer-by-layer traversal process of the LSM-Tree can be skipped, which greatly saves the time overhead of the query stage and can significantly improve the point query and range query capabilities of the system.

[0070] 2. Low-overhead indexing process.

[0071] The present invention utilizes the compression process of LSM-Tree to establish the index. The reason for establishing the index in this process is: first, it will delete expired files and insert new files, and will centrally modify the files in the original database. At this time, the index needs to be updated synchronously; second, in this process, the ordered string table will be read from the disk into the memory, and the key-value pairs therein will be traversed. The window period during which all the key-value pairs in the ordered string table are traversed can enable the present invention to accurately record the information in the index, and can avoid the original search misreading problem caused by only recording the boundary values ​​of the ordered string table and the Bloom filter. Based on the above two reasons, the present invention establishes the index in the compression stage, and because the index establishment process is integrated in the compression stage, a large amount of new file I / O overhead is not introduced, so the system realizes a low-cost, high-return index construction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is the overall structure of BIVXDB.

[0073] Figure 2 It is the compression flow chart of BIBO (bottom-in-bottom-out) case in BIVXDB.

[0074] Figure 3This is the compression flow chart for UIUO (Unbottom-In-Unbottom-Out) case in BIVXDB.

[0075] Figure 4 This is an introduction to the occurrence of BIUO (Bottom-In-Unbottom-Out) in BIVXDB.

[0076] Figure 5 This is the compression flow chart for the BIUO (bottom-in-unbottom-out) case in BIVXDB.

[0077] Figure 6 This is the BIVXDB range query process.

[0078] Figure 7 It is the BIVXDB point query process.

[0079] Figure 8 This is how the write performance of different systems changes with the amount of data.

[0080] Fig. 9 This is how the point query performance of different systems changes with the amount of data.

[0081] Fig.10 is how the range query performance of different systems varies with the range query length.

[0082] Fig.11 This is how different systems perform on the YCSB default load. DETAILED DESCRIPTION

[0083] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0084] The overall structure of the present invention is as follows Figure 1As shown, the present invention retains most of the structure of RocksDB, so when data is written, it will first be written into Memtable. When Memtable is full, it will be converted into ImmutableMemtable, and a new Memtable will be created to continue to receive the written data. Immutable Memtable will be refreshed to the L0 layer in the disk in the background and converted into an ordered string table file for storage. In Rocksdb, there will be multiple ordered string tables in the L0 layer, and there will be intersections and disorder between these files. Afterwards, when the number of files in the L0 layer reaches the threshold, the Compaction process will be triggered, and these disordered ordered string table files will be re-sorted and merged, and stored in the L1 layer. Afterwards, the data of each layer will be compacted when the capacity reaches the threshold, so the ordered string table files in each layer after L1 are ordered and have no intersection. Because of this feature of RocksDB, the present invention will ignore the L0 layer when indexing and index other layers. This study stores BIVX in DRAM and modifies the original compression process of Rocksdb. By using the window period of reading all the data in the ordered string table file into the memory during the process, all the data in the ordered string table file involved in the compression operation is divided according to the underlying index and stored in the index of the corresponding data area of ​​the underlying layer. In this way, a global index of the log structure merge tree (LSM-Tree) is established. Since the update of BIVX is performed with the compression operation, BIVX will be updated at any time with the change of the scale and number of layers of the log structure merge tree (LSM-Tree).

[0085] 1. BIVX Design

[0086] BIVX is the abbreviation of Bottom Information Invert Index. BIVX is implemented using the Map data structure in C++. When a BIVX is established for an LSM-Tree, the corresponding relationship between the two is as follows: Figure 1As shown. The present invention divides the entire data space into 2n+1 areas according to the boundary values ​​of n ordered string table files stored in the bottom layer of LSM-Tree. Afterwards, BIVX will use the file numbers of the ordered string table files that have intersections with each data domain to fill the index to achieve the purpose of fast search. The process of judging whether there is an intersection and updating BIVX is performed when all the key-value pairs of the generated ordered string table files are traversed during the compression phase. The purpose of traversing all key-value pairs during the compression process to make judgments is to avoid misjudgments that may occur when judging only by the boundary values ​​of the generated ordered string table files. Among them, this study calls the data area used for indexing index keys (Index-Keys), and the stored file numbers are index values ​​(Index-Values). An index key and its corresponding index value constitute an index item (Index-Item).

[0087] In BIVXDB, when an LSM-Tree is created, an empty BIVX is created at the same time; afterwards, BIVX is updated during the compression phase as LSM-Tree data is continuously written. During the compression process, whether the input and output files exist in the bottom layer will affect whether the index keys in BIVX need to be modified. Therefore, the present invention divides compression into four cases based on whether the input files (SSTables to be compressed in the RocksDB system) and the output files are in the bottom layer:

[0088] a) The input is a non-bottom-level file, and the generated file is written into the bottom level (UIBO).

[0089] When the input file of the compression process is the upper layer file and the output file is the lower layer, it means that there is no intersection between the newly generated file and the existing lower layer file, so it will not affect the index keys that have been written. Therefore, in this case, you only need to directly insert the file boundary value newly written to the lower layer into BIVX, and delete the file number of the file involved in the compression process from each index item.

[0090] b) The input is the bottom layer file, and the generated file is also written into the bottom layer (BIBO).

[0091] When there is an underlying file in the input file during the compression process, and the output file is also written to the underlying file, BIVX needs to go through two steps: deleting the old index items and inserting the new index items. Among them, how to transfer the information in the old index items to the new ones has become a problem that needs to be solved. Figure 2 The process of compressing S4 at the bottom layer (nth layer) of the LSM-Tree and S10 at the n-1th layer, and storing the output files S21 and S22 back to the bottom layer of the LSM-Tree. Figure 2 The dotted part in the diagram is shown in Figure 1. It is mainly divided into three processes: first, the data of S4 and S10 will be read into the memory and then merged and sorted; then the data will be stored in S21 and S22 respectively; finally, the information of S21 and S22 will be updated to the LSM-Tree. Figure 2 This process is shown in detail below the dotted line.

[0092] Step: b1: Back up the index key to be deleted and delete the expired data in the index key.

[0093] Since S4 is at the bottom of the LSM-Tree, any change to it will affect the data area division in BIVX. Therefore, the system first selects the index items to be deleted from BIVX (1: S4 & 21: S20, S10, S4), and deletes them from BIVX after backing them up.

[0094] Step b2: Insert new index key.

[0095] Since the newly generated files S21 and S22 will be output to the bottom layer, they will also affect the division of the data area. Therefore, the system should also insert their boundary values ​​into BIVX in sequence. The index entries in the black box in the figure are all newly inserted.

[0096] Step b3: Compare the intersection between the new index area and the old area, and copy the index values ​​with intersection to the new index.

[0097] The system will traverse the backed-up entries and insert all index data into the entries with intersections in the new index. In this way, the information in the original BIVX is completely inherited.

[0098] Step b4: Delete the expired index value.

[0099] The system needs to compare whether the files corresponding to all index values ​​are within the range corresponding to the index key, and delete the expired and redundant data therein to facilitate the subsequent search process and reduce the space overhead of BIVX.

[0100] c) The input is a non-bottom-level file, and the generated file is also output to the non-bottom-level file (UIUO).

[0101] When the ordered string table files in layers other than the bottom layer are compressed, the update of BIVX is limited to the index value. Therefore, its update process is mainly concentrated in the traversal and output process in the compression phase.

[0102] Figure 3An example is used to illustrate the above process. When S20 and S10, which are not at the bottom of the LSM-Tree, are merged, the corresponding BIVX update steps are as follows: Figure 3 Shown in the lower half of the dotted line.

[0103] Step c1: store a marker list according to BIVX to mark whether the compression process output has an intersection with each range.

[0104] The system needs to build a tag list according to the index keys in BIVX. The tag list is used to mark whether the key-value pairs output by the compression process have intersections with the data areas represented by each index key.

[0105] Step c2: When the compression process outputs key-value pairs in sequence, determine whether they intersect with the ranges in BIVX. If so, mark them in the list.

[0106] During the LSM-Tree compression process, the traversal and output stage determines whether each key-value pair has an intersection with the data area represented by each index key. If there is an intersection, it is marked in the corresponding area. In this process, in order to reduce the judgment overhead, the system first finds the first value (21) in the index that is not less than the first output key (4 in the example in the figure) and marks it. Subsequently, the system will no longer modify the list until it encounters the first value (26) greater than 21 in the output, and then finds the first value (38) not less than it from the list again and marks it. Repeat the above steps until all output keys have been traversed.

[0107] Step c3: Update BIVX according to the tag list and delete the index values ​​that are expired and do not intersect with the boundary value and range.

[0108] According to the records in the list, filter out the information corresponding to the data range, and update the number of the ordered string table to the corresponding index item.

[0109] d) The input is a bottom-level file, but the output file is not written to the bottom-level location (BIUO).

[0110] As the amount of data continues to increase, when the number of layers of LSM-Tree changes from n to n+1, the "bottom layer" will gradually transition from the original n layers to the n+1 layer. During this process, BIVX will also be continuously updated with compression, gradually transitioning the range from the n layer to the n+1 layer. In this process, BIVX will simultaneously have data information from the n+1 layer (new bottom layer) and the n layer (original bottom layer). Since each layer of LSM-Tree is ordered and non-intersecting, but this principle cannot be guaranteed between different layers, if this process is not handled properly, the information at the bottom layer will be destroyed and will not be able to be effectively indexed. This process is like Figure 4shown.

[0111] Since the scope of the n+1 layer will gradually expand in this process, its intersection with the nth layer will also become larger and larger. What the present invention needs to do is to gradually inherit the information of the nth layer stored in BIVX in this process, and gradually update and assign it to the n+1th layer.

[0112] A fourth situation may occur when compression is performed during this transition phase: the input file of the compression process contains the bottom layer file located at the nth layer, but the output file will be stored back to this area again at the same time. In this case, the index key information is still constructed in BIVX with the information of the input file, but the target layer of the output file is no longer the bottom layer in the current situation. If the index key of BIVX is updated in the update method of the UIUO case, the original data will be lost. Therefore, the present invention proposes a new composite update method to update BIVX.

[0113] First, the present invention traverses all the key-value pairs output during the compression process in a UIUO manner, and inserts their information into the current BIVX. Afterwards, in order to gradually reduce the proportion of the nth layer in the index key in the current BIVX, the present invention chooses to delete the information of the input file and backs up its information to the first area that is larger than the index key to be deleted. In this process, since BIVX will select an area that is less than or equal to the index key when searching, choosing to back up the information to an index key that is larger than it will not affect the query. By combining the two update methods, the present invention can simultaneously retain old information, update information, and reduce the proportion of nth layer files in the BIVX index key. At the same time, since the present invention will store a maximum value mark (MAX) information in the BIVX backend for buffering, the information will be continuously deleted from the nth layer and gradually stored in the MAX area, and then slowly transition to the n+1th layer. The process is roughly as follows Figure 5 shown.

[0114] Figure 5 It shows the whole process of updating BIVX when the system compresses S23 and S20 and places the output files in the nth layer, such as Figure 5 As shown, the system first reads out all the data of the two and traverses them in turn. At this time, according to the traversal information, the new file number will be inserted into index item 97, so that its index value becomes S23, S28 and other original indexes, and the record of expired data S23 will be deleted from the index. However, it is found that the output file is at the nth layer, so the information of the two index items 88&97 should be backed up to the first area larger than the index key to be deleted, that is, the ∞ index item (MAX area), and the two index items 88&97 should be deleted. So far, the compression process in the BIUO case is completely completed.

[0115] 2. Analysis of BIVXDB operation process

[0116] The core operations of the BIVXDB system include writing (Put()), range query (Scan()), and point query (Get()) processes, which involve the management and retrieval of key-value pairs in the database. In BIVXDB, data update and deletion operations do not directly modify or remove old data items, but instead adopt a version control and mark deletion-based approach to handle these changes. Specifically, when an update operation is performed, the system writes a new data version, which will contain the updated value while retaining the old version of the data with the same key. This approach allows BIVXDB to support efficient data updates while providing historical versions of data. Each version of the data item is assigned a unique timestamp or version number to distinguish the order of different versions. For deletion operations, BIVXDB also does not immediately remove the corresponding data item from physical storage. Instead, it writes a special deletion marker to indicate that the key-value pair has been deleted. When performing a read operation, the system checks these deletion markers and filters out the data items marked as deleted before returning the results. This approach simplifies the processing flow of deletion operations and makes future data recovery possible. Through such a design, BIVXDB not only improves the flexibility and efficiency of data operations, but also maintains the traceability of historical versions of data, meeting the needs of data management in complex application scenarios. In addition, by utilizing version control and deletion markers, BIVXDB can reduce data redundancy and waste of storage space while ensuring operational efficiency, further optimizing the overall performance of the system.

[0117] a) Range query process

[0118] The introduction of BIVX significantly improves the efficiency of LSM-Tree-based database systems when performing range queries. Traditionally, range searches require traversing all layers of the LSM-Tree layer by layer, performing a global search on the ordered string table files of each layer, and then determining whether these ordered string table files contain the target range of data one by one. This process is not only inefficient, but also significantly slows down as the amount of data grows. In contrast, the design of BIVX allows the system to skip this layer-by-layer global search and directly use BIVX for efficient range queries.

[0119] The range query process of BIVXDB is as follows Figure 6As shown. In the specific implementation, the range query process starts with building an iterator, which first includes the Memtable in memory and the ImmutableMemtable converted to a read-only state. This step ensures that the latest written data can be queried immediately, maintaining the real-time nature of the data. Next, considering that BIVX does not contain information about files in the L0 layer (the files stored in this layer are unordered and there are intersections between files), the system needs to perform special processing on the L0 layer, that is, all ordered string table files in the L0 layer are also included in the search range of the iterator.

[0120] Furthermore, BIVX provides direct index information for each fine-grained data area, including the file number that stores the data interval. Through BIVX, the system can directly locate the specific file containing the target range of data without traversing other irrelevant files, which greatly reduces the time required for searching. Subsequently, the system directly reads the corresponding ordered string table files from the disk based on these file numbers and adds them to the iterator.

[0121] In the last step, the system performs comprehensive processing on the data of all related ordered string table files in the iterator, including filtering out the data within the target range, merging duplicate or updated records, and sorting the results. This not only ensures the accuracy of the query results, but also greatly improves the speed of range queries.

[0122] In this way, BIVX provides a more efficient range query solution for LSM-Tree-based key-value storage systems, significantly reducing query latency and improving the system's ability to process large-scale data.

[0123] b) Point query process

[0124] In BIVXDB, the point query process achieves efficient and accurate key-value pair lookup through a series of refined steps. The process is as follows: Figure 7As shown in the figure. This process starts with the Memtable in memory and its read-only version ImmutableMemtable. The latest written data is stored in these two structures. Prioritizing queries here can quickly obtain the most recent write, update, and delete information. If the target key is not found in these two structures, it is further filtered in the first layer of storage on the disk, that is, the ordered string table file of the L0 layer. Since the files of the L0 layer may overlap, this step requires checking all the files of this layer one by one. If the target value is still not found in the L0 layer, it is then searched in other layers. Since the ordered string table files in other layers are all recorded in the BIVX index, the query operation can use the BIVX index to quickly locate the key value range that may contain the target key, narrowing the search range to several files. In order to further improve the accuracy of the search, the query process retains the Bloom filter and tries to use it to filter the files determined by the BIVX index again to quickly exclude those files that may not contain the target key. Finally, the target key is found from the filtered files and the query results are output. If the key exists in the database, the corresponding value is returned; if not, an indication of not found is returned. This efficient query process avoids the previous LSM-Tree layer-by-layer search process, ensuring that BIVXDB can respond to point query requests quickly and accurately even when processing large amounts of data.

[0125] c) Writing process

[0126] BIVXDB adopts the data writing strategy of RocksDB. In the initial stage, data is first written to the in-memory Memtable, a sort-based data structure, to ensure fast writing speed. When the Memtable reaches its capacity limit, it is converted to a read-only Immutable Memtable, and a new Memtable is created to continue receiving new written data. This design avoids interruption of writing operations. Next, the Immutable Memtable is asynchronously flushed to the L0 layer on the disk, and the data is stored in the form of an ordered string table file, which is a file format that supports efficient range queries. After writing, BIVXDB performs a compression operation, which is a background process designed to organize and merge ordered string table files to reduce storage usage and improve query efficiency. In addition, during the compression process, BIVXDB updates the BIVX index, which is its core component responsible for providing efficient data retrieval capabilities. This approach not only combines the speed advantages of in-memory computing and the large capacity advantages of disk storage, but also maintains efficient query performance even in frequent data update and deletion operations by timely updating the BIVX index, thereby optimizing the data writing and query process and improving the overall performance and scalability of the database.

[0127] d) Crash consistency

[0128] The mechanism for achieving crash consistency in BIVXDB is similar to that of RocksDB, mainly using the WAL (Write Ahead Log) strategy. This strategy ensures that in the event of a crash, any unfinished data operations can be recovered through logs, thereby ensuring data integrity and consistency. However, for the BIVX index in BIVXDB, since its update operation is not incremental, the traditional WAL mechanism cannot be directly applied. Therefore, BIVXDB adopts a special strategy: after each BIVX index update is completed, the updated index information is immediately synchronized and stored to disk. This step is for BIVXDB to ensure that the consistency and integrity of the index can be maintained even in the event of a system crash, further enhancing the reliability of the system.

[0129] In summary, the present invention proposes a "BIVX" index constructed using underlying information to improve the query performance of LSM-Tree. By taking advantage of the orderly, large number and densely distributed characteristics of the underlying files, a dynamic underlying information index is cleverly constructed, so that LSM-Tree no longer needs to search layer by layer when querying, which significantly improves the query performance of the storage system, especially the range query performance.

[0130] The present invention realizes data traversal through the reading process of storage files in the compression stage of LSM-Tree, accurately and efficiently records information in the index, and realizes index establishment without introducing additional reading and writing overhead.

[0131] The following is a specific experiment to illustrate the LSM-Tree key-value storage system for establishing a query index using underlying information provided by the present invention.

[0132] 1. Dataset

[0133] When evaluating write performance and point query performance, technicians mainly used the benchmark tools provided by RocksDB and LevelDB for testing, and used YCSB as an auxiliary tool for verification. However, when testing range query performance, technicians faced some challenges. The range query interfaces provided by RocksDB and LevelDB are both iterator-based, while BIVXDB implements its own proprietary scan interface, and it is impossible to directly use the iterator interface in the benchmark tool to test range query performance. In order to meet this challenge, the present invention adopts the method of using YCSB for testing.

[0134] YCSB is a general cloud service benchmarking tool that provides a variety of workload types, including range query operations. The following table shows the default load of YCSB. Through YCSB, the present invention can customize the workload of range query operations to simulate different query scenarios and measure the performance of BIVXDB in these scenarios, as shown in Table 1.

[0135]

[0136] Table 1

[0137] 2. Evaluation indicators

[0138] First, dbbench is used to test the write performance and point query performance of the present invention (BIVXDB) and some existing solutions (LevelDB and RocksDB). Then, YCSB is used to test the range query performance of each solution. Finally, YCSB is used to test the experimental performance of each solution under the default workload.

[0139] 3. Experimental results

[0140] from Figure 8 It was observed that as the amount of data gradually increased from 20GB to 80GB, the write performance of BIVXDB, RocksDB, and LevelDB all showed a certain degree of downward trend. This phenomenon is common in storage systems, because as the amount of data increases, the system must process more data files and indexes, which in some cases increases the complexity of the data write path, especially when compaction operations occur frequently. Despite this, BIVXDB always maintains better write performance than RocksDB and LevelDB throughout the process of data growth. This result shows that although BIVXDB introduced additional operations in the compaction phase to optimize query performance, these improvements did not have a significant negative impact on the overall write performance. On the contrary, BIVXDB can maintain or even improve write performance while achieving a significant improvement in query performance through an effective indexing strategy.

[0141] according to Fig. 9, it is observed that BIVXDB continues to outperform RocksDB in terms of read performance, especially as the amount of data continues to grow, this performance difference becomes more and more significant. LevelDB's query performance shows better read performance when the amount of data is small, which is better than the other two database systems. This is mainly due to its simple and efficient design and optimized processing of small-scale data sets, which enables quick positioning and retrieval of required information when processing less data. However, as the amount of data continues to increase, LevelDB's performance advantage begins to weaken. This is because the log structure merge tree (LSM-Tree) mechanism used by LevelDB faces more frequent merge operations and larger read amplification problems when processing large-scale data, which gradually reduce query efficiency. Therefore, although LevelDB shows strong read performance in the early stage, as the amount of data grows, BIVXDB's performance improvement gradually emerges, and finally shows superior query performance in large-scale data processing. This performance is attributed to the unique design of BIVXDB's index structure. Since the files at level 0 in LSM-Tree are unordered and have intersections with each other, BIVX starts to store information from the first level. When the amount of data is small, a large amount of data is concentrated in the 0th layer. At this time, BIVXDB's efficient indexing mechanism cannot be fully utilized, resulting in a small performance gap compared with RocksDB in the initial comparison, and relatively poor performance compared with LevelDB. However, as the amount of data grows, more data gradually migrates to lower layers, and BIVXDB's indexing mechanism begins to play a key role, greatly improving the query speed. At the same time, LevelDB and RocksDB use a top-down layer-by-layer search strategy. In comparison, BIVXDB's query efficiency from the bottom up using indexes will be higher, especially when processing large-scale data sets, its performance advantage is more obvious, which fully demonstrates BIVXDB's advanced design and efficient response to query needs in a big data environment.

[0142] like Fig.10As shown in the figure, BIVXDB shows an overwhelming performance advantage over the other two key-value storage systems, with its range query throughput almost 2 to 6 times that of the other two. RocksDB shows a trend of gradually decreasing query performance as the query length increases. This is because RocksDB relies on global iterators when performing range queries, and the overhead of iterators increases with the query length, resulting in a decrease in range query performance. LevelDB performs better than RocksDB, but the specific trend is basically the same as RocksDB. The performance of BIVXDB is more complex, showing a fluctuating pattern. The reason behind this fluctuating performance is that BIVXDB uses an inverted index mechanism for the underlying information. The inverted index is established based on the maximum and minimum values ​​of the underlying files, so that when performing range queries, BIVXDB can use the index to quickly exclude a large number of irrelevant files and only access and read files that may contain target data, greatly improving query efficiency. However, BIVXDB's inverted index strategy also brings certain limitations. When the length of the query range is not an integer multiple of the length of the underlying file, the indexing mechanism may not be able to fully exert its performance advantages, resulting in a slight decrease in query efficiency. This explains the wave-like performance fluctuations exhibited by BIVXDB in the experiment.

[0143] From the experimental results Fig.11It can be observed that BIVXDB shows significant performance improvement compared with RocksDB in most test scenarios, especially in non-update-intensive workloads. This performance advantage is mainly attributed to the significant improvement of point query performance by the BIVXDB structure. BIVXDB significantly improves query efficiency by establishing an underlying information index structure and utilizing it in the point query process, especially when processing a large number of point query operations, it can quickly locate data, thereby reducing data access latency. However, in the data loading phase and the experimental scenario of Workload D, which is mainly based on update operations, the performance of BIVXDB is slightly lower than that of RocksDB. This is mainly because when BIVXDB builds and updates BIVX, it needs to introduce additional comparison and record operations in the compaction phase to maintain the index, which sacrifices write performance to a certain extent. Especially in an update-intensive environment, frequent update operations will lead to frequent changes in the index, increasing the maintenance burden of the system, thereby affecting the overall write efficiency. In terms of range queries, BIVXDB's performance advantage over RocksDB is particularly obvious. In the test of Workload E, BIVXDB's performance is more than six times that of RocksDB. This significant improvement is due to BIVXDB's design for optimizing range queries. By building an index structure and using it to directly read files, BIVXDB can handle range query requests more efficiently, greatly reducing the access overhead of RocksDB using only iterators to traverse each layer of files, thereby improving query efficiency. In summary, although BIVXDB's write performance will be affected in certain specific scenarios, its performance improvement in key operations such as point queries and range queries fully demonstrates the effectiveness of BIVXDB's improvement measures.

[0144] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. An LSM-Tree key-value storage system that uses underlying information to establish a query index, characterized in that: The system comprises: a memory data storage module, a disk data storage module of LSM-Tree structure and an index module; A memory data storage module is used to generate a Memtable data structure for storing data, and when the Memtable data structure is full of data, the Memtable data structure is converted into an Immutable Memtable data structure, and a new Memtable data structure is generated to continue to receive the written data; The disk data storage module is used to convert the Immutable Memtable data structure into an ordered string table file and store it in the L0 layer of the LSM-Tree structure. i When the number of files in the layer reaches the threshold, the compression operation is performed and the generated ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number; An index module is used to construct an inverted index of underlying information; wherein the index key of the inverted index of the underlying information is the boundary value of the ordered string table file at the bottom of the LSM-Tree structure, and the index value of the inverted index of the underlying information is the file number of the ordered string table file at the bottom of the LSM-Tree structure.

2. The system according to claim 1, characterized in that In the case where the input file of the compression operation is a non-bottom-layer ordered string table file and the output file is an ordered string table file written to the bottom layer, the process of updating the bottom-layer information inverted index includes: Get the index key of the output file; While inserting the index key of the output file into the underlying information inverted index, the file number corresponding to the input file is deleted from all index values ​​of the underlying information inverted index to obtain an updated underlying information inverted index.

3. The system according to claim 1, characterized in that In the case where the input file of the compression operation is an underlying ordered string table file and the output file is an ordered string table file written to the underlying layer, the process of updating the underlying information inverted index includes: Step S31: Obtain the index key of the output file; Step S32: determining whether the inverted index of the underlying information has an intersection with the range of the index key of the output file; Step S33: If there is an intersection between the inverted index of the underlying information and the range of the index key of the output file, jump to step S34; otherwise, jump to step S38; Step S34: backing up the original index key in the bottom-level information inverted index, and deleting the original index key from the bottom-level information inverted index; Step S35: inserting the index key of the output file into the underlying information inverted index; Step S36: comparing the intersection of the original index key and the index value of the new index key inserted into the bottom-level information inverted index, and copying the index value with the intersection to the new index key to obtain a preliminarily updated bottom-level information inverted index; Step S37: Delete the file number corresponding to the input file from all index values ​​of the initially updated bottom-level information inverted index to obtain an updated bottom-level information inverted index; Step S38: while inserting the index key of the output file into the bottom-level information inverted index, the index key of the input file is deleted from the bottom-level information inverted index to obtain an updated bottom-level information inverted index.

4. The system according to claim 1, characterized in that In the case where the input file of the compression operation is a non-bottom layer file and the output file is a file written to the non-bottom layer, the process of updating the bottom layer information inverted index includes: Construct a tag list according to the index key in the inverted index of the underlying information; When the key-value pairs output by the mark compression process intersect with the data areas represented by each index key, the data areas are marked in the mark list; The underlying information inverted index is updated according to the mark list, and specific index values ​​in the underlying information inverted index are deleted; wherein the specific index values ​​include: expired index values, index values ​​that do not intersect with boundary values, and index values ​​that do not intersect with the data area.

5. The system according to claim 4, characterized in that In the case where the input file of the compression operation is a bottom-level file and the output file is a file written to a non-bottom-level file, the process of updating the bottom-level information inverted index includes: Traversing all key-value pairs outputted during a compression process, wherein the compression process is a compression process in which the input file is a non-bottom layer file and the output file is a file written to the non-bottom layer; Insert the key-value pair information into the current underlying information inverted index; Delete the information of the input file and back up the information of the input file to the first area larger than the index key to be deleted; A maximum value marking information is stored in the back end of the bottom information inverted index for buffering, so that the maximum value marking information is deleted from the nth layer of the LSM-Tree and gradually stored in the MAX area, and then transitioned to the n+1th layer of the LSM-Tree.

6. The system according to any one of claims 1 to 5, characterized in that: The system further comprises: an information synchronization module, wherein the information synchronization module is used to synchronously store index information in the bottom-level information inverted index into a disk.

7. A range query method based on the LSM-Tree key-value storage system according to any one of claims 1 to 6, characterized in that: The range query method comprises: Step S71: construct an iterator; Step S72: perform a range query on the data in the Memtable data structure using the iterator, and add the screening file obtained based on the Memtable data structure to the iterator, and then jump to step S73; Step S73: Use the iterator to perform a range query on the data in the Immutable Memtable data structure, and add the screening file obtained based on the Immutable Memtable data structure to the iterator, and then jump to step S74; Step S74: perform a range query on the ordered string table file in the L0 layer of the LSM-Tree structure using the iterator, and add the screening file obtained based on the L0 layer of the LSM-Tree structure to the iterator, and then jump to step S75; Step S75: Obtain the file number of the ordered string table file that meets the query condition in the bottom-level information inverted index, and add the filtered file obtained based on the bottom-level information inverted index to the iterator, and then jump to step S76; Step S76: Comprehensively obtain the filter file based on the Memtable data structure, the filter file based on the Immutable Memtable data structure, the filter file based on the L0 layer of the LSM-Tree structure, and the filter file based on the inverted index of the underlying information to obtain a range query result that meets the query condition.

8. The range query method according to claim 7, characterized in that: The processing of the ordered string table file includes: Filter out the data within the target range in the ordered string table file; Merging and / or updating the filtered data to generate preliminary query results; The preliminary query results are sorted to obtain range query results that meet the query conditions.

9. A point query method based on the LSM-Tree key-value storage system according to any one of claims 1 to 6, characterized in that: The point query method comprises: Step S91: Obtain the target key corresponding to the query condition; Step S92: perform a target key query in the Memtable data structure, and when the target key is obtained, jump to step S97; otherwise, jump to step S93; Step S93: perform a target key query in the Immutable Memtable data structure, and when the target key is obtained, jump to step S97; otherwise, jump to step S94; Step S94: perform a query on the target value corresponding to the target key at the L0 layer of the LSM-Tree structure, and if the target key is obtained based on the target value, jump to step S96; otherwise, jump to step S95; Step S95: Locate the key value range that may contain the target key based on the inverted index of the underlying information, and filter the key value range that may contain the target key through the Bloom filter, and then jump to step S96; Step S96: Output the point query results that meet the query conditions.

10. A method for writing data based on the LSM-Tree key-value storage system according to any one of claims 1 to 6, characterized in that: The data writing method comprises: Write the original data into the Memtable data structure; When the Memtable data structure reaches the upper capacity limit, the Memtable data structure is converted into an ImmutableMemtable data structure, and a new Memtable data structure is generated to continue to receive the written data; Convert the Immutable Memtable data structure into an ordered string table file and store it in the L0 layer of the LSM-Tree structure; In the LSM-Tree structure, i When the number of files in the layer reaches the threshold, the compression operation is performed and the generated ordered string table is stored in the LSM-Tree structure. i+1 Layer; where i is a natural number; The data space is divided into several data domains based on the boundary values ​​of the ordered string table files at the bottom of the LSM-Tree structure, and the file numbers of the ordered string table files that have intersections with each data domain are used to fill the bottom information inverted index.

Images