An LSMT read optimization method based on RadixSpline to accelerate key-value pair retrieval

Abstract

This invention discloses an LSMT read optimization method based on RadixSpline to accelerate key-value pair retrieval, comprising: constructing and training a RadixSpline model; the RadixSpline model includes a prefix table and a model array; the prefix table is used to determine the prefix of newly added model array elements while constructing the model array, and when the prefix of a newly added model array element is inconsistent with the prefix of an existing model array element, the new prefix and the index of the model array element it points to are recorded in the prefix table; the elements in the model array are consistent with the elements in the dataset; the dataset is used to store key-offset pairs; the trained RadixSpline model is integrated into the LSM-Tree architecture, so that the RadixSpline model replaces the original key-value pair retrieval method within the SSTable; the error range of the key-value pair offset is obtained by prediction using the RadixSpline model, and then key-value pairs are retrieved in the SSTable by linear search within the offset error range.

Description

Technical Field

[0001] This invention belongs to the field of database system storage technology, and specifically relates to an LSMT read optimization method based on RadixSpline to accelerate key-value pair retrieval. Background Technology

[0002] Log-Structured-Merge Tree (LSMT) is a low-level storage engine structure that adapts to disk hardware characteristics, converting a large number of random writes into sequential writes to achieve high-efficiency write capabilities. It is widely used in data-intensive systems. Traditional LSMT stores keys and values ​​together, leading to severe write amplification problems during inter-level merging due to duplicate value writes to disk. To address this issue, Wisckey employs a key-value separation design, storing values ​​in separate value log files and storing only pointers to values ​​in the SSTable, significantly reducing the amount of disk writes during inter-level merging.

[0003] Learning indexes apply artificial intelligence concepts to database systems, replacing traditional database indexes with machine learning models. They maintain the mapping between keys and their locations, predict the offset of input keys, and ultimately limit the lookup of key-value pairs to a certain error range. If the key exists within this range, its actual location can be found through linear search. In Wisckey, the SSTable stores pointers of equal size, and the data in the SSTable is sorted and unchanged, meeting the application requirements of learning indexes. RadixSpline is a learning index model that does not support updates, but it has the advantage of building the model only once during data traversal.

[0004] Finding an existing key-value pair in an SSTable involves loading the file, searching, and reading the value. Because Wisckey separates key and value pairs, the SSTable storing the same number of key-value pairs occupies significantly less space than a non-key-value separated SSTable, allowing for substantial caching in memory. This greatly improves the time spent searching on high-speed storage devices, and this invention is based on this read-optimized design. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an LSMT read optimization method based on RadixSpline to accelerate key-value pair retrieval.

[0006] A first aspect of this invention provides an LSMT read optimization method based on RadixSpline to accelerate key-value pair retrieval, the method comprising the following steps:

[0007] Construct and train a RadixSpline model; the RadixSpline model includes a prefix table and a model array; the prefix table is used to determine the prefix of newly added model array elements while constructing the model array. When the prefix of a newly added model array element is inconsistent with the prefix of an existing model array element, the new prefix and the index of the model array element it points to are recorded in the prefix table; the elements in the model array are consistent with the elements in the dataset; the dataset is used to store key-offset pairs.

[0008] The trained RadixSpline model is integrated into the LSM-Tree architecture, allowing the RadixSpline model to replace the original key-value pair retrieval method within the SSTable;

[0009] The error range of key-value pair offsets is obtained by predicting using the RadixSpline model, and then key-value pairs are retrieved in the SSTable by linear search within the error range of the offsets.

[0010] Furthermore, integrating the trained RadixSpline model into the LSM-Tree architecture includes:

[0011] At each point when the log structure merge tree SSTable is generated, the list of key-value pairs of the generated log structure merge tree SSTable is sent to the RadixSpline model for storage;

[0012] Remove the original read amplification optimization structure of the SSTable structure, including removing the binary search block, filter block, and LSM-Tree iterator.

[0013] Furthermore, integrating the trained RadixSpline model into the LSM-Tree architecture also includes:

[0014] The key-value pair list of the log structure merge tree SSTable is processed by subtracting the maximum and minimum values ​​in the generated log structure merge tree SSTable and recording the number of prefix 0s in the difference, commonBits. The prefix bits of the keys recorded in the prefix table are calculated starting from the commonBits bit from the most significant bit.

[0015] Furthermore, the error range for predicting key-value pair offsets using the RadixSpline model includes:

[0016] For an input key, the offset is predicted using the RadixSpline model, resulting in an error range [lower, upper]; where lower is the first lower bound of error and upper is the first upper bound of error.

[0017] Divide the first lower error limit (lower) and the first upper error limit (upper) by the number of key-value pairs in the block to obtain the second lower error limit (idxLower) and the second upper error limit (idxUpper). Check the traversed blocks in the range [idxLower, idxUpper] and perform a binary search on each traversed block until the query result is obtained.

[0018] Several key-value pairs are stored in a block.

[0019] Furthermore, the error range for obtaining key-value pair offsets through prediction using the RadixSpline model also includes:

[0020] Determine the relationship between the first lower bound (lower) and the maximum value of the log structure merge tree (SSTable). If the first lower bound (lower) is greater than the log structure merge tree (SSTable), it indicates that the input key is not in the log structure merge tree (SSTable).

[0021] A second aspect of the present invention provides an electronic device including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described LSMT read optimization method based on RadixSpline for accelerated key-value pair retrieval.

[0022] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described LSMT read optimization method based on RadixSpline for accelerated key-value pair retrieval.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] 1. The method proposed in this invention reduces the retrieval time of key-value pairs in SSTable and reduces the overall read operation latency of the merged tree of the key-value separated log structure.

[0025] 2. Compared with existing methods that use learned index models in SSTables, the method proposed in this invention enables the online training of the RadixSpline model when the SSTable is generated, rather than as an offline task. The offline task is to read all SSTables to train the model when the system is not serving read and write operations.

[0026] 3. Compared with existing methods that use learned index models in SSTable, the method proposed in this invention removes the original read optimization design within SSTable. The retrieval of key-value pairs within SSTable no longer relies on binary search blocks and filter blocks, as in existing studies that apply learned index technology to log structure merging trees. For example, when Bourbon's error range predicted by the learned index model spans multiple blocks, it needs to use binary search blocks to determine the specific block. This invention only relies on the trained RadixSpline model, which also reduces the space occupied by SSTable. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the composition of the model in this invention and how to predict the offset of a key-value pair using the model;

[0029] Figure 2 This is an example illustration of generating a prefix table while generating a model point array in this invention;

[0030] Figure 3 This is a diagram illustrating the training timing of the model during the generation of the log structure merging tree SSTable in this invention;

[0031] Figure 4 This is a schematic diagram illustrating the use of the RadixSpline model for key-value pair offset retrieval within an SSTable according to the present invention.

[0032] Figure 5 This is a comparison chart of read operation latency between the method of this invention and the traditional SSTable read optimization design;

[0033] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0035] Furthermore, the technical solutions of the various embodiments can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0036] This invention provides an LSMT read optimization method based on RadixSpline to accelerate key-value pair retrieval. The invention aims to accelerate the retrieval process of key-value pairs in the merged tree of a key-value separated log structure using a RadixSpline model that can be constructed in one step during data traversal. When the SSTable is stored in a high-speed storage device such as memory, this invention can effectively reduce the retrieval time of key-value pairs in the SSTable, thereby reducing the overall read operation latency of the system. Specifically, it includes the following steps:

[0037] Step S1: Build and train the RadixSpline model.

[0038] The RadixSpline model contains a prefix table and a model array. The elements of the model array are identical to the elements of the dataset, which is used to store key-offset pairs.

[0039] A well-trained RadixSpline model enables the line segment determined by two adjacent elements in the model array to fit the cumulative distribution function of the sorted dataset within the error range, thus limiting the linear search operation in the dataset to the range of the cumulative distribution function.

[0040] The prefix table is used to speed up the lookup of model array elements. When constructing the model array, the prefix table is used to determine the prefix of the newly added model array element. If it is inconsistent with the previous one, the new prefix and the index of the model array element it points to are recorded in the prefix table.

[0041] like Figure 2This is an example illustration of generating a prefix table simultaneously with generating a model point array in this invention. For simplicity, only the keys of the model array are retained in this diagram, while the offset of the key in the dataset is omitted. In this example, the key has 8 bits, and the prefix bit is set to 3. The model array and the prefix table are generated simultaneously when traversing the dataset. When key 17 (00010001) is added to the model array, the prefix table is empty. The prefix 0 (000) and the index (0) of 17 in the model array are taken and added to the prefix table. After that, the prefixes of keys 21 (00010101) and 24 (00011000) added to the model array are both 0, which is equal to the last element in the current prefix table. Therefore, the prefix table remains unchanged until 48 (00110000) is added to the model array. The prefix 1 (001) and the index (3) of 48 in the model array are added to the prefix table. The process of adding 96 (01100000) to the model array and <011,6> to the prefix table follows the same principle. However, when 223 (11011111) is added to the model array, its index is 7, and its prefix is ​​6 (110). Since the last element of the current prefix table is 3, all prefixes in the range [4,6] should be paired with 7 and added to the prefix table.

[0042] Figure 3 This diagram illustrates the training timing of the model during the generation of the log structure merged tree SSTable, as described in this invention. The generation of a new SSTable occurs when the log structure merged tree triggers the compaction mechanism. This includes the creation of the SSTable from the in-memory table ImmutableMemTable and the merging and sorting of data between the two disk layers to eliminate expired data and generate a new SSTable inserted into the higher layer. In log structure merged tree storage engines such as LevelDB, these are referred to as Minor Compaction and Major Compaction, respectively. These are the two times when the index model needs to be trained on the SSTable. This invention is based on the open-source LevelDB. In the LevelDB source code, the final disk write functions for the two types of compactions are leveldb::BuildTable() and leveldb::DBImpl::FinishCompactionOutputFile(), respectively. At these two entry points, the key list of the new SSTable is input into the RadixSpline model module to train the corresponding model.

[0043] Step S2: Integrate the trained RadixSpline model into the LSM-Tree architecture.

[0044] Step S2 specifically includes the following sub-steps:

[0045] Step S201: At each moment when the log structure merge tree SSTable is generated, the list of key-value pairs of the generated log structure merge tree SSTable is sent to the RadixSpline model for storage.

[0046] Furthermore, processing the key-value pair list of the log structure merge tree SSTable also includes: subtracting the maximum and minimum values ​​in the generated log structure merge tree SSTable, recording the number of prefix 0s in the difference commonBits, and calculating the prefix bits of the keys recorded in the prefix table starting from the commonBits bit from the high bit.

[0047] Step S202: Cancel the original read amplification optimization design of the SSTable structure, including canceling the binary search block, filter block and LSM-Tree iterator, and realize the replacement of the original key-value pair retrieval method in SSTable with RadixSpline model.

[0048] Step S3: Obtain the error range of key-value pair offsets by predicting using the RadixSpline model, and then perform a linear search within the offset error range to retrieve key-value pairs in the SSTable.

[0049] For an input key, the offset is predicted using the RadixSpline model, resulting in an error range [lower, upper]. The first lower error bound (lower) and the first upper error bound (upper) are each divided by the number of key-value pairs in the block to obtain the second lower error bound (idxLower) and the second upper error bound (idxUpper). Within the range [idxLower, idxUpper], the traversed blocks are checked, and a binary search is performed on each traversed block until the query result is obtained. Several key-value pairs are stored in one block.

[0050] like Figure 1 This diagram illustrates the error range for predicting key-value pair offsets using the RadixSpline model in this invention. During the dataset traversal, a greedy PLR (Piecewise Linear Regression) method is used to insert qualified key-offset pairs into the model array, establishing a piecewise linear regression model. For example, the line segment determined by elements with indices 0 and 1 in the model array can predict the offset of keys in the range [key0, key1), and the line segment determined by elements with indices 1 and 2 can predict the offset of keys in the range [key1, key2). The prefix table is used to accelerate the traversal process of the model array elements; the construction and acceleration process of the prefix table is described in... Figure 2 The explanation is as follows. For input keys in the range [key1, key2)... s The offset is calculated from the line segment determined by the elements at indices 1 and 2 in the model array.s Since the indexing model is trained and built while maintaining a certain error range, the calculated offset must be added to and subtracted from the given error during training to obtain the offset. max and offset min In [offset] min ,offset max Within the range of ], a linear search is performed in the dataset to determine the key. s Does it exist in the dataset? If it does, obtain its exact offset. s Retrieve the value from the actual storage device.

[0051] like Figure 4 This diagram illustrates the method for retrieving key-value pair offsets within an SSTable using only the RadixSpline model, without relying on binary search blocks or filter blocks. For an input key, its offset is first predicted using the RadixSpline model, obtaining an error range [lower, upper]. The relationship between lower and the maximum value of the SSTable is then determined. If lower is larger than the maximum value of the SSTable, the input key is not in the SSTable. The SSTable using the RadixSpline model disables prefix compression for key storage, so each block stores the same number of entries. Both lower and upper are divided by this number to obtain idxLower and idxUpper. The traversed blocks within the range [idxLower, idxUpper] are checked, and a binary search is performed on each traversed block until a result is found.

[0052] like Figure 5 This is a comparison chart of read operation latency between the present invention and the traditional SSTable read optimization design, including the average latency comparison of the entire read operation and the average latency comparison for retrieving data pairs within the SSTable. The implementation is a modification of the open-source LevelDB. The original SSTable read optimization design uses binary search blocks and Bloom filters. The experimental machine is configured with two 18-core Intel(R) Xeon(R) Gold 6240C CPUs @ 2.60GHz, 32GB*12GB 2666MHz DDR4 DRAM, Ubuntu 20.04.4LTS operating system, kernel version 5.4.0, and an INTELSSDPEDME016T4F disk. The experiment generated 10... 6The number of key-value pairs is loaded in random order, with the key type being uint64_t. When the SSTable portion of the merged tree of the key-value separated log structure is entirely cached in memory, the average latency for retrieving data pairs within the SSTable in this invention is 0.796 microseconds, which is 0.496 times that of the binary search block and Bloom filter method. Since this portion accounts for a high percentage of the time in read operations when the SSTable is in memory, the overall read latency is also significantly reduced. The average read operation latency of this invention is 2.987 microseconds, which is 0.715 times that of the binary search block and Bloom filter method.

[0053] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the LSMT read optimization method based on RadixSpline for accelerated key-value pair retrieval as described above. Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities for the LSMT read optimization method based on RadixSpline for accelerating key-value pair retrieval provided in this embodiment of the invention. Except for... Figure 6 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0054] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the LSMT read optimization method based on RadixSpline for accelerated key-value pair retrieval as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0055] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for optimizing LSMT reads based on RadixSpline to accelerate key-value pair retrieval, characterized in that, The method includes the following steps: A RadixSpline model is constructed and trained. The RadixSpline model includes a prefix table and a model array. The prefix table is used to determine the prefix of newly added model array elements during model array construction. When the prefix of a newly added model array element is inconsistent with the prefix of an existing model array element, the new prefix and the index of the model array element it points to are recorded in the prefix table. The elements in the model array are consistent with the elements in the dataset. The dataset is used to store key-offset pairs. The trained RadixSpline model is integrated into the LSM-Tree architecture, allowing the RadixSpline model to replace the original key-value pair retrieval method within the SSTable; The error range of key-value pair offsets is obtained by predicting using the RadixSpline model, and then key-value pairs are retrieved in the SSTable by linear search within the error range of the offsets. The integration of the trained RadixSpline model into the LSM-Tree architecture includes: At each point when the log structure merge tree SSTable is generated, the list of key-value pairs of the generated log structure merge tree SSTable is sent to the RadixSpline model for storage; Remove the original read amplification optimization structure of the SSTable structure, including removing the binary search block, filter block, and LSM-Tree iterator; Integrating the trained RadixSpline model into the LSM-Tree architecture also includes: The key-value pair list of the log structure merge tree SSTable is processed by subtracting the maximum and minimum values ​​in the generated log structure merge tree SSTable and recording the number of prefix 0s in the difference, commonBits. The prefix bits of the keys recorded in the prefix table are calculated starting from the commonBits bit from the most significant bit.

2. The LSMT read optimization method based on RadixSpline for accelerating key-value pair retrieval according to claim 1, characterized in that, The error range for predicting key-value pair offsets using the RadixSpline model includes: For an input key, the offset is predicted using the RadixSpline model, resulting in an error range [lower, upper]; where lower is the first lower bound of error and upper is the first upper bound of error. Divide the first lower bound (lower) and the first upper bound (upper) by the number of key-value pairs in the block to obtain the second lower bound (idxLower) and the second upper bound (idxUpper). Check the traversed blocks in the range [idxLower, idxUpper] and perform a binary search on each traversed block until the query result is obtained. Several key-value pairs are stored in a block.

3. The LSMT read optimization method based on RadixSpline for accelerating key-value pair retrieval according to claim 2, characterized in that, The error range for predicting key-value pair offsets using the RadixSpline model also includes: Determine the relationship between the first lower bound (lower) and the maximum value of the log structure merge tree (SSTable). If the first lower bound (lower) is greater than the log structure merge tree (SSTable), it indicates that the input key is not in the log structure merge tree (SSTable).

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the LSMT read optimization method based on RadixSpline for accelerated key-value pair retrieval as described in any one of claims 1-3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the LSMT read optimization method based on RadixSpline for accelerated key-value pair retrieval as described in any one of claims 1-3.

Images