Three-layer LSM tree block index-based read optimization method

Abstract

The invention belongs to the field of high-performance storage systems and computer system structures, and particularly discloses a read optimization method based on a three-layer LSM tree block index.The hierarchy of an LSM tree is changed into a three-layer structure, the three-layer structure comprises a segmentation layer, a middle layer and a storage layer, and the segmentation layer is placed in a memory. According to the design, a Bloom filter is removed, the target position of the data block is directly indexed in a hash table mode, the data block can be read more quickly, and the search delay of the Bloom filter is reduced. Secondly, data blocks of all SSTable files of a disk are configured with floating thresholds, and hot data blocks are floated to the upper layer every time when access is performed for a certain number of times, so that a target key can be read on a shorter path when a next reading request arrives, and a faster reading request response is achieved.

Description

Technical Field

[0001] This invention relates to the field of high-performance storage systems and computer architecture, and specifically to a read optimization method based on a three-level LSM tree block index. Background Technology

[0002] Log Structure Merge Tree (LSM) is currently chosen as the underlying storage engine by many databases. For example, Google's LevelDB, Meta's RocksDB, and Apache Cassandra all use LSM as their core architecture because it offers better write performance and relatively acceptable read performance compared to B+ trees. LSM trees use a skip list structure (called a Memtable) as a buffer for data writing in memory. Data is stored in batches in memory at a time, and due to the skip list structure, these batches of data are stored sequentially in memory. Before writing data to memory, LSM trees first write it to a log structure on disk (called a Write-Ahead Log). This write-ahead log serves as a recovery mechanism in case of data loss in memory. After the data is compressed and persisted to disk, the WAL is deleted because the data has been persisted. The underlying disk structure of an LSM tree is divided into... … (referred to as) … , We call it the upper management. There are n levels (referred to as the lower level), and the capacity of each level is M times that of the previous level (default is 10 times). When the total size of the data in a certain level reaches a threshold, compression will be triggered. Layer data compression to The layer (called Major Compaction) is where the high performance of LSM tree writing comes in – it writes to disk in an append-only manner. The LSM tree has multiple layers, and a background thread continuously handles Minor and Major compactions. The higher layers of the LSM tree primarily store newer data flushed from memory, while the lower layers, due to their larger capacity, continuously push data from higher layers into lower layers through Major compactions. The LSM tree's good read performance stems from the fact that, besides… Outside of layer 1, all data in the remaining layers is ordered. Because... The data in the layer is duplicated, therefore, when reading... The layer needs to read all the files (called SSTables) to get the latest data, but ~ Since the data in the layer is ordered, binary search can be used to locate the SSTable where the data is located.

[0003] But the existing LSM tree structure has several problems in reading, first, because The layer data adopts the way of appending writing without sorting, so Multiple SSTables of the layer may exist duplicate keys, which leads to the read amplification problem for The query of the layer data needs to traverse all SSTable files containing the target key, and find the latest key after reading them all into memory, which leads to the read amplification problem. Secondly, due to the disk level structure of the LSM tree, each read needs to find the target key from The layer starts from the layer and finds it down layer by layer, and if the target key is hit in the low layer ), it will cause multiple invalid I / O, which will significantly increase the delay of each query. Finally, due to the use of Bloom filter, it is used to check whether the target key exists in the current file during each query process, and there is a false negative situation, which all correspondingly increases the query delay. Therefore, there is an urgent need for a read optimization design that can improve query delay, read amplification, and less memory occupation. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a read optimization method based on three-layer LSM tree block index, which solves the problems mentioned in the background art.

[0005] To achieve the above purpose, the present application provides the following technical scheme: a read optimization method based on three-layer LSM tree block index, comprising the following steps: Step 1, initializing the LSM tree structure; Step 2, after receiving the read request, first query whether the Memtable skip list in the memory hits the target key, if it hits, directly return; if the Memtable does not hit, go to find the ImmuTable skip list, if it hits, directly return; Step 3, if the key is not hit in Memtable and ImmuTable, go to find in the segmented layer, first locate the target segment where the target key may exist according to the anchor binary search, the search rule is that the largest anchor key is less than or equal to the target key, and then use binary search in the segment to directly locate the target key; Step 4, if the target key is not queried in the segmented layer, go to query the immutable segmented layer, first locate the target segment by inter-segment search, and then use intra-segment binary search to directly locate the target key; Step 5, if the target key is not hit in the segmented layer and the immutable segmented layer, query the hot KV pair in the B+ tree, and increase the hotness value of the corresponding KV pair after successful query; Step 6, if the components of the memory do not hit the target key, go to the disk LSM tree intermediate layer query; Step 7, if not hit in the intermediate layer, go to the storage layer for query.

[0006] Preferably, in step S1, the level of LSM tree structure is changed to a three-layer structure, respectively, a segment layer, an intermediate layer and a storage layer; the segment layer is placed in the memory, and the data is organized in a segmented manner, each segment stores a batch of sequential range keys, and all segments are sorted by key order, and the smallest key of each segment is used as the anchor key of the segment, indicating a starting key of the segment; when querying data, the binary search method between segments is used to quickly locate the target segment, and after locating the target segment, the binary search method is used again to quickly locate the target key within the segment; when the segment layer reaches the threshold, the segment layer becomes an immutable segment layer, at this time the immutable segment layer can be read but cannot be written, and new writes will create new segment layer components to receive new data writes, and the immutable segment layer will wait for the background thread to compress all data to the disk intermediate layer.

[0007] Preferably, the intermediate layer is the first layer of the disk, and is used to store data compressed from the memory, and the storage layer is the last level of the disk, and has a large capacity for storing most of the data; since the intermediate layer uses sequential sorting for storage, the data in the intermediate layer and the compressed data in the immutable segment layer are sorted in the memory before being written back to the intermediate layer, allowing multiple immutable segment layers to exist in the memory; if the immutable segment layer is not fully compressed to the intermediate layer but the segment layer is full, the segment layer is allowed to change to a new immutable segment layer, and the new immutable segment layer component is placed at the tail of a queue, and the background compression thread will compress all immutable segment layer components in the queue to the intermediate layer in a first-in-first-out manner.

[0008] Preferably, a floating threshold is set for each data block in the SSTable of the storage layer and the intermediate layer, and the floating means that if a data block is frequently accessed in the disk, the data block is considered a hot data block, so the data block should be floated to the upper layer to increase the hit rate of target query; The KV pairs floated from the data blocks of the intermediate layer are stored in the memory using a B+ tree, and the floating rule is that the data block of the SSTable file in the disk increases the floating threshold every time it is accessed, and after each access, the floating threshold of the accessed data block is compared with the average floating threshold of the upper layer, and when the average floating threshold of the upper layer is reached, the data block is floated to the upper layer, and the repeated key-value pairs in the target layer are compared and discarded, and only the latest data is left; For the floating rule of the intermediate layer, if the threshold maximum of the accessed data block of the intermediate layer, all the KV pairs of the data block are floated to the B+ tree in the memory for caching; the data to be compressed to the disk in the immutable segment layer is compared with the hot KV pairs in the B+ tree, and the data in the B+ tree is updated and then written to the disk; the cold data in the intermediate layer is compressed preferentially at each time of compression and merging, and the hot read data floated is kept in the upper layer of the disk; a hotness mark is set for each specific KV pair in the B+ tree leaf node, and the data with low hotness is eliminated whenever new data comes.

[0009] Preferably, in step S6, specifically comprising: querying the hash table of the intermediate layer, first through the specified hash function obtaining a hash value of the target key, extracting the high 2 bytes of the hash value to compare with the target bucket data in the to whether there is a match, if not, go to the conflict list of the corresponding bucket from the MRU end, if there is a matching item, directly locate the data block in the SSTable file according to the data block position of the index, and then compare the complete key; If the corresponding key is not cached in the hash table, the SSTable file in the intermediate layer is searched according to the original LSM tree logic, first locate the SSTable file that may contain the target key in this layer in the memory by using binary search, and then return the index position index of this file in the file list in this layer; After obtaining the index of the SSTable file, it is judged that the index must be less than the number of files in this layer, because the index starts from 0, when the index is equal to the number of files in this layer, it means that the key does not hit any file in this layer; According to the obtained index, the metadata of the target file in this layer is obtained, the file number is obtained according to the metadata, the target Data block file block is obtained by querying the index block according to the file number, and finally the key and the index of the file block are cached in the hash table of the intermediate layer; the caching rule is first to sequentially calculate the hash value until an empty bucket without conflict is found to store successfully, if all the buckets are in conflict, the data is stored in the MRU end of the conflict list in the calculated bucket, the data of the stored key is calculated by using the calculation function; After a successful query, the floating threshold for the corresponding accessed data block is increased, and it is determined whether the floating threshold has reached the maximum floating threshold of the intermediate layer. If it has, all data in the data block is cached in the B+ tree in memory. Before inserting into the B+ tree, it is necessary to determine whether the B+ tree has reached the configured capacity. If it has reached the configured capacity, cold data is discarded until all data in the data block can be accommodated by the B+ tree.

[0010] Preferably, the hash table of the query intermediate layer specifically refers to replacing the use of a Bloom filter by indexing the block information in the SSTable of this layer using a hash table. Specifically, this includes: using a cuckoo filter and a linked list to cache the information of this layer; after each read, a hash function is applied to the target key to locate the storage bucket; the hash table of n buckets uses n hash functions to calculate sequentially until an empty bucket is found, at which point the high 2 bytes of the key's hash value are stored; the specific value stored is calculated using a separate hash function; if all buckets calculated by the hash functions are full... The value is then placed into the MRU end of the collision list corresponding to the bucket calculated by the last hash function. For collisions, a doubly linked list method combining LRU and MRU is used to resolve them. One end of the collision list represents the node that has not been used for the longest time, and the other end represents the most frequently used node. Whenever a node in a collision is accessed, the node is removed from its current position and placed into the MRU end of the list. When the MRU end needs to be accessed, the head->pre of the head node of the list is used to access it, which is used to quickly locate the most frequently accessed node in each judgment.

[0011] Preferably, the size of the hash table is set to not exceed 30% of the current layer size of the index. The calculation rule is that the sum of the number of data bytes in the hash table does not exceed 30% of the total number of bytes in the current layer of the index. Each time an insertion occurs, the threshold of the current layer hash table is calculated. If the threshold is exceeded, the longest unused node is removed from the LRU list of the bucket corresponding to the key, and then the new node is inserted into one end of the MRU list.

[0012] The beneficial effects of this invention are: 1) Faster read request response time: The original LSM tree uses a serial approach to read requests. We will perform a query on each layer, here we will The layer, or segmented layer in this design, is stored in memory in segments. The sequential storage of segments, both within and between segments, allows for faster target key matching, replacing the traditional LSM tree approach. Layer repeated file reading case. Secondly, the experiment shows that in the file query process, the query delay of the Bloom filter accounts for most of the total query delay, so the application designs to remove the Bloom filter and use the hash table to directly index the target position of the data block. The method can read the data block faster and reduce the search delay of the Bloom filter. Secondly, we configure a floating threshold for all the data blocks of the SSTable file on the disk. When a certain number of accesses are accessed, the hot data block will be floated to the upper layer. When the next read request comes, the target key will be read on a shorter path.

[0013] 2) More accurate data block positioning: The data positioning of the traditional LSM tree structure usually needs to query the target SSTable file number in the metadata of the memory first, then query the object structure of the corresponding file in the cache according to the file number, and then query the Bloom filter and index block according to the SSTable to locate the data block. However, we can directly locate the target block faster by directly caching the position of the internal data block of the SSTable in the hash table.

[0014] 3) Less memory occupation: In the hash table, the key is stored in the form of high 2 bytes. When querying the hash table, the hash value of the hash function is compared with the high 2 bytes. If they do not match, they are skipped directly. If they match, the data block inside the corresponding SSTable file is located, and then the complete key is compared. Therefore, the memory occupation of the hash table is smaller compared with the complete key. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The figure is a flowchart of the read optimization method based on the three-layer LSM tree block index in the embodiment of the application. Figure 2 The figure is a design architecture diagram of the read optimization method based on the three-layer LSM tree block index in the embodiment of the application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0017] Please refer to Figure 1 and Figure 2 The application provides a technical solution: a read optimization method based on a three-layer LSM tree block index. The method flow is shown in Figure 1 The level of the LSM tree is changed to a three-layer structure, as shown in Figure 2As shown, respectively, are the segment layer, the intermediate layer and the storage layer, the segment layer is placed in memory, and the data is organized in a segmented manner, each segment stores a batch of sequential range keys, all segments are sorted by key order, and the segments are also sorted by key order, and the smallest key in each segment is used as the anchor key of the segment, indicating the starting key of the segment. Since the segment layer is in memory, the sequential sorting method used by ImmuTable when refreshing to the segment layer will be faster than the traditional method on disk, because there is no need to read the disk data out and sort it before writing it back. When querying data, we first use the binary search method between segments to quickly locate the target segment, and then use the binary search method within a segment to quickly locate the target key. When the segment layer reaches the threshold, we change the segment layer to an immutable segment layer, at this time the immutable segment layer can be read but cannot be written, new writes will create new segment layer components to receive new data writes, and the immutable segment layer will wait for the background thread to compress all data to the disk's intermediate layer. The intermediate layer is the first layer of the disk, used to store data compressed from memory, and the storage layer is the last layer of the disk, with a large capacity to store most of the data. Since the intermediate layer uses sequential sorting, the data in the intermediate layer and the compressed data in the immutable segment layer need to be sorted in memory before being written back to the intermediate layer, so there may be a high delay, therefore we allow multiple immutable segment layers in memory, if the immutable segment layer has not been completely compressed to the intermediate layer but the segment layer is full, we still allow the segment layer to change to a new immutable segment layer, and place the new immutable segment layer component at the end of a queue, the background compression thread will compress all immutable segment layer components in the queue in the order of first-in, first-out, this can effectively reduce the write pause problem caused by slow compression.

[0018] For the long path read problem of LSM tree. Since we reduce the level, the files of the intermediate layer and the storage layer will be smaller than the original LSM tree 、 The layer is much larger, in the original LSM tree structure query logic, there will be each layer file metadata in memory, in memory first using binary search to locate the file number, and then use the file number to find the disk, so we reduce the level of appropriate increase the number of file binary search will be lower than multiple invalid I / O search delay. And we will set a floating threshold for each data block in the storage layer and the intermediate layer SSTable, the meaning of floating is that if a data block is frequently accessed on the disk, we think that the data block is a hot data block, so the data block should be floated to the upper layer, increase the hit rate of target search. We use B+ tree to store the KV pair in memory by the data block of the intermediate layer, the floating rule is that the data block of the SSTable file on the disk increases the floating threshold every time it is accessed, after each access we will compare the floating threshold of the accessed data block with the average floating threshold of the upper layer, when it reaches the average floating threshold of the upper layer, we will float the data block to the upper layer. For the repeated key-value pairs of the target layer, we will compare and discard the old redundant data, and only keep the latest data. For the floating rule of the intermediate layer, if the threshold of the accessed intermediate layer data block is the largest, we will float all the KV pairs of the data block to the B+ tree in memory. For the update problem of B+ tree, we will compare the data to be compressed to the disk in the immutable segment layer with the hot KV pairs in the B+ tree, update the data in the B+ tree and then write it to the disk. For the problem that continuous floating may cause the capacity of the intermediate layer to be too large, we design to compress the cold data in the intermediate layer during each compression and merging, and keep the hot read data in the upper layer of the disk as much as possible. For the problem of B+ tree elimination, we set a hotness label for each specific KV pair in the B+ tree leaf node, and whenever new data comes, we will eliminate the data with relatively low hotness.

[0019] For the read delay problem caused by the original structure Bloom filter, we remove the use of Bloom filter in our new design, instead of using a hash table to index the block information in the layer SSTable in the disk structure respectively, specifically we use a cuckoo filter and a linked list to cache the layer information, every time after a read, we hash the target key to locate the stored bucket, a total of n buckets of hash table we use n hash functions to calculate sequentially, until we find an empty bucket after the high 2 bytes of the hash value of the key are stored, the specific value we calculate through a separate hash function, if all the hash functions calculated buckets are full, we will put the value into the MRU end of the conflict linked list in the bucket corresponding to the value calculated by the last hash function. For the conflict case, we use a combination of LRU and MRU double linked list method to solve, the head of the conflict linked list represents the longest time unused node, and the other end represents the most frequently used node, every time the node in the conflict is accessed, we delete the node from the current position and put it into the MRU end of the linked list, when accessing the MRU end, we use the head->pre of the head node to access, which is used to quickly locate the most frequently accessed node when judging. For the size of the hash table, we set it not more than 30% of the current layer size of the index, the calculation rule is that the total data byte number in the hash table is not more than 30% of the total byte number of the index in the layer, every time we calculate the threshold of the current layer hash table, if it exceeds the threshold, we eliminate the longest unused node in the LRU list of the key corresponding bucket, and then insert the new node into the MRU end. For the block invalidation problem caused by compression, we will extract all the block information in the SSTable involved in the compression process, and delete the corresponding data in the hash table after extraction.

[0020] The read optimization method based on three-layer LSM tree block index of the application comprises the following implementation steps: (1) first query whether the target key is hit in the Memtable skip list in the memory, if hit, return directly, as shown in a of Figure 2 .

[0021] (2) if the Memtable is not hit, go to find the ImmuTable skip list, if hit, return directly, as shown in b of Figure 2 .

[0022] (3) if the key is not hit in the Memtable and ImmuTable, go to find in the segmented layer, first locate the target segment where the target key may exist according to the anchor bisection, as shown in Figure 2As shown in c, the search rule is that the largest anchor key is less than or equal to the target key, and then a binary search is used within the segment to directly locate the target key, such as... Figure 2 As shown in d.

[0023] (4) If the target key is not found in the segmentation layer, the query will proceed to the immutable segmentation layer. The logic is the same as that of the segmentation layer: first, an inter-segment search is performed to locate the target segment, and then an intra-segment binary search is used to directly locate the target key, such as... Figure 2 As shown in e.

[0024] (5) If the target key is not found in either the segmented layer or the immutable segmented layer, then query the hot key-value pairs in the B+ tree. If the query is successful, add a heat value to the corresponding key-value pair, such as... Figure 2 As shown in f.

[0025] (6) If the target key is not found in any of the memory components, the search proceeds to the LSM tree on disk. The intermediate hash table is queried, first by using the specified hash function. Obtain a hash value for the target key, extract the high 2 bytes of the hash value and compare them with those in the bucket. arrive Check if there is a match in the target bucket data; if not, then proceed... The corresponding bucket's conflict list is searched from the MRU end. If a match is found, the data block in the SSTable file is located directly based on the index's data block position, and then the complete key is compared. Figure 2 As shown in g.

[0026] (7) If the corresponding key is not cached in the hash table, the SSTable file in the intermediate layer is searched according to the logic of the original LSM tree. First, the binary search method is used in memory to locate the SSTable file that may contain the target key in this layer, and then the index position of this file in the file list of this layer is returned.

[0027] (8) After obtaining the SSTable file index, it is determined that the index must be less than the number of files in this layer. Since the index starts from 0, when the index is equal to the number of files in this layer, it means that the key does not match any file in this layer.

[0028] (9) Obtain the metadata of the target file at this layer based on the obtained index, obtain the file number based on the metadata, then query the index block based on the file number to obtain the target data block (file block), and finally cache the key and the index of the file block in the hash table of the intermediate layer. The caching rule is to first... arrive The hash value is sequentially calculated until an empty bucket without collision is found for smooth storage, and if all buckets are collided, the data is stored in The MRU end of the collision chain table in the calculated bucket, the data of the stored key is calculated using a dedicated calculation function.

[0029] (10) After the query is successful, the floating threshold of the corresponding access data block is increased, and it is judged whether the floating threshold reaches the maximum floating threshold of the middle layer, if it is reached, all data in the data block is cached to the B+ tree in the memory, before being inserted into the B+ tree, it is needed to judge whether the B+ tree reaches the configured capacity, if the configured capacity is reached, the colder data is eliminated until all data in the data block can be accommodated by the B+ tree.

[0030] (11) If it is not hit in the middle layer, the file in the storage layer is queried, the query logic is the same as steps (6)-(10), the only difference is that the data block of the storage layer is floated to the middle layer instead of the memory, as shown in h in Figure 2

[0031] So far, the introduction of the read optimization method based on the three-layer LSM tree block index designed by the application is completed.

[0032] Each block in the flowchart or block diagram can represent a module, a program segment or a part of code, and the module, the program segment or the part of code include one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementation modes, the functions marked in the blocks can also occur in an order different from that marked in the figure. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a special hardware-based system that performs the specified functions or actions, or can be implemented by a combination of special hardware and computer instructions.

[0033] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0034] ​​The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the application and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0035] It should be understood that, the term "and / or" used herein is merely an association relationship of the associated objects, and indicates that there can be three relationships, for example, A and / or B, which can represent the three cases of A existing alone, A and B existing simultaneously, and B existing alone. In addition, the character " / " herein generally represents that the front and rear associated objects are in an "or" relationship.

[0036] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if determined" or "if [a stated condition or event] is detected" can be interpreted to mean "when determined" or "in response to determining" or "when [a stated condition or event] is detected" or "in response to detecting [a stated condition or event]."

[0037] Although the present application has been described in detail with reference to the foregoing embodiments, the technical solutions recorded in the foregoing embodiments can be modified by those skilled in the art, or some technical features can be replaced by equivalent features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A read optimization method based on three-tier LSM tree block index, characterized in that, Comprise the following steps: Step 1, initialize LSM tree structure; Step 2, after receiving a read request, first query whether the Memtable skip list in memory hits the target key, if it hits, return directly; if the Memtable does not hit, go to find the ImmuTable skip list, if it hits, return directly; Step 3, if the key is not hit in Memtable and ImmuTable, go to find in the segmented layer, first locate the target segment where the target key may exist according to the anchor binary search, the search rule is that the maximum anchor key is less than or equal to the target key, and then use binary search to directly locate the target key in the segment; Step 4, if the target key is not queried in the segmented layer, go to query the immutable segmented layer, first locate the target segment by inter-segment search, and then use inter-segment binary search to directly locate the target key; Step 5, if the target key is not hit in the segmented layer and the immutable segmented layer, query the hot KV pair in the B+ tree, and increase the hotness value of the corresponding KV pair after successful query; Step 6, if the target key is not hit in the memory components, go to query in the middle layer of the disk LSM tree; Step 7, if it is not hit in the middle layer, go to query in the storage layer. 2.The read optimization method based on three-layer LSM tree block index according to claim 1, characterized in that: In step S1, the level of LSM tree structure is changed to three-layer structure, which is segmented layer, middle layer and storage layer; the segmented layer is placed in memory, which organizes data in a segmented way, each segment stores a batch of sequential range keys, all segments are sorted by key, and the minimum key of each segment is used as the anchor key of the segment, indicating the starting key of the segment; when querying data, first use inter-segment binary search to quickly locate the target segment, and then use inter-segment binary search to quickly locate the target key in a segment; when the segmented layer reaches the threshold, the segmented layer becomes an immutable segmented layer, at this time the immutable segmented layer can be read but cannot be written, new write will create a new segmented layer component to receive new data write, and the immutable segmented layer will wait for the background thread to compress all data to the middle layer of the disk. 3.The read optimization method based on three-layer LSM tree block index according to claim 2, characterized in that: The middle layer is the first layer of the disk, used to store data compressed from memory, and the storage layer is the last layer of the disk, with large capacity for storing most of the data; since the middle layer uses sequential sorting to store data, the data in the middle layer and the compressed data in the immutable segmented layer are sorted in memory before being written back to the middle layer, allowing multiple immutable segmented layers in memory; if the segmented layer is full but the immutable segmented layer has not been completely compressed to the middle layer, the segmented layer is allowed to change to a new immutable segmented layer, and the new immutable segmented layer component is placed at the tail of a queue, and the background compression thread will compress all immutable segmented layer components in the queue according to the first-in-first-out rule. 4.The read optimization method based on three-layer LSM tree block index according to claim 2, characterized in that: A floating threshold is set for each data block in the storage layer and the intermediate layer. Floating means that if a data block is frequently accessed on the disk, it is considered a hot data block, so the data block should be floated to the upper layer to increase the hit rate of target lookup. The KV pairs floated by the data block of the intermediate layer are stored in the B+ tree in the memory. The floating rule is that the data block of the SSTable file on the disk increases the floating threshold every time it is accessed. After each access, the floating threshold of the accessed data block is compared with the average floating threshold of the upper layer. When the average floating threshold of the upper layer is reached, the data block is floated to the upper layer. For the repeated key-value pairs in the target layer, the comparison is made and the old redundant data is discarded, and only the latest data is left. For the floating rule of the intermediate layer, if the threshold of the accessed data block of the intermediate layer is the largest, all KV pairs of the data block are floated to the B+ tree in the memory for caching. The data in the B+ tree is updated after comparing the data to be compressed to the disk with the hot KV pairs in the B+ tree, and then written to the disk. The cold data in the intermediate layer is compressed first during each compression and merging, and the hot read data is kept in the upper layer of the disk. A hotness mark is set for each specific KV pair in the B+ tree leaf node. When new data comes, the data with low hotness is discarded.

5. The read optimization method based on three-tier LSM tree block indexing according to claim 1, wherein: In step S6, it specifically includes: To query the hash table of the intermediate layer, first use the specified hash function. Obtain a hash value for the target key, extract the high 2 bytes of the hash value and compare them with those in the bucket. arrive Check if there is a match in the target bucket data; if not, then proceed... The corresponding bucket's conflict chain is searched from one end of the MRU list. If a match is found, the data block in the SSTable file is located directly based on the index data block position, and then the complete key is compared. If the corresponding key is not cached in the hash table, the SSTable file in the intermediate layer is searched according to the logic of the original LSM tree. First, the binary search method is used to locate the SSTable file that may contain the target key in the memory, and then the index position index of the file in the file list of this layer is returned. After obtaining the SSTable file index index, it is judged that the index must be less than the number of files in this layer, because the index starts from 0. When the index is equal to the number of files in this layer, it means that the key does not hit any file in this layer. According to the index obtained, the metadata of the target file of this layer is obtained, the file number is obtained according to the metadata, the target Data block file block is obtained by querying the index block according to the file number, and finally the key and the index of the file block are cached in the hash table of the intermediate layer; the caching rule is first to Calculate the hash value in sequence until the empty bucket without conflict is found for successful storage, if all buckets are conflicted, the data is placed in MRU end of the conflict chain table in the calculated bucket, and the data of the key is calculated using the calculation function; After the query is successful, the floating threshold of the corresponding accessed data block is increased, and it is judged whether the floating threshold reaches the maximum floating threshold of the intermediate layer. If it is reached, all data in the data block is cached to the B+ tree in the memory. Before being inserted into the B+ tree, it is necessary to judge whether the B+ tree reaches the configured capacity. If the configured capacity is reached, the cold data is discarded until all data in the data block can be accommodated by the B+ tree.

6. The read optimization method based on three-tier LSM tree block indexing according to claim 5, characterized in that: The hash table of the query intermediate layer is specifically using a hash table to index the block information in the layer SSTable to replace the use of a Bloom filter, and specifically includes: using a cuckoo filter and a linked list to cache the layer information, and when reading once, a hash function is used on the target key to locate the stored bucket, a total of n buckets of the hash table use n hash functions to sequentially calculate, and after an empty bucket is found, the high 2 bytes of the hash value of the key are stored, the specific value stored is calculated by a separate hash function, if all the buckets calculated by the hash functions are full, the value is placed in the MRU end of the conflict linked list in the bucket corresponding to the value calculated by the last hash function; for the conflict, a combination of LRU and MRU is used to solve the problem, the head of the conflict linked list represents the node that has not been used for the longest time, and the other end represents the node that is most frequently used; when the node in the conflict is accessed, the node is deleted from the current position and placed in the MRU end of the linked list, and when the MRU end is accessed, the head->pre of the head node of the linked list is used to access, which is used to quickly locate the most frequently accessed node when judging each time.

7. The read optimization method based on three-layer LSM tree block index according to claim 5 or 6, characterized in that: The size of the hash table is set to be no more than 30% of the size of the current layer of the index, and the calculation rule is that the total number of data bytes in the hash table is no more than 30% of the total byte size of the layer of the index, the threshold value of the current layer hash table is calculated each time the insertion is performed, if the threshold value is exceeded, the longest unused node in the LRU list of the key corresponding bucket is eliminated, and then the new node is inserted into the MRU end.

Images