A hot read optimization design method based on an LSM tree key-value storage system

Abstract

This invention belongs to the field of high-performance storage systems and computer architecture, specifically disclosing a hot-read optimization design method based on an LSM tree key-value storage system. First, an ordered key-value mapping component, i.e., a hot locator, is designed to maintain the mapping relationship between the largest key in a hot SSTable and the corresponding SSTable file pointer. With this structure, during the query process, candidate SSTable files at any level can be quickly located based on the target key, thus avoiding the additional overhead of traditional layer-by-layer traversal and effectively reducing read amplification problems. Second, a two-layer hash cache is designed to cache fine-grained hot key-value pairs. This structure reduces the probability of collisions through a hierarchical hashing mechanism, improving cache utilization while ensuring query efficiency. Furthermore, since its cached objects are stable key-value pairs rather than underlying data blocks, it can effectively avoid cache invalidation problems caused by data block reorganization during compression, thereby improving cache hit rate and reducing additional memory overhead.

Description

Technical Field

[0001] This invention relates to the field of high-performance storage read optimization technology, and specifically to a hot-read optimization design method for an LSM tree key-value storage system. Background Technology

[0002] Log-Structured Merge Tree (LSM) has become a core index structure in modern key-value storage systems. For example, Google's LevelDB, Meta's RocksDB, and systems like Apache Cassandra, HBase, Dynamo, and cLSM all use LSM trees as their core storage engine. Compared to traditional index structures such as B+ trees and red-black trees, LSM trees can fully utilize the advantages of sequential I / O to improve write performance, while minimizing the adverse impact on read performance by maintaining the ordered organization of data.

[0003] LSM trees employ different data organization methods in memory and storage layers to achieve a balance between read and write performance. In the memory layer, the LSM tree uses a skip list structure to organize data, typically called a MemTable. When the MemTable's capacity reaches a preset threshold, its state is marked as read-only, transforming it into a non-writable MemTable (immuTableMemTable). At this point, subsequent key-value pairs are imported into a newly created MemTable. When the system detects the existence of an ImmuTable MemTable, it triggers a background thread to flush the batch of data to persistent storage media (usually SSD), thus completing the data persistence process. In the storage layer, the LSM tree uses a file structure called SSTable to organize data. Each SSTable consists of data blocks composed of key-value pairs flushed from the ImmuTable MemTable, a Bloom filter, an index block, and other metadata. The Bloom filter is used to quickly determine whether a target key might exist in the current SSTable, while the index block is used to efficiently locate the data block containing the target key.

[0004] In SSDs, LSM trees use a multi-level storage structure to organize SSTables, typically divided into multiple levels (denoted as LSM). to ).in, The layer is located at a higher level, while The first layer is located at a lower level. From top to bottom, the capacity of each layer increases exponentially; for example, the first... The capacity of a layer is usually the first Approximately 10 times the size of the layer. Data is first written... Layer. When Once a layer reaches its capacity threshold, its data will be compressed and pushed down to the next layer (i.e., ...). This process, often referred to as major compaction, involves reading, merging, sorting, and rewriting multiple SSTables. Due to the merge sort, except... Outside of the first layer, key-value pairs in all other layers maintain a globally ordered state. This characteristic allows the system to quickly locate the target key using efficient search algorithms such as binary search, thereby effectively improving query performance and reducing search overhead.

[0005] Compression is a key technique in LSM trees, used to compress trees that have reached a capacity threshold. Layer data is reorganized and written to the next layer. In this process, the system typically starts from the first... Select an SSTable from the layer, and from the first The layer selects one or more SSTables whose key ranges overlap with its own, performs deduplication, merge, and sort operations on the data in these files, and finally generates a new SSTable file and writes it. Layers. Because the capacity of each layer in an LSM tree within an SSD increases from top to bottom (i.e., the lower the level, the larger the capacity), this mechanism allows most data to gradually sink to the lower, larger levels, thereby improving storage space utilization efficiency. Simultaneously, through continuous merge and sort operations, the LSM tree ensures that, except for... The data in each layer outside the main layer remains in an ordered state, thereby supporting efficient query operations and improving overall retrieval performance.

[0006] However, existing LSM-based key-value storage systems still have several problems that need to be solved. First, because SSDs use a multi-level data organization structure, query operations usually need to start from... Searching down layer by layer until... Layers. For those located at lower levels (such as...) For hot data at each layer, this layer-by-layer query mechanism inevitably increases the access path length, thereby introducing higher query latency and further exacerbating the read amplification problem. Secondly, because SSTables within a layer are typically generated using an append-only approach, and their internal data is not guaranteed to be globally ordered. Therefore, during a query, it's necessary to traverse all SSTable files that might contain the target key and check them one by one in ascending order to locate the latest version of the key-value pair. Research indicates that in extreme cases... A layer might need to query up to 12 SSTable files, resulting in read amplification of hundreds of times. Finally, although LSM-based key-value stores typically introduce block caching to improve read performance, the compression process reorganizes and rewrites data blocks in the SSTable, invalidating references to older data blocks in the original cache and thus reducing cache hit rate. This not only weakens the optimization effect of caching on read performance but also incurs additional memory overhead. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a hot read optimization design method based on an LSM tree key-value storage system. This method introduces two key components: a hot locator and a two-layer hash cache, aiming to improve the access efficiency of hot data and reduce read amplification, thus solving the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a hot-read optimization design method based on an LSM tree key-value storage system, comprising the following steps: S1. System initialization: Create a hot locator and a two-level hash cache in memory; S2. Based on the read request, check if the target key is matched in the Memtable in memory. If matched, return directly; if not matched, search the ImmuTable. If matched, return directly. S3. If the target key is not found in either Memtable or ImmuTable, then query the hot locator. Query the hot SSTable index at each level of the hot locator, starting from the L0 ordered mapping and continuing to the last ordered mapping. The query rule is to return the information of the first entry whose maximum key is greater than or equal to the target key, and obtain the corresponding SSTable information based on the entry information. S4. If the target SSTable information is located in the hot locator, then query the double-layer hash cache according to the SSTable file number, i.e., SSTableID. The query rules are: first, locate the target bucket by hashing the file number; second, query the inner hash table according to the target bucket; and finally, return successfully after locating the bucket of the target key in the inner hash table. S5. If the query fails in the hot locator or the two-level hash cache, then query layer by layer according to the query logic of the original LSM tree. S6. After a successful query, increase the popularity of the corresponding SSTable and calculate whether the popularity value has reached the predefined threshold. If the SSTable has not reached the predefined threshold, return the query result directly. S7. If the SSTable reaches the predefined threshold, add the SSTable index to the hot locator, insert the maximum key of the SSTable, cache the pointer of the SSTable for subsequent access to directly locate the target key, and cache the key-value pair in the double-layer hash cache. S8. Update the LRU list and record the last access time. Subsequent accesses will discard the key node that has not been accessed for the longest time and has the lowest access time based on its popularity and the last access time.

[0009] Preferably, the hot locator is an ordered key-value mapping component used to maintain the mapping relationship between the largest key in the hot SSTable and the corresponding SSTable file pointer. The key-value stores the largest key of each SSTable and its corresponding file pointer, and the keys are ordered. With the help of the ordered nature of the keys, the first entry that meets the query conditions can be quickly located by searching, thereby determining the target SSTable.

[0010] Preferably, the hot locator has a hierarchical index system corresponding to the LSM tree hierarchy, which can search layer by layer in hierarchical order to ensure that the SSTable containing the latest data is hit first, and an array structure is used to uniformly manage the capacity of the ordered mapping; and an index synchronization update mechanism is introduced during the compression process: when the SSTable participating in the compression already has a corresponding index entry in the hot locator, after the compression is completed, the newly generated SSTable is automatically added to the hot locator, and the mapping relationship between its maximum key and the file pointer is established. At the same time, the original invalid index entries are removed in a timely manner.

[0011] Preferably, the dual-layer hash cache consists of nested inner and outer hash tables and an LRU table. The outer hash table maps SSTable IDs to the corresponding inner hash table, while the inner hash table further maps keys to nodes in the LRU linked list. Through this hierarchical mapping mechanism, data in different SSTables is divided into independent inner hash spaces, thereby effectively reducing the probability of hash collisions and shortening the query path. At the same time, the inner hash table, combined with the LRU eviction policy, retains only frequently accessed hot key-value pairs. When the cache capacity reaches its limit, the least recently used entry is evicted first.

[0012] Preferably, the dual-level hash cache adopts a hotspot determination strategy at the SSTable level. Specifically, only when an SSTable is identified as a hot file and included in the hot locator index structure is the overall key-value pair within it considered to meet the hot data determination criteria. Furthermore, a synchronous update mechanism is introduced during the compression process: when the ImmutableMemtable is flushed to the SSD, the system uses an iterator to traverse the key-value pairs in the Memtable. During the traversal, the corresponding entries in the dual-level hash cache are updated simultaneously to ensure that the key-value pairs in the cache remain valid at all times.

[0013] Preferably, in step S5, the query logic of the original LSM tree is as follows: First, query the L0 level of the LSM tree. If there is no SSTable containing the target key in the L0 level, then query the SSTables in the other levels until an SSTable containing the target key is found, then the query is successful.

[0014] Preferably, the predefined threshold is as follows: when the number of accesses to a certain SSTable reaches 0.5% of the total number of accesses, it is identified as a hot SSTable, and the mapping relationship between the maximum key of the SSTable and its file pointer is inserted in the hot locator.

[0015] The beneficial effects of this invention are: 1) Faster response time for hot read requests: By indexing hot SSTables, the system can quickly locate target SSTable files at any level in the LSM tree. For frequently accessed SSTables that have sunk to lower levels, queries can directly hit the index without traversing each level, thus significantly reducing access latency. Simultaneously, the nested hash table cache (two-level hash cache) efficiently manages fine-grained key-value pairs, effectively reducing hash collisions. When facing repeated access to hot data, this cache avoids repeated access to Bloom filters and index blocks, thereby reducing I / O overhead and significantly improving read speed and overall response performance.

[0016] 2) More Precise Hot Data Identification: This invention experimentally sets the threshold for identifying hot SSTables at 0.5% of the total load requests. This threshold effectively avoids two types of problems: firstly, it prevents the expansion of hot locator entries caused by premature index addition, thus reducing query efficiency; secondly, it prevents the index from being added too late, causing hot SSTables to be unable to be quickly located within a certain period. Through this precise threshold control, the system can accurately identify truly hot SSTables while maintaining no interference with cold SSTables, thereby ensuring the efficiency of the index structure and the stability of query performance.

[0017] 3) More stable hot-read performance: In this invention, by synchronously updating the hot locator index and fine-grained key-value pairs in the dual-layer hash cache, the system can effectively avoid index invalidation caused by compression and the impact of ImmutableMemtable refresh on the validity of cached key-value pairs. This mechanism ensures that hot data can always be quickly located and accessed throughout the entire system operation, thereby maintaining the stability of hot-read performance and ensuring that the system can still provide efficient and reliable responses under high load and dynamic access modes. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the hot-read optimization design method for an LSM tree-based key-value storage system. Figure 2 A diagram of the read architecture optimized for hot reads in an LSM tree-based key-value storage system. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1 This invention provides a technical solution: a hot-read optimization design method based on an LSM tree key-value storage system. By introducing two key components into memory—a hot locator and a two-level hash cache—it aims to improve the access efficiency of hot data and reduce read amplification. First, an ordered key-value mapping component (called the "hot locator") is designed to maintain the mapping relationship between the largest key in the hot SSTable and the corresponding SSTable file pointer. With this structure, during the query process, candidate SSTable files at any level can be quickly located based on the target key, thereby avoiding the additional overhead of traditional level-by-level traversal and effectively reducing the read amplification problem. For duplicate files generated due to multiple versions within a layer, the hot locator can directly locate the SSTable containing the latest data, avoiding redundant file scanning. Secondly, a nested hash structure (called a "two-layer hash cache") is designed to cache fine-grained hot key-value pairs. This structure reduces the probability of collisions through a hierarchical hashing mechanism, improving cache utilization while ensuring query efficiency. Furthermore, because its cached objects are stable key-value pairs rather than underlying data blocks, it effectively avoids cache invalidation issues caused by data block reorganization during compaction, thereby improving cache hit rate and reducing additional memory overhead.

[0021] I. Thermal Positioner First, in an LSM tree, each SSTable typically contains a large number of key-value pairs. Therefore, given a target key, efficiently locating the corresponding SSTable is crucial. To address this, this invention employs an ordered key-value mapping structure to maintain index information. The key-value pairs store the maximum key of each SSTable and its corresponding file pointer, and the keys are ordered. Leveraging the ordered nature of the keys, the first entry satisfying the query conditions can be quickly located through efficient searching (such as binary search), thus determining the target SSTable. Second, since key-value pairs in an LSM tree become progressively older from top to bottom, a single global index structure would struggle to determine whether the located SSTable contains the latest version of data. To solve this problem, the hot locator is designed as a hierarchical index system corresponding to the LSM tree hierarchy, searching layer by layer in hierarchical order to ensure priority is given to SSTables containing the latest data. Finally, to effectively control the index size and reduce memory overhead, the system uses an array structure to uniformly manage the capacity of the ordered mapping. When the index size reaches a preset threshold, entries will be comprehensively evaluated based on access time and access frequency, and those index entries that have not been accessed for a long time and have low popularity will be eliminated first, in order to maintain the efficiency and stability of the index structure.

[0022] It's important to note that the hot locator primarily indexes hot SSTable files; therefore, the system doesn't immediately add the corresponding SSTable pointer to the structure after each access. Instead, a reasonable threshold determination mechanism is crucial for identifying hot data and improving query efficiency. If the threshold is set too low, too many SSTables will be misidentified as hot files, causing a rapid increase in entries in the hot locator, increasing index maintenance and query overhead, and reducing overall query efficiency. Conversely, if the threshold is set too high, it will delay the identification of hot SSTables, preventing the hot locator from effectively adding new entries for an extended period. This will force subsequent queries to rely on layer-by-layer traversal, hindering the advantages of index acceleration. Based on this trade-off, this invention has determined an empirical threshold through experimental analysis: when the number of accesses to an SSTable reaches 0.5% of the total number of accesses, it is identified as a hot SSTable, and a mapping relationship between the SSTable's maximum key and its file pointer is inserted into the hot locator. This strategy effectively improves the location efficiency of hot data while controlling the index size, thereby optimizing overall query performance.

[0023] The compression process merges old SSTable files and generates new SSTables, which invalidates existing index entries in the hot locator. This not only affects query efficiency but also causes invalid indexes to occupy additional memory space. To solve this problem, this invention introduces an index synchronization update mechanism during compression: when an SSTable participating in compression already has a corresponding index entry in the hot locator, the newly generated SSTable is automatically added to the hot locator after compression, and a mapping relationship between its maximum key and file pointer is established. Simultaneously, invalid index entries are promptly removed. This mechanism ensures that the index structure remains consistent with the underlying data, thereby maintaining index validity. Through this design, the system can continuously maintain the ability to quickly locate hot data throughout the entire workload execution, avoiding performance degradation caused by compression operations and further improving overall query efficiency and space utilization.

[0024] II. Two-level hash cache Traditional hash tables typically use linked structures (such as linked lists) to resolve hash collisions. However, when frequently accessed entries are located at the end of the collision chain, the access path length increases significantly, leading to additional access latency and reduced overall read performance. To alleviate these problems, this invention designs a two-layer hash structure for more efficient entry lookup. This two-layer hash cache consists of nested hash tables and an LRU (Least Recently Used) list: the outer hash table maps SSTable IDs to corresponding inner hash tables, and each inner hash table further maps keys to nodes in the LRU linked list. Through this hierarchical mapping mechanism, data from different SSTables is divided into independent inner hash spaces, effectively reducing the probability of hash collisions and shortening the query path. Furthermore, to control space overhead while improving performance, the inner hash tables incorporate an LRU eviction policy, retaining only frequently accessed hot key-value pairs. When the cache capacity reaches its limit, the system prioritizes eviction of the least recently used entries, thereby maintaining a high cache hit rate and query efficiency with limited memory resources.

[0025] In LSM-based key-value stores, traditional methods typically employ global indexes to cache fine-grained key-value pairs to compensate for the limitations of block caches in terms of lookup granularity. However, such global indexing mechanisms lack the ability to accurately characterize hot data. Existing research shows that hot data exhibits significant time-varying characteristics in real-world workloads. As access patterns change, previously hot key-value pairs may quickly become cold data, and these outdated entries will remain in the cache for extended periods, consuming significant amounts of invalid space and reducing cache utilization. To address these issues, this invention employs a hot data determination strategy at the SSTable level. Specifically, only when an SSTable is identified as a hot file and included in the hot locator index structure is its internal key-value pairs considered to meet the criteria for hot data. In this way, the system can characterize access locality at a coarser granularity, effectively avoiding over-responding to transient hotspots, thereby reducing invalid cache usage and improving overall cache management efficiency.

[0026] The compression process deletes old SSTable files and generates new ones, leading to cache and index invalidation. Traditional LSM-based key-value stores suffer from block cache invalidation due to this mechanism, resulting in reduced read performance and additional invalid space usage. To address this issue, this invention introduces a synchronous update mechanism in the dual-level hash cache. When the Immutable Memtable is flushed to the SSD, the system uses an iterator to traverse the key-value pairs in the Memtable. During this traversal, the corresponding entries in the dual-level hash cache are simultaneously updated to ensure that the key-value pairs in the cache remain valid. By synchronously updating the cache in the background, foreground query performance is almost unaffected, while ensuring that cache validity is continuously maintained throughout system operation, thereby effectively improving the read efficiency of frequently accessed data and reducing invalid space usage.

[0027] Example 2 A hot-read optimization design method based on an LSM tree key-value storage system, the overall steps of which are as follows: Figure 1 As shown, the operation includes the following implementation process: (1) Check if the target key is matched in the Memtable in memory. If matched, return directly, e.g. Figure 2 As shown in step 1.

[0028] (2) If the Memtable is not found, then search for the ImmuTable (Memtable). If a match is found, return directly. Figure 2 As shown in step 2.

[0029] (3) If the target key is not found in either the Memtable or the ImmuTable, the query will proceed to the hot locator. Since the hot locator uses multiple independent ordered mapping components to cache the hot SSTable indexes at each level of the LSM tree, the query needs to start from the L0 ordered mapping and continue to the last ordered mapping. The query rule is to return the information of the first entry whose maximum key is greater than or equal to the target key, and then obtain the corresponding SSTable information based on the entry information, such as... Figure 2 As shown in step 3.

[0030] (4) If the target SSTable information is located during hot location, the double-hash cache is queried based on the SSTable's file number (i.e., SSTableID). Since the file number is unique, all hot SSTables have a corresponding inner hash table for caching the hot key-value pairs in that hot SSTable. The specific query rules are: first, locate the target bucket using the file number; second, query the inner hash table based on the target bucket; and finally, return successfully after locating the bucket of the target key in the inner hash table. Figure 2 As shown in step 4.

[0031] (5) If the query fails in the hot locator or the two-level hash cache, then query layer by layer according to the query logic of the original LSM tree.

[0032] (6) Assuming the target key is 605, first query the L0 level of the LSM tree. If there is no SSTable containing the target key in the L0 level, then query the SSTables in other levels, such as... Figure 2 As shown in step a.

[0033] (7) If the SSTable containing the target key is located after querying the L4 layer, the query is successful. A specific example is shown below. Figure 2 As shown in step b.

[0034] (8) After a successful query, increase the heat value of the corresponding SSTable and calculate whether the heat value reaches the predefined threshold (i.e., 0.5% of the total load request).

[0035] (9) If SSTable does not reach the predefined threshold, the query result will be returned directly.

[0036] (10) If the SSTable reaches a predefined threshold, the maximum key of the corresponding SSTable (i.e., 989, the example in this embodiment) is added to the appropriate position in the L4 ordered mapping, and the pointer of the SSTable is cached for subsequent access to directly locate the target key, such as Figure 2 As shown in step c.

[0037] (11) Add the SSTable with SSTable ID 35 to the two-level hash cache (i.e., the outer hash table), such as Figure 2 As shown in step d.

[0038] (12) Create a separate inner hash table for this SSTable, and cache key 605 in the inner hash table, such as Figure 2 As shown in step e.

[0039] (13) Update the LRU list and record the last access time. Subsequent accesses will eliminate the key node that has not been accessed for the longest time and has the lowest access time based on the popularity and the last access time.

[0040] This concludes the introduction to the hot-read optimization design of the LSM tree key-value storage system described in this invention.

[0041] It should be noted that, in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0042] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0043] It should be understood that the term "and / or" used in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0044] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0045] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hot-read optimization design method for an LSM tree key-value storage system, characterized in that, Includes the following steps: S1. System initialization: Create a hot locator and a two-level hash cache in memory; S2. Based on the read request, check if the target key is matched in the Memtable in memory. If matched, return directly; if not matched, search the ImmuTable. If matched, return directly. S3. If the target key is not found in either Memtable or ImmuTable, then query the hot locator. Query the hot SSTable index at each level of the hot locator, starting from the L0 ordered mapping and continuing to the last ordered mapping. The query rule is to return the information of the first entry whose maximum key is greater than or equal to the target key, and obtain the corresponding SSTable information based on the entry information. S4. If the target SSTable information is located in the hot locator, then query the double-layer hash cache according to the SSTable file number, i.e., SSTable ID. The query rule is: first, locate the target bucket by hashing the file number; second, query the inner hash table according to the target bucket; and finally, return successfully after locating the bucket of the target key in the inner hash table. S5. If the query fails in the hot locator or the two-level hash cache, then query layer by layer according to the query logic of the original LSM tree. S6. After a successful query, increase the popularity of the corresponding SSTable and calculate whether the popularity value has reached the predefined threshold. If the SSTable has not reached the predefined threshold, return the query result directly. S7. If the SSTable reaches the predefined threshold, add the SSTable index to the hot locator, insert the maximum key of the SSTable, cache the pointer of the SSTable for subsequent access to directly locate the target key, and cache the key-value pair in the double-layer hash cache. S8. Update the LRU list and record the last access time. Subsequent accesses will discard the key node that has not been accessed for the longest time and has the lowest access time based on its popularity and the last access time.

2. The hot-read optimization design method for an LSM tree-based key-value storage system according to claim 1, characterized in that: The hot locator is an ordered key-value mapping component used to maintain the mapping relationship between the largest key in the hot SSTable and the corresponding SSTable file pointer. The key stores the largest key of each SSTable and its corresponding file pointer, and the keys are ordered. With the help of the ordered nature of the keys, the first entry that meets the query conditions can be quickly located by searching, thereby determining the target SSTable.

3. The hot-read optimization design method for an LSM tree-based key-value storage system according to claim 1 or 2, characterized in that: The hot locator has a hierarchical index system corresponding to the LSM tree hierarchy, which can search layer by layer in hierarchical order to ensure that the SSTable containing the latest data is hit first. It also uses an array structure to uniformly manage the capacity of the ordered mapping. Furthermore, an index synchronization update mechanism is introduced during the compression process: when the SSTable participating in the compression already has a corresponding index entry in the hot locator, after the compression is completed, the newly generated SSTable is automatically added to the hot locator, and the mapping relationship between its maximum key and the file pointer is established. At the same time, the original invalid index entries are removed in a timely manner.

4. The hot-read optimization design method for an LSM tree-based key-value storage system according to claim 1, characterized in that: The dual-layer hash cache consists of nested inner and outer hash tables and an LRU table. The outer hash table maps SSTable IDs to the corresponding inner hash table, while the inner hash table further maps keys to nodes in the LRU linked list. Through this hierarchical mapping mechanism, data in different SSTables is divided into independent inner hash spaces, thereby effectively reducing the probability of hash collisions and shortening the query path. At the same time, the inner hash table, combined with the LRU eviction policy, retains only frequently accessed hot key-value pairs. When the cache capacity reaches its limit, the least recently used entry is evicted first.

5. The hot-read optimization design method for an LSM tree-based key-value storage system according to claim 1 or 4, characterized in that: The dual-level hash cache employs a hotspot determination strategy based on SSTables. Specifically, an SSTable is considered to satisfy the hotspot data determination criteria only when it is identified as a hot file and included in the hot locator index structure. Furthermore, a synchronous update mechanism is introduced during compression: when an Immutable Memtable is flushed to the SSD, the system uses an iterator to traverse the key-value pairs in the Memtable. During this traversal, the corresponding entries in the dual-level hash cache are updated simultaneously to ensure that the key-value pairs in the cache remain valid.

6. The hot-read optimization design method for an LSM tree-based key-value storage system according to claim 1, characterized in that: In step S5, the query logic of the original LSM tree is as follows: First, query the L0 level of the LSM tree. If there is no SSTable containing the target key in the L0 level, then query the SSTables in the other levels until an SSTable containing the target key is found, then the query is successful.

7. The hot-read optimization design method for an LSM tree-based key-value storage system according to claim 1, characterized in that: The predefined threshold is as follows: when the number of accesses to a certain SSTable reaches 0.5% of the total number of accesses, it is identified as a hot SSTable, and the mapping relationship between the maximum key of the SSTable and its file pointer is inserted into the hot locator.

Images