Key value storage system indexing method oriented to NVM-NVMe SSD hybrid architecture

Abstract

The invention discloses a key value storage system indexing method oriented to an NVM-NVMe SSD (Non-Volatile Memory-Non-Volatile Memory Express Solid State Disk) hybrid architecture, which comprises the following steps of: constructing a heterogeneous storage architecture taking cold and hot data perception as a core driving mechanism, coordinating and managing two types of storage media, namely a non-volatile memory NVM and a solid state disk NVMe SSD, and realizing efficient identification, layered writing and dynamic migration of cold and hot data. According to the method, the characteristics of low delay, durability and byte addressing of the NVM are utilized, the frequency and I / O overhead of Flush and Compaction operations are reduced in a data write-in path, meanwhile, the problem of mixed storage of cold and hot data is avoided, and therefore the overall performance and storage efficiency of a system are remarkably improved. In addition, by introducing an asynchronous migration module, the method can dynamically adapt to the change of data popularity along with time evolution, and effectively support high performance and high availability of the key value system in long-term operation.

Description

Technical Field

[0001] This invention relates to the field of key-value storage system indexing technology, and in particular to an indexing method for key-value storage systems under an NVM-NVMe SSD hybrid architecture. Background Technology

[0002] With the continuous evolution of information technology, the speed of global data generation and accumulation continues to accelerate, especially the explosive growth of unstructured data. Massive amounts of text, image, audio, and video data are continuously generated from applications such as social media, smart terminals, audio and video platforms, and generative artificial intelligence. These unstructured data types are complex and structurally diverse, gradually becoming key assets driving enterprise intelligent transformation and digital upgrading. Faced with such a massive and ever-growing data scale, how to build an unstructured data storage system with high concurrency, high throughput, and strong adaptability has become a crucial challenge to be addressed in cloud computing, big data platforms, and edge computing environments. While traditional relational databases have a mature theoretical and practical foundation in transaction consistency and structured data management, their fixed-schema table structure design and engine architecture optimized for random access limit their performance in handling frequent small file reads and writes and dynamic expansion, making it difficult to meet the performance requirements of emerging data-intensive scenarios. Therefore, non-relational databases (NoSQL) have emerged and, with their flexible data models, good scalability, and efficient read and write capabilities, have gained widespread application in unstructured data management.

[0003] Among numerous NoSQL database architectures, key-value stores (KVStore) based on the Log-Structured Merge-Tree (LSM-Tree) structure stand out, becoming one of the main solutions for handling large-scale unstructured data. LSM-Tree significantly improves write throughput and reduces I / O overhead from random writes by appending write operations to a write buffer in memory and writing them to disk in batches. Furthermore, its hierarchical structure maintains disk data ordering through periodic compaction operations, achieving efficient read-write balancing and space reclamation mechanisms. Therefore, the LSM-Tree architecture has become a core technology in NoSQL databases, widely used in high-performance critical systems such as search engines, e-commerce platforms, content recommendation, and social services. With the continuous expansion of data scale and the increasing read / write access load, the overall performance of key-value stores becomes increasingly important.

[0004] However, in-depth analysis revealed that the main bottlenecks restricting further performance improvement lie in two key processes within the system: Flush and Compaction. These two operations not only incur frequent disk I / O overhead but can also significantly interfere with foreground request processing, becoming core obstacles affecting system throughput and response efficiency. Specifically, when the write buffer in memory (such as a memory table) reaches its capacity limit, the system needs to flush its data to the SSTable (sorted string table) at level L0 of the LSM-Tree to free up memory space for subsequent writes. This flushing process involves file generation and writing, thus triggering frequent low-level I / O operations, resulting in significant performance jitter in high-concurrency write scenarios. Furthermore, to maintain the read efficiency and storage order of the LSM-Tree, the system also needs to periodically perform Compaction operations, selecting and merging SSTable files with overlapping key ranges from adjacent levels into new files. Because compaction involves moving and rewriting a large amount of data between multiple levels, the process is accompanied by intensive read and write I / O loads. This not only exacerbates the write amplification effect of the storage medium, but also makes it very easy to compete for resources with the foreground write path, resulting in a significant increase in write latency and seriously affecting the overall write throughput and response performance of the system.

[0005] Existing research mainly focuses on the LSM-Tree structure itself, the compaction mechanism, and the evolution of storage media, proposing various optimization strategies to improve the overall performance of key-value storage systems. To alleviate the performance bottleneck caused by compaction operations, some works attempt to reduce the number of LSM-Tree layers to decrease the frequency of compaction triggers. Other studies, from a data organization perspective, introduce multiple fencepointers to partition key-value data into regions, effectively limiting the amount of data involved in each compaction. Furthermore, some studies optimize the compaction process itself by more rationally selecting the set of SSTable files to be merged, reducing unnecessary I / O burden while ensuring read and write performance. On the other hand, with the development of new non-volatile storage technologies, some studies introduce non-volatile memory (NVM) to further improve system performance. One type of research uses NVM to expand the write buffer capacity in DRAM, thereby reducing the frequency of flush operations; another type places NVM between DRAM and NVMe SSD as a high-speed intermediate cache layer to cache some of the most recently written data, alleviating the pressure on the backend compaction.

[0006] While existing research has improved the overall performance of key-value storage systems to some extent, a core problem that is generally overlooked is the lack of a mechanism to perceive and differentiate between hot and cold data states. Current mainstream designs often treat all data equally and store it on NVM or NVMe SSDs in the write path, failing to effectively differentiate based on access frequency or data activity. This indiscriminate storage strategy cannot fundamentally solve the performance bottleneck caused by frequent flush and compaction triggers, and still exacerbates write amplification and I / O conflicts, limiting further performance improvements of key-value systems.

[0007] Based on the above problems, this invention proposes an indexing method and system for key-value storage systems under an NVM-NVMe SSD hybrid architecture. The system consists of a parallel storage architecture composed of non-volatile memory (NVM) and solid-state drives (NVMe SSD), which respectively undertake the storage tasks of high-frequency and low-frequency data. Summary of the Invention

[0008] To address the aforementioned problems, this invention aims to provide an indexing method for key-value storage systems in an NVM-NVMe SSD hybrid architecture. This method dynamically analyzes data access characteristics during data writing and compaction, accurately identifying hot and cold data and storing them separately in NVM and NVMe SSDs, thereby achieving a high-efficiency balance between performance and storage cost. Through this awareness-driven hierarchical storage approach, the system can significantly reduce the proportion of hot data in NVMe SSDs, effectively reducing the frequency of compaction operations caused by multiple versions of the same data. Furthermore, considering the media characteristics of NVM, this invention constructs a lightweight PMTable in NVM based on a persistent skip list structure. Utilizing NVM's byte addressing capabilities, it can be constructed through memory copying of a hot-mutable memory table in DRAM, avoiding additional I / O overhead during the flush process. In addition, data merging between PMTables can be accomplished through internal pointer adjustments, rather than traditional file-level rewriting, further reducing the data organization cost within NVM. On the NVMe SSD side, since hot data is effectively offloaded to NVM for storage during the write phase, the NVMe SSD mainly carries cold data, which significantly reduces the frequency of compaction triggering and resource overhead, and releases greater scheduling flexibility for background merging operations.

[0009] To address the issue of inconsistent hot and cold data storage caused by the dynamic changes in data hot and cold status over time, this invention designs an asynchronous hot and cold data migration module. This module runs as a task queue and background thread, enabling on-demand migration and location updates of hot and cold data between NVM and NVMe SSDs without blocking foreground requests. This mechanism ensures that the hot or cold status of data remains consistent with its storage location throughout its lifecycle, preventing hot data from being cold-processed or cold data from occupying high-performance storage resources for extended periods.

[0010] This invention provides an indexing method for key-value storage systems under an NVM-NVMe SSD hybrid architecture, comprising:

[0011] During Put / Update / Delete operations, the system uses a hot / cold data sensing module to determine the popularity of key-value pairs. Hot key-value pairs are written directly to the hot variable memory table, while cold key-value pairs are written to the cold variable memory table. When the cold variable memory table is full, the system triggers a refresh operation, writing the cold data to an SSTable based on an embedded index block structure and storing it in the NVMe SSD. When the hot variable memory table is full, the system triggers a refresh operation, writing the hot data to a PMTable based on a persistent skip list index structure and storing it in the NVM.

[0012] During a Get operation, the system first reads the key-value pair from the hot variable memory table / cold variable memory table. If no match is found, it searches layer by layer in the LSM-Tree in NVM. If no match is found again, it reads from the NVMe SSD.

[0013] Among them, NVM and NVMe SSDs periodically trigger compaction operations to merge duplicate data and clean up useless old data.

[0014] In one embodiment of the present invention, the NVM triggers a compaction operation using a tiering strategy, including the following steps:

[0015] In this layer, the target number of PMTables is selected and merged using internal pointers. During compaction, cold data is migrated through an asynchronous cold and hot data migration module to reduce the space usage of NVM.

[0016] In one embodiment of the present invention, the NVMe SSD determines a threshold based on the proportion of invalid keys and triggers a compaction operation, as expressed by the formula:

[0017]

[0018] Among them, UniqueKeys(SST1,SST2,…,SST) n) represents the number of unique keys in each SSTable, NumKeys(SST i ) represents the total number of keys in the i-th SSTable, and r represents the proportion of invalid keys.

[0019] In one embodiment of the present invention, the hot and cold data sensing module adopts a cache structure of four lists: T1 / T2 / G1 / G2, wherein:

[0020] The main cache list T1 is used to record data items that have been accessed recently but have not been used frequently.

[0021] The main cache list T2 is used to record frequently accessed data items;

[0022] Ghost list G1 is used to track historical entries that have been removed from T1;

[0023] The ghost list G2 is used to track historical entries that have been removed from T2.

[0024] In one embodiment of the present invention, the hot / cold sensing module employs an adaptive cache replacement ARC strategy, and the processing flow for key-value pair data includes:

[0025] First access: If the data is not in T1 / T2 / G1 / G2, the cold and heat sensing module marks it as having an unknown temperature and inserts it into T1 first. This type of data can be written to the cold variable memory table.

[0026] Hitting T1: This means that the data has been accessed again, migrated to T2, and marked as hot data. This type of data can be written to a hot variable memory table.

[0027] Hitting T2: Data remains at T2 and accumulates access frequency; this type of data is judged as a stable hotspot.

[0028] Hitting G1: This indicates that previously active data has reappeared. The system determines that it should enhance its response to recent access behavior, appropriately increase the capacity of T1, and re-add the data to T2.

[0029] Hitting G2: This means that data that was previously accessed frequently has reappeared. The system will pay more attention to the access frequency and increase the capacity of T2 accordingly, and the data will be added back to T2.

[0030] In one embodiment of the present invention, the system maintains a persistent hash index in NVM to record the location of valid key-value pairs within the SSTable in the NVMe SSD layer.

[0031] In one embodiment of the present invention, when the NVM and NVMe SSD periodically trigger the compaction operation, the hot and cold data blocks are identified based on the hot and cold data perception module, and the identified hot and cold data blocks are migrated based on the asynchronous hot and cold data migration module.

[0032] In one embodiment of the present invention, the NVM triggers a compaction operation, the steps of which include:

[0033] When a compaction operation occurs at the L0 layer of the LSM-Tree in NVM, the system identifies the hot and cold attributes of the data blocks currently participating in the merging through the hot and cold data awareness module. When a data block is determined to be cold data, the system adds the cold data block to the asynchronous cold task queue. The background thread in the asynchronous hot and cold data migration module autonomously polls the cold task queue, extracts tasks from it, and reads the corresponding cold data block from NVM and writes it to the SSTable in the NVMe SSD. At the same time, the storage location information of the data block in the NVMe SSD is updated and written to the persistent hash index maintained in NVM.

[0034] In one embodiment of the present invention, the NVMe SSD triggers a compaction operation, the steps of which include:

[0035] When a compaction operation occurs within a storage bucket in an NVMe SSD, the system identifies the hot and cold attributes of the data blocks currently participating in the merging through the hot and cold data awareness module. When a data block is determined to be hot data, the system adds the hot data block to the asynchronous hot task queue. The background thread in the asynchronous hot and cold data migration module then autonomously polls the cold task queue, extracts the task, and reads the corresponding hot data block from the NVMe SSD and writes it to the PMTable in NVM. At the same time, it updates the corresponding new location information in the persistent hash index.

[0036] The beneficial effects of this invention are:

[0037] This invention proposes an indexing method and system for key-value storage systems under an NVM-NVMe SSD hybrid architecture. By constructing a heterogeneous storage architecture with hot / cold data awareness as the core driving mechanism, it coordinates and manages both non-volatile memory (NVM) and solid-state drives (SSDs), achieving efficient identification, tiered writing, and dynamic migration of hot and cold data. The system design fully leverages the low latency, persistence, and byte-addressing characteristics of NVM, reducing the frequency and I / O overhead of flush and compaction operations in the data write path, while avoiding the problem of mixed hot and cold data storage, thus significantly improving the overall system performance and storage efficiency. Furthermore, by introducing an asynchronous migration module, this invention can dynamically adapt to changes in data popularity over time, effectively supporting the high performance and high availability of the key-value system during long-term operation. The advantages of this invention are as follows:

[0038] 1) Through the hot and cold data sensing module, the system can accurately identify the access frequency of data during the writing and compaction process, and write hot data to NVM and cold data to SSD accordingly, which significantly reduces unnecessary compaction frequency and improves system throughput.

[0039] 2) In NVM, a persistent skip list-based data structure (PMTable) is used, combined with its byte addressing capability, to achieve direct memory copying of the memory table in DRAM to disk, avoiding the file I / O overhead of the traditional flush process. Simultaneously, PMTable merging is performed through pointer operations, further reducing I / O overhead.

[0040] 3) Hot data resides in NVM first, which effectively reduces the write pressure and data redundancy of SSD, thereby reducing the probability of SSTable compaction and reducing I / O overhead and resource contention of the foreground write path by background merging operations.

[0041] 4) Introducing an asynchronous hot and cold data migration module based on task queues and background threads, supporting on-demand data flow between NVM and SSD, ensuring that data always resides on the most suitable storage medium according to its popularity, improving access efficiency and saving resources. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the NVM-NVMe SSD hybrid architecture of the present invention;

[0043] Figure 2 A schematic diagram illustrating the process of writing an indexing method for an existing key-value storage system;

[0044] Figure 3 This is a schematic diagram of the writing process of the indexing method in the key-value storage system of the present invention;

[0045] Figure 4 This is a schematic diagram of the cold and heat data sensing module structure of the present invention;

[0046] Figure 5 This is a schematic diagram of the data processing flow of the cold and heat data sensing module of the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the compaction operation principle of the storage medium of the present invention.

[0048] Figure 7 This is a schematic diagram of the compaction operation process of the storage medium of the present invention. Detailed Implementation

[0049] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0050] The existing method's conventional write process is as follows: Figure 2 As shown, existing methods lack a mechanism to perceive and differentiate between hot and cold data states. They often treat all data equally and store it on NVM or NVMe SSDs in the write path, failing to effectively differentiate based on access frequency or data activity. This indiscriminate storage strategy cannot fundamentally solve the performance bottleneck caused by frequent flush and compaction triggers, exacerbates write amplification and I / O conflicts, limits further performance improvements in key-value systems, and leads to the problem of chaotic hot and cold data storage due to the dynamic change of data hot and cold states over time.

[0051] To effectively address the aforementioned issues, this invention discloses an indexing method for key-value storage systems in an NVM-NVMe SSD hybrid architecture. It employs a tiered storage approach based on hot / cold data awareness to reduce the impact of I / O operations on system performance. Simultaneously, different data organization methods are used within each storage medium to optimize the key-value storage system. An asynchronous hot / cold data migration module is designed, running as a task queue and background thread. This module enables on-demand migration and location updates of hot / cold data between NVM and NVMe SSDs without blocking foreground requests. This mechanism ensures that the hot / cold status of data remains consistent with its storage location throughout its lifecycle, preventing hot data from being cold-processed or cold data from occupying high-performance storage resources for extended periods.

[0052] Example 1:

[0053] This embodiment discloses an indexing method for key-value storage systems under an NVM-NVMe SSD hybrid architecture, such as... Figure 1 and Figure 3 As shown, it includes:

[0054] When performing Put / Update / Delete operations in RocksDB, the system first uses the hot / cold data awareness module to determine the popularity of the key-value pair and writes the hot / cold key-value pairs into the corresponding write-ahead log (WAL) for backup.

[0055] Next, hot key-value pairs are written directly to the hot mutable memory table, while cold key-value pairs are written to the cold mutable memory table. When the cold mutable memory table is full, the system triggers a flush operation, writing the cold data to an SSTable based on an embedded index block structure stored on an NVMe SSD. The hot mutable memory table is then flushed to a PMTable based on a persistent skip list index structure stored in non-volatile memory (NVM).

[0056] Generally, hot mutable memory tables are refreshed less frequently than cold mutable memory tables. This delayed refresh strategy allows hot data to remain in memory for a longer period, thus accelerating access to this data. In contrast, cold data is periodically migrated to NVMe SSDs to balance storage efficiency and data access performance.

[0057] In RocksDB, during a Get operation, the key-value pairs are first read from both the hot mutable and immutable memory tables, as well as the cold mutable and immutable memory tables. If no match is found, the search proceeds layer by layer through the LSM-Tree in persistent memory. If still no match is found, the final read operation is performed on the NVMe SSD. To avoid high read latency caused by high I / O overhead during reads from the NVMe SSD layer, the system maintains a persistent hash index to record the positions of valid key-value pairs within the SSTable in the NVMe SSD layer. This approach allows the target key to be located with a single I / O operation, thereby improving read efficiency.

[0058] Furthermore, in order to maintain the high performance of LSM-Tree in NVM and NVMe SSD, LSM-Tree needs to trigger a compaction operation periodically to merge duplicate data and clean up useless old data.

[0059] In NVM, PMTables are first written to the L0 layer of the LSM-Tree. When the number of PMTables in the L0 layer reaches the compaction threshold, a compaction operation is triggered.

[0060] During compaction, NVM employs a tiering strategy. Specifically, it selects the target number of PMTables within the current tier and completes the merging process using internal pointers. This process avoids data movement, significantly reducing I / O overhead during merging. Simultaneously, during compaction, an asynchronous cold / hot data migration module migrates cold data to further reduce NVM's space footprint.

[0061] In NVMe SSDs, SSTables are first written to a storage bucket. When the number of SSTables in the storage bucket reaches a preset compaction threshold, the system will determine whether compaction needs to be performed.

[0062] The specific method is as follows: Each SSTable maintains a HyperLogLog estimator, which is used to efficiently estimate the number of unique keys in key-value pairs within the bucket. Typically, if multiple SSTables have a large overlap in key ranges, they may contain multiple versions of the same key value, resulting in a large number of invalid key-value pairs. Therefore, when the system collects multiple SSTables with a high degree of key overlap, it will evaluate them according to the following steps:

[0063] First, count the total number of keys in these SSTables. Assume there are n SSTables, where the total number of keys in the i-th SSTable is NumKeys(SST). i ).

[0064] Then, the number of unique keys is calculated by calling the HyperLogLog's MergedEstimate() API to calculate the number of unique keys in these SSTables, denoted as UniqueKeys(SST1,SST2,…,SST). n ).

[0065] Then, calculate the proportion of invalid keys by comparing the total number of keys with the total number of unique keys, and obtain the proportion r of invalid keys, as shown in the formula:

[0066]

[0067] Here, 'r' represents the proportion of invalid keys. If 'r' exceeds a preset trigger threshold, it indicates a large amount of redundant data in the SSTable storage. The system will then trigger a compaction operation to merge valid key values ​​and clean up invalid data. During compaction, an asynchronous hot data migration module will also migrate hot data to improve access to it. If 'r' does not reach the trigger threshold, the system will delay compaction execution and wait for the next assessment. The core objective of this delay mechanism is to improve the utilization of NVMe SSD storage space while controlling the compaction trigger frequency based on a custom threshold, thereby reducing the performance impact on foreground write requests.

[0068] Example 2:

[0069] Based on Example 1, the structure of the hot and cold data sensing module is optimized, specifically, as follows: Figure 4 and Figure 5 As shown, the hot and cold data sensing module adopts a cache structure of four lists: T1 / T2 / G1 / G2.

[0070] The hot and cold data perception module employs an adaptive cache replacement (ARC) strategy to fully combine historical access frequency with recent access behavior, thereby achieving more refined and dynamic data popularity judgment.

[0071] During system operation, the hot and cold data sensing module is responsible for sensing the data popularity in real time on the write path. The core design is based on a four-list cache structure, which combines the advantages of LRU (Least Recently Used) and LFU (Least Frequently Used) strategies.

[0072] Specifically, the module maintains two main cache lists: T1 records data items that have been accessed recently but have not been used frequently, and T2 records data items that have been accessed frequently; it also maintains two ghost lists, G1 and G2, to track historical entries that have been removed from T1 and T2, respectively.

[0073] This architecture enables the system to dynamically evaluate and adjust the allocation ratio of cached resources between recently accessed and frequently accessed resources, thereby achieving adaptive hot-spot identification.

[0074] When a write request arrives from the front end, the hot / cold sensing module processes it according to the following procedure:

[0075] First access (new key): If the data is not in T1 / T2 / G1 / G2, the module marks it as having an unknown temperature and inserts it into T1 first. This type of data can be written to the cold variable memory table. If T1+T2 exceeds the cache limit, a replacement is triggered. The replacement strategy is adaptively adjusted based on the feedback from G1 / G2.

[0076] Hit T1 (secondary access): This indicates that the data has been accessed again, migrated to T2, and marked as hot data. This type of data can be written to a hot-moving memory table.

[0077] Hitting T2 (frequent access): Data remains at T2 and the access frequency accumulates. This type of data is judged as a stable hotspot.

[0078] Hitting G1 (high relevance): This indicates that previously active data has reappeared. The system determines that it should enhance its response to recent access behavior, appropriately increase the capacity of T1, and re-add the data to T2.

[0079] Hitting G2 (high frequency): This means that data that was previously accessed frequently has reappeared. The system will pay more attention to the access frequency and increase the capacity of T2 accordingly; the data will also be added back to T2.

[0080] Unlike traditional LRU or LFU strategies, ARC does not rely on fixed hot / cold threshold judgment mechanisms. Instead, it dynamically adjusts strategy weights through real-time feedback and hit status, exhibiting excellent adaptability. This feature enables the module to effectively identify data temperature during the write phase, avoiding misjudgments of data temperature due to missing information during the initial write process. This is particularly suitable for real-world business scenarios with frequent fluctuations in data access patterns and rapid changes in hotspots. Leveraging the flexibility and accuracy of ARC in hot / cold identification, this invention achieves highly reliable heat perception capabilities in the foreground write path, providing solid criterion support for subsequent hot / cold tiered storage strategies and asynchronous hot / cold migration mechanisms, while further improving the overall system performance and resource utilization efficiency.

[0081] Example 3:

[0082] Based on Example 1 or 2, this example optimizes the periodic triggering of compaction operations for NVM and NVMe SSDs. Specifically, as follows: Figure 6 and Figure 7 As shown, when NVM and NVMe SSD periodically trigger compaction operations, the cold and hot data blocks are identified based on the cold and hot data perception module, and the identified cold and hot data blocks are migrated based on the asynchronous cold and hot data migration module.

[0083] In NVM, the system will be in the LSM-Tree L n-1 When a compaction operation occurs at a layer, the cold / hot data awareness module identifies the cold / hot attributes of the data blocks currently participating in the merging. When a data block is determined to be cold data, the system does not immediately migrate it out. Instead, it constructs a migration task that encapsulates the metadata of the cold data block and adds it to a separate asynchronous cold task queue.

[0084] The entire compaction process continues as usual, eventually generating a new PMTable file, while the old PMTable file containing cold data is logically marked as recyclable.

[0085] Subsequently, the background thread in the asynchronous cold and hot data migration module autonomously polls the cold task queue, extracts tasks from it, and reads the corresponding cold data blocks from NVM and writes them to the SSTable in the NVMe SSD. At the same time, it updates the storage location information of the data block in the NVMe SSD and writes it to the persistent hash index maintained in NVM to ensure the correctness and consistency of subsequent read requests.

[0086] Accordingly, to address the issue of hot data potentially impacting read performance in NVMe SSDs, a symmetrical hot data migration mechanism is designed on the NVMe SSD side. During each SSTable compaction operation in the NVMe SSD, the hot data awareness module assesses the current data block's activity level. If some data is identified as hot data, the system does not immediately migrate this data back. Instead, it packages its relevant information into a hot migration task and adds it to the asynchronous hot task queue. The NVMe SSD compaction operation continues, generating a new SSTable, while the old SSTable containing hot data is marked as reclaimable. The background thread of the asynchronous hot data migration module then retrieves tasks from the hot task queue, reads the corresponding hot data blocks, writes them to the newly generated PMTable in NVM, and simultaneously updates their corresponding new location information in the persistent hash index.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An indexing method for key-value storage systems under an NVM-NVMe SSD hybrid architecture, characterized in that, include: During Put / Update / Delete operations, the system uses the hot and cold data sensing module to determine the popularity of key-value pairs. Hot key-value pairs are directly written to the hot variable memory table, and cold key-value pairs are written to the cold variable memory table. When the cold variable memory table is full, the system triggers a refresh operation to write the cold data to an SSTable based on an embedded index block structure and store it in the NVMe SSD. When the hot variable memory table is full, the system triggers a refresh operation, writing the hot data into a PMTable based on a persistent skip list index structure and storing it in NVM; During a Get operation, the system first reads the key-value pair from the hot variable memory table / cold variable memory table. If no match is found, it searches layer by layer in the LSM-Tree in NVM. If no match is found again, it reads from the NVMe SSD. Among them, NVM and NVMe SSDs periodically trigger compaction operations to merge duplicate data and clean up useless old data.

2. The indexing method for a key-value storage system under a hybrid NVM-NVMe SSD architecture according to claim 1, characterized in that, The NVM triggers a compaction operation using a tiering strategy, including the following steps: In this layer, the target number of PMTables is selected and merged using internal pointers. During compaction, cold data is migrated through an asynchronous cold and hot data migration module to reduce the space usage of NVM.

3. The indexing method for a key-value storage system under an NVM-NVMe SSD hybrid architecture according to claim 1, characterized in that, The NVMe SSD triggers a compaction operation based on a threshold determined by the proportion of invalid keys, as shown in the formula: Among them, UniqueKeys(SST1,SST2,…,SST) n ) represents the number of unique keys in each SSTable, NumKeys(SST i ) represents the total number of keys in the i-th SSTable, and r represents the proportion of invalid keys.

4. The indexing method for key-value storage systems under a hybrid NVM-NVMe SSD architecture according to claim 1, characterized in that, The hot and cold data sensing module adopts a cache structure of four lists: T1 / T2 / G1 / G2, wherein: The main cache list T1 is used to record data items that have been accessed recently but have not been used frequently. The main cache list T2 is used to record frequently accessed data items; Ghost list G1 is used to track historical entries that have been removed from T1; The ghost list G2 is used to track historical entries that have been removed from T2.

5. The indexing method for a key-value storage system under an NVM-NVMe SSD hybrid architecture according to claim 4, characterized in that, The hot / cold sensing module uses an adaptive caching strategy to replace ARC, and the processing flow for key-value pair data includes: First access: If the data is not in T1 / T2 / G1 / G2, the cold and heat sensing module marks it as having an unknown temperature and inserts it into T1 first. This type of data can be written to the cold variable memory table. Hitting T1: This means that the data has been accessed again, migrated to T2, and marked as hot data. This type of data can be written to a hot variable memory table. Hitting T2: Data remains at T2 and accumulates access frequency; this type of data is judged as a stable hotspot. Hitting G1: This indicates that previously active data has reappeared. The system determines that it should enhance its response to recent access behavior, appropriately increase the capacity of T1, and re-add the data to T2. Hitting G2: This means that data that was previously accessed frequently has reappeared. The system will pay more attention to the access frequency and increase the capacity of T2 accordingly, and the data will be added back to T2.

6. The indexing method for a key-value storage system under an NVM-NVMe SSD hybrid architecture according to claim 1, characterized in that, The system maintains a persistent hash index in NVM to record the location of valid key-value pairs within the SSTable in the NVMe SSD layer.

7. The indexing method for a key-value storage system under a hybrid NVM-NVMe SSD architecture according to claim 1, characterized in that, When the NVM and NVMe SSD periodically trigger the compaction operation, the cold and hot data blocks are identified based on the cold and hot data perception module, and the identified cold and hot data blocks are migrated based on the asynchronous cold and hot data migration module.

8. The indexing method for a key-value storage system under an NVM-NVMe SSD hybrid architecture according to claim 7, characterized in that, The NVM triggers a compaction operation, including the following steps: The system in NVM LSM-Tree L n-1 When a compaction operation occurs in a layer, the cold and hot data awareness module identifies the cold and hot attributes of the data blocks currently participating in the merging. When a data block is determined to be cold data, the system adds the cold data block to the asynchronous cold task queue. The background thread in the asynchronous cold and hot data migration module autonomously polls the cold task queue, extracts tasks from it, and reads the corresponding cold data block from NVM and writes it to the SSTable in the NVMe SSD. At the same time, the storage location information of the data block in the NVMe SSD is updated and written to the persistent hash index maintained in NVM.

9. The indexing method for a key-value storage system under an NVM-NVMe SSD hybrid architecture according to claim 7, characterized in that, The NVMe SSD triggers a compaction operation, including the following steps: When a compaction operation occurs within a storage bucket in an NVMe SSD, the system identifies the hot and cold attributes of the data blocks currently participating in the merging through the hot and cold data awareness module. When a data block is determined to be hot data, the system adds the hot data block to the asynchronous hot task queue. The background thread in the asynchronous hot and cold data migration module then autonomously polls the cold task queue, extracts the task, and reads the corresponding hot data block from the NVMe SSD and writes it to the PMTable in NVM. At the same time, it updates the corresponding new location information in the persistent hash index.

Images