Multi-NDP system read-write acceleration method based on key value separation technology

Abstract

The invention discloses a multi-NDP (Near-Data Processing) system read-write acceleration method based on a key value separation technology, and the method comprises the steps: when data is written, splitting a data key value pair written into a batch processing region of a multi-NDP system, determining target equipment, then carrying out key processing and value processing, carrying out key storage on keys, and carrying out data storage on the target equipment; a value corresponding to the key is added to a value log file of the target equipment; during data query, hash operation is carried out on a query key, target device position information of a corresponding value is obtained, then the master-slave multi-level hot cache is queried, and query retrieval of the corresponding value is carried out according to the obtained value position index. According to the method, the key value separation technology is introduced into the multi-NDP system, when the Compaction is executed, the execution time of the Compaction is shortened, the influence of the Compaction on the writing operation of the user is reduced, and the writing performance is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-NDP system data reading and writing processing, and in particular to a multi-NDP system reading and writing acceleration method based on key-value separation technology. Background Art

[0002] According to forecasts, the total amount of global data is expected to grow from 44ZB (1ZB = 10^21 bytes) in 2020 to 175ZB in 2025. Among them, unstructured data used in specific data-driven application fields (such as social media) will account for more than 80% of the total global data. Therefore, efficient storage and processing of these unstructured data has become a cutting-edge hot technology for current big data intelligent processing and has received widespread attention from the industry.

[0003] The complexity and diversity of unstructured data make it difficult for traditional relational databases to store and manage. Key-value storage based on LSM-tree data organization is widely used to efficiently manage unstructured data due to its simplicity and flexibility, such as LevelDB, RocksDB, Redis, and Cassandra. In LSM-tree, data is first written to a memory component with a log organization structure, and then dumped to a disk component with a hierarchical sorted string structure periodically in batches, converting random writes into sequential writes to improve write performance. When the memory component reaches a predetermined threshold, the data is sorted and dumped into a new SSTable. Over time, multiple SSTables with overlapping key ranges will accumulate on the disk, resulting in a waste of storage space and reduced read performance. At this time, LSM-tree regularly performs compaction operations to merge SSTables with overlapping key ranges into new tables without overlapping key ranges, thereby reducing disk space and optimizing read performance.

[0004] Traditional CPU-centric architectures optimize compaction, such as WiscKey, HashKV, and gLSM. The compaction operations of these methods still involve the movement of a large amount of data between storage devices (such as SSDs and HDDs) and the CPU, which brings huge overhead.

[0005] Therefore, the Near Data Processing (NDP) architecture was introduced to handle compaction operations. For example, CoKV, PStore, and nKV start from the perspective of a single NDP system, allowing compaction to be processed closer to the storage medium, reducing data movement and thus accelerating compaction. However, the processing capacity of a single NDP is difficult to cope with the processing needs of large-scale data; therefore, MStore proposed a multi-NDP architecture based on this.

[0006] However, the multi-NDP system represented by MStore still has some problems: in MStore, by introducing multiple NDP devices, although the capacity of each layer of LSM-tree is expanded and the hierarchical structure of LSM-tree is reduced, this design significantly reduces the writing frequency of high-level data and the number of high-level compactions, thereby effectively reducing the write amplification effect. Although this method performs well under write-intensive and read-intensive loads, the write amplification problem still exists significantly under mixed read-write loads; in addition, in large-scale data environments (such as 100GB of data), studies have observed that the improvement of MStore system in write performance optimization far exceeds the optimization of read performance, which leads to serious asymmetry between read and write performance, limiting the overall performance of the system under read-intensive loads.

[0007] Therefore, in reality, a multi-NDP system read and write acceleration method based on key-value separation technology is needed to improve the read and write performance of the multi-NDP system and reduce the write amplification effect. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a multi-NDP system read and write acceleration method based on key-value separation technology, introduce key-value separation technology into the multi-NDP system, reduce the execution time of compaction, improve the read and write performance of the multi-NDP system, and reduce the write amplification effect.

[0009] The embodiment of the present invention provides a multi-NDP system read-write acceleration method based on key-value separation technology, including a multi-NDP system composed of a host end and a device end, when:

[0010] When writing data, the key-value pairs of data written into the batch processing area of ​​the multi-NDP system are split, the key is hashed to determine the target device, and then key processing and value processing are performed. After key processing, the target device stores the key, and the corresponding value after value processing is appended to the value log file of the target device; during the value processing process, when the system cannot transmit all data at one time, the multi-NDP system value storage adopts a pipeline flush strategy;

[0011] When querying data, the query key is first hashed, and then a master-slave multi-level hot cache strategy is adopted to sequentially search the memory table (MemTable), the immutable memory table (Immutable MemTable), the host cache, the device cache, and the LSM-tree (Log-Structured Merge Tree) of the target device. Once the location information of the target value is obtained, the index information is returned and the corresponding value is queried based on the index.

[0012] When the percentage of dirty data in the value log file exceeds the threshold and garbage collection is triggered, a local garbage collection (GC) operation is performed through a background thread.

[0013] Furthermore, when the data is written:

[0014] Key processing includes: writing the key to the memory table (MemTable) of the corresponding target device. When the MemTable reaches the predetermined threshold, the system converts it into an immutable memory table (Immutable MemTable), and then triggers the key Flush operation to write the data to the target device disk and generate an SST file; when the Compaction operation is triggered, the system performs the key Compaction operation in the background, updates the dirty key index table, and merges and organizes the SST files;

[0015] The value processing includes: writing the value into the vTable in the value buffer vBuffer of the target device, and when the vTable reaches a predetermined threshold, writing the value into the target device value log file through a pipeline Flush strategy.

[0016] Furthermore, the pipeline Flush strategy includes:

[0017] The system performs time limit management based on the average transmission time of a single vTable in a non-blocking situation. After the time limit is exceeded, the transmission right will be transferred to the next device. The transmission priority between devices is determined based on the number of full vTables. The more vTables there are, the higher the transmission priority of the device.

[0018] When the vTable transmission priorities of multiple devices are similar and reach the transmission threshold at the same time, the system estimates the maximum parallelism based on the network bandwidth upper limit, schedules the groups, and verifies and confirms each vTable.

[0019] Furthermore, when querying the data, a master-slave multi-level hot cache strategy is adopted, including:

[0020] The system performs a hash operation on the query key to determine the target storage device;

[0021] The system queries the target device's memory table (MemTable) in turn. If a matching key is found, the corresponding value index is returned. If not found, the system continues to query the device's immutable memory table (Immutable MemTable);

[0022] If the value is still not found in the Immutable MemTable, the host's key cache is queried and the corresponding value index is returned accordingly;

[0023] If the key is still not found in the host's key cache, the target device's key cache is further queried and the corresponding value index is returned accordingly;

[0024] If the key is still not found in the target device's key cache, query the target device's LSM-tree layer by layer starting from Level 0 until the key is found, and return the corresponding value index accordingly, or confirm that the key does not exist;

[0025] According to the obtained value index information, the corresponding data is retrieved in the target device value log and the result is returned to the user.

[0026] Furthermore, the master-slave multi-level hot cache strategy includes two levels of cache, namely, the host side and the device side:

[0027] When writing data, the host maintains a key mapping table, and determines whether the number of times the key is written exceeds 3 times according to the mapping table. If it exceeds, it is considered a hot key, and the key and the corresponding value index information will be directly written into the host cache;

[0028] If it is not a hot key, the data is directly written to the SST file of the target device. If the compaction operation is triggered at this time, the system will write the latest version of the repeated key and its value index into the device cache;

[0029] When a key in the device-side cache appears three or more times, the key is marked as a high-frequency access key and synchronized to the host-side cache. These high-frequency access keys will be placed at the front of the host-side cache.

[0030] Furthermore, it also includes: when multiple threads access the cache at the same time, the system adopts a read-write lock (Read-WriteLock) concurrency control mechanism, so that read operations can be performed in parallel, while write operations are performed exclusively.

[0031] Furthermore, during cache replacement, a combination of the LRU (Least Recently Used) strategy and the LFU (Least Frequently Used) strategy based on access frequency is adopted to ensure that frequently accessed data is retained first when cache space is limited.

[0032] Furthermore, the local garbage collection (GC) operation includes: dividing the value data storage into a cold layer and a hot layer. When writing a value, the system determines whether the key corresponding to the value is unique. If the key is unique, the value data is written into the cold layer vlog (value log). If the key corresponding to the value is not unique or has been updated multiple times, the value data is written into the hot layer vlog (value log);

[0033] When the vlogs in the cold or hot layer reach the preset threshold, the system performs garbage collection on the vlogs in the hot layer first.

[0034] Furthermore, the performing of the GC operation by the background thread includes:

[0035] The system continuously monitors the overall load through background threads, including CPU utilization, memory usage, and I / O operation frequency, which are publicly expressed as:

[0036]

[0037] Among them, GC_Trigger is the threshold for starting garbage collection. When this threshold is reached or exceeded, the system will start garbage collection; N represents the length of the monitoring time window, that is, the average load of the system in the past N time units; CPU_Utilizationi is the CPU utilization in the i-th time unit; Memory_Usage i is the memory usage in the i-th time unit; IO_Frequency i is the frequency of I / O operations in the i-th time unit. α, β, and γ are weighting coefficients used to adjust the impact of each load indicator on the garbage collection trigger.

[0038] Beneficial effects of the present invention:

[0039] 1. The multi-NDP system key-value separation of the present invention introduces the key-value separation technology in the multi-NDP system for the first time. By separating the key and value processing, when executing compaction, each NDP device only needs to process the key merging operation, which significantly reduces the execution time of compaction and reduces the impact of compaction on user write operations, thereby greatly improving the write performance.

[0040] 2. The multi-NDP system of the present invention adopts a pipeline Flush strategy, which schedules the value data of multiple NDP devices through pipelines so that these data do not have to compete for limited bandwidth at the same time, thereby minimizing the Flush waiting time of the value data.

[0041] 3. The multi-NDP system of the present invention adopts a master-slave multi-level hot cache strategy: a strategy is proposed to store more hot keys through two-level caches on the host side and the device side. This strategy achieves effective collaboration between the host side and the device side, and synchronizes the hot data on the device side to the host side in a timely manner, thereby maximizing the fast reading performance of the hot data and improving the response speed and overall efficiency of the system.

[0042] 4. The multi-NDP system of the present invention adopts a dirty data garbage collection (GC) strategy. When processing the dirty data of the value log files inside each NDP device, a layered processing strategy is adopted. The value data written for the first time is stored in the cold layer, and the value data written repeatedly subsequently is stored in the hot layer. The system gives priority to garbage collection of the hot layer to minimize the number of garbage collections of the cold layer with high overhead and long time. In addition, the garbage collection process of the value log file is executed by the background thread and does not block the read and write operations of the foreground user. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the process of writing data in a multi-NDP system of the present invention;

[0044] Figure 2 The present invention is a master-slave multi-level hot cache strategy for key processing;

[0045] Figure 3 It is a schematic diagram of the process of multi-NDP system data query of the present invention;

[0046] Figure 4 The present invention is a flow chart of the pipeline Flush strategy for value processing;

[0047] Figure 5 The figure is a flow chart of the background thread executing GC operation according to the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0049] Multi-NDP systems represented by MStore have write amplification problems under mixed read and write loads; in addition, in large-scale data environments (for example, 100GB of data), the improvement of the MStore system in write performance optimization far exceeds the optimization of read performance, which leads to serious asymmetry between read and write performance, limiting the overall performance of the system under read-intensive loads.

[0050] In view of the above problems, the present invention provides a multi-NDP system read and write acceleration method based on key-value separation technology, which introduces key-value separation technology into the multi-NDP system, reduces the execution time of compaction, improves the read and write performance of the multi-NDP system, and reduces the write amplification effect. The method includes:

[0051] When data is written, the key-value pairs written to the multi-NDP system are split, where the key is hashed to determine the target device, and the value corresponding to the key is appended to the value log file of the target device.

[0052] like Figure 1 As shown, key processing includes: writing the key to the memory table (MemTable) of the corresponding target device. When the MemTable reaches a predetermined threshold, the system converts it into an immutable memory table (Immutable MemTable), and then triggers a key Flush operation to write the data to the target device disk and generate an SST file. In order to optimize storage space and system performance, the system performs a key Compaction operation in the background, updates the dirty key index table, and merges and organizes the SST files, thereby reducing storage fragmentation and improving data reading efficiency.

[0053] At the same time, a master-slave multi-level hot cache strategy can be adopted for the key storage of multi-NDP systems. The master-slave multi-level hot cache strategy adopts two-level cache modules on the host side and the device side to optimize data reading efficiency and improve the overall I / O performance of the system.

[0054] like Figure 2 As shown in the figure, when the user writes data, the system first queries the mapping table to determine whether the written data is a hot key. If it is a hot key, the data will be directly written to the host cache. The host cache mainly stores frequently written keys and their corresponding value location pointers to reduce data access latency.

[0055] If it is not a hot key, the data is written directly to the SST file of the target device. If the compaction operation is triggered at this time, the system will write the latest version of the key and its value position index that appear repeatedly during the compaction iteration process into the device-side cache to reduce the disk I / O operation when reading subsequent data.

[0056] When a key in the device cache appears three or more times, the key is marked as a high-frequency access key and synchronized to the host cache. In the host cache, these high-frequency access keys will be placed at the front of the cache to ensure that they can be quickly hit in future read operations. In this way, the system can dynamically adjust the cache strategy, improve the hit rate of data access, and further optimize the read performance.

[0057] When there is concurrent access, the system adopts a concurrency control mechanism based on read-write locks (Read-Write Lock), so that when multiple threads access the cache at the same time, read operations can be performed in parallel, while write operations will exclusively occupy the lock to avoid data inconsistency.

[0058] In terms of cache replacement strategy, the system adopts a multi-level elimination mechanism, which combines the LRU (Least Recently Used) strategy and the LFU (Least Frequently Used) strategy based on access frequency to ensure that frequently accessed data is retained first when cache space is limited.

[0059] This design can not only effectively handle concurrent access by multiple users under high load, but also dynamically adjust cache content when access patterns change, thereby maintaining efficient data access performance.

[0060] like Figure 1 As shown, the value processing includes: when the key is determined by the hash operation to determine the target device, its corresponding value will be written into the value table (vTable) in the value buffer (vBuffer) of the device. Each vBuffer manages multiple vTables. If a vTable reaches the transmission threshold, when the vTable reaches the predetermined threshold, it is written into the value log file through the pipeline Flush strategy.

[0061] like Figure 4 As shown, the process of the pipeline flush strategy includes: in the case where the vTable of only a single device is fully loaded, the value data can be flushed directly to the cold layer vlog file of the target device through the cross-layer data transmission protocol of the multi-NDP system.

[0062] To prevent a single device from taking up too much transmission time, the system performs time-limited management based on the average transmission time of a single vTable without blocking. After the time slice is exceeded, the transmission right will be transferred to the next device.

[0063] The transfer priority between devices is determined based on the number of full vTables. The more vTables there are, the higher the transfer priority of the device. For example, if device A has three full vTables in its vBuffer and device B has only one, device A will be given priority for data transfer. This design is intended to free up more space for the device's vBuffer to avoid blocking data writes to the device.

[0064] If the vTables of multiple devices are fully loaded at the same time and exceed the maximum transmission bandwidth that can be sent at a time. When the transmission priorities of the vTables of multiple devices are similar and reach the transmission threshold at the same time, the system will schedule and group based on the maximum parallelism estimated according to the network bandwidth upper limit to avoid bandwidth contention.

[0065] For example, if the size of a single vTable is 64MB and the network bandwidth is 1000Mb / s, the maximum parallelism is about two vTables. In this case, the system transfers data as a group of two vTables, making full use of the network bandwidth and avoiding transmission delays caused by bandwidth contention.

[0066] At the same time, while the vTable packets to be sent are waiting for transmission, the vBuffer can still accept data from the user. These data will generate a new vTable and be placed in the vBuffer to ensure that the system can still maintain efficient data writing operations during data transmission without affecting the overall performance.

[0067] During the data transmission process, in order to ensure the consistency and integrity of the data, the system also verifies and confirms each vTable. Before transmission, the system will generate a verification code for each vTable, and perform a verification comparison after the transmission is completed to ensure that the data has not been tampered with or damaged during the transmission process. Once the verification is passed, the target device will feedback the transmission result, and the system will update the transmission status and record the transmission log based on the feedback information. These logs record in detail the time, device, data volume, verification results and other information of each transmission, which is not only used for subsequent data tracking and problem troubleshooting, but also as a basis for system optimization.

[0068] As dirty data accumulates in log files, the system will periodically perform garbage collection (GC) operations to clean up invalid data to free up storage space and maintain efficient system operation. During the GC operation, the system will query the dirty data index table and process the dirty data in batches to improve operation efficiency.

[0069] like Figure 5 As shown in the figure, when processing the dirty data of the internal value log files of each NDP device, the system adopts a refined hierarchical processing strategy to improve the overall storage efficiency and performance of the system.

[0070] The value log file is a key component in the NDP device for storing value data in key-value pairs. During system operation, due to the frequent update and writing of value data, a large amount of dirty data may be generated in the value log file. Effective management and recycling of these dirty data is the key to maintaining system performance.

[0071] The value data storage in the present invention is divided into a cold layer and a hot layer to achieve targeted processing of data of different frequencies.

[0072] like Figure 5 As shown, when a value data is written, the system first determines whether the key corresponding to the value is unique. If the key is unique, the value data is written to the cold layer vlog (value log).

[0073] The cold layer vlog is mainly used to store data that is written for the first time and has a low update frequency, thereby reducing data migration and garbage collection operations and reducing the overall system overhead.

[0074] If the key corresponding to the value is not unique or has been updated multiple times, its value data is written to the hot layer vlog (value log).

[0075] The hot layer vlog is designed for data that is updated frequently. The system needs to perform hot layer garbage collection (GC) more frequently to manage this frequently updated data and ensure that the hot layer does not affect performance due to the accumulation of dirty data.

[0076] Specifically, when the vlogs of the cold layer or the hot layer reach the preset threshold, the system will monitor the current system status to decide whether to trigger garbage collection. If the conditions are met, the system will prioritize garbage collection operations on the hot layer vlogs. Since the data in the hot layer vlogs is updated frequently, failure to recycle in time may lead to a decline in system performance. Therefore, prioritizing dirty data in the hot layer vlogs is an important strategy for system optimization.

[0077] In contrast, the garbage collection operation of the cold layer vlog will be postponed until necessary to reduce system resource consumption and latency due to the low data update frequency and high GC overhead.

[0078] In the specific execution process of garbage collection, the present invention designs a processing mechanism based on background threads to optimize the concurrent processing capability of the system under a high-load environment.

[0079] The core advantage of this mechanism is that it can automatically perform garbage collection of value log files without blocking foreground user read and write operations. Specifically, the system continuously monitors the overall load through background threads, including CPU utilization, memory usage, and I / O operation frequency, as shown in Formula 1:

[0080]

[0081] Among them, GC_Trigger is the threshold for starting garbage collection. When this threshold is reached or exceeded, the system will start garbage collection; N represents the length of the monitoring time window, that is, the average load of the system in the past N time units; CPU_Utilizationi is the CPU utilization in the i-th time unit; Memory_Usage i is the memory usage in the i-th time unit; IO_Frequency i is the frequency of I / O operations in the i-th time unit. α, β, and γ are weighting coefficients used to adjust the impact of each load indicator on the garbage collection trigger.

[0082] Through this hierarchical storage and background processing mechanism, the present invention successfully optimizes dirty data management while maintaining efficient system operation, thereby improving the performance and efficiency of overall data storage.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-NDP system read and write acceleration method based on key-value separation technology, characterized in that: Including a multi-NDP system consisting of a host and a device, when: When writing data, the key-value pairs of data written into the batch processing area of ​​the multi-NDP system are split, the key is hashed to determine the target device, and then key processing and value processing are performed. After key processing, the target device stores the key, and after value processing, the corresponding value is appended to the value log file of the target device; During value processing, when the system cannot transmit all data at once, the multi-NDP system value storage adopts a pipelined flush strategy; When querying data, the query key is first hashed, and then a master-slave multi-level hot cache strategy is adopted to search the MemTable, Immutable MemTable, host cache, device cache, and LSM-tree of the target device in sequence. Once the location information of the target value is obtained, the index information is returned and the corresponding value is queried based on the index. When the percentage of dirty data in the value log file exceeds the threshold and garbage collection is triggered, local garbage collection operations are performed through background threads.

2. According to claim 1, the multi-NDP system read and write acceleration method based on key-value separation technology is characterized in that: When the data is written: Key processing includes: writing the key to the MemTable of the corresponding target device. When the MemTable reaches the predetermined threshold, the system converts it into an Immutable MemTable, then triggers the key Flush operation, writes the data to the target device disk and generates an SST file; when the Compaction operation is triggered, the system performs the key Compaction operation in the background, updates the dirty key index table, and merges and organizes the SST files; The value processing includes: writing the value into the vTable in the value buffer vBuffer of the target device, and when the vTable reaches a predetermined threshold, writing the value into the target device value log file through a pipeline Flush strategy.

3. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 2 is characterized in that: The pipeline Flush strategy includes: The system manages the transmission time limit based on the average transmission time of a single vTable in a non-blocking situation. After the time limit is exceeded, the transmission right will be transferred to the next device. The transmission priority between devices is determined based on the number of full vTables. The more vTables there are, the higher the transmission priority of the device. When the vTable transmission priorities of multiple devices are similar and reach the transmission threshold at the same time, the system estimates the maximum parallelism based on the network bandwidth upper limit, schedules the groups, and verifies and confirms each vTable.

4. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 1 is characterized in that: When querying the data, a master-slave multi-level hot cache strategy is adopted, including: The system performs a hash operation on the query key to determine the target storage device; The system queries the MemTable of the target device in turn. If a matching key is found, the corresponding value index is returned. If not found, the Immutable MemTable of the device is queried. If the value is still not found in the Immutable MemTable, the host cache is queried and the corresponding value index is returned accordingly; If the value is still not found in the host cache, the target device cache is further queried and the corresponding value index is returned accordingly; If the key is still not found in the target device cache, query the target device's LSM-tree layer by layer starting from Level 0 until the key is found, and return the corresponding value index accordingly, or confirm that the key does not exist; According to the obtained value index information, the corresponding data is retrieved in the target device value log and the result is returned to the user.

5. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 4 is characterized in that: The master-slave multi-level hot cache strategy: When writing data, the host maintains a key mapping table, and determines whether the number of times the key is written exceeds 3 times according to the mapping table. If it exceeds, it is considered a hot key, and the key and the corresponding value index information will be directly written into the host cache; If it is not a hot key, the data is directly written to the SST file of the target device. If the compaction operation is triggered at this time, the system will write the latest version of the repeated key and its value index into the device cache; When a key in the device-side cache appears three or more times, the key is marked as a high-frequency access key and synchronized to the host-side cache. These high-frequency access keys will be placed at the front of the host-side cache.

6. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 5 is characterized in that: Also includes: When multiple threads access the cache simultaneously, the system uses a Read-Write Lock concurrency control mechanism, where read operations can be performed in parallel, while write operations are performed exclusively.

7. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 6 is characterized in that: When replacing the cache, a combination of the LRU strategy and the LFU strategy based on access frequency is adopted to ensure that frequently accessed data is retained first when cache space is limited.

8. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 1 is characterized in that: The local garbage collection operation includes: The value data storage is divided into cold layer and hot layer. When writing a value, the system determines whether the key corresponding to the value is unique. If the key is unique, the value data is written to the cold layer vlog. If the key corresponding to the value is not unique or has been updated multiple times, the value data is written to the hot layer vlog. When the vlogs in the cold or hot layer reach the preset threshold, the system performs garbage collection on the vlogs in the hot layer first.

9. The multi-NDP system read-write acceleration method based on key-value separation technology according to claim 1, characterized in that: The garbage collection operation is performed by a background thread, including: The system continuously monitors the overall load through background threads, including CPU utilization, memory usage, and I / O operation frequency, which are publicly expressed as: Among them, GC_Trigger is the threshold for starting garbage collection. When this threshold is reached or exceeded, the system will start garbage collection; N represents the length of the monitoring time window, that is, the average load of the system in the past N time units; CPU_Utilizationi is the CPU utilization in the i-th time unit; Memory_Usage i is the memory usage in the i-th time unit; IO_Frequency i is the I / O operation frequency in the i-th time unit; α, β, γ are weighting coefficients used to adjust the impact of each load indicator on garbage collection triggering.

Images