A multi-ndp ssd device key-value separation collaborative storage system combined with partitioned namespace

Abstract

The application discloses a kind of multi-NDP SSD equipment key value separation collaborative storage systems of combining partition namespace, host end responds user read-write request, using differentiated write path processing write request according to Value size, form SST file and Vlog file;Dynamic selection NDP SSD equipment, complete data disk landing;Multi-NDP SSD equipment end, hot Vlog file or cold Vlog file is executed differential recycling mechanism, and combined with partition namespace is classified and stored dynamically managed Zone.The system of the application adopts host end and multi-NDP SSD architecture, on the basis of reserving LSM-tree high-efficiency key index capability, value data is written into each NDP SSD equipment different Zone, and Zone is managed, and allocation and recycling operation of Zone are completed.Effectively reduce the write amplification and data moving overhead caused by traditional Compaction and Vlog GC, alleviate the limitation of traditional SSD internal FTL on system optimization effect, improve the throughput and stability of key value storage system in high concurrency, write-intensive scenario.

Description

Technical Field

[0001] This invention relates to the field of key-value data storage technology, and in particular to a multi-NDPSSD device key-value separation collaborative storage system that combines partitioned namespaces. Background Technology

[0002] With the rapid development of cloud storage, distributed databases, and metadata management applications, key-value storage systems are facing continuously increasing write pressure and more diverse data objects. LSM-trees (Log-Structured Merge-Trees), due to their ability to transform random writes into sequential writes, possess high write performance and good scalability, and have been widely used in key-value storage systems such as LevelDB and RocksDB. However, LSM-trees rely on background compaction operations to maintain a multi-level ordered structure to reduce space redundancy and improve query efficiency. During this process, data is repeatedly rewritten between different levels, leading to significant write amplification. Under continuously updated, write-intensive loads, frequent compaction not only consumes a large amount of I / O bandwidth and computing resources but also reduces overall system throughput and increases tail latency.

[0003] To alleviate the aforementioned problems, a key-value separation method was proposed. This method stores the metadata information of keys and values ​​in an SSTable, while storing larger values ​​in a separate Value Log (Vlog) via append-only writes. This allows the compaction process to handle only smaller key and index information, thus avoiding repeated migration of large values ​​during hierarchical merging and significantly reducing write amplification. However, key-value separation does not eliminate background maintenance overhead; instead, it shifts the system performance bottleneck from SSTable compaction to Vlog garbage collection (GC). As data is continuously updated, a large number of invalid key-value pairs accumulate in the Vlog. GC needs to identify, migrate, and reclaim space for valid data, which introduces additional I / O overhead and new write amplification, thereby impacting overall system performance.

[0004] Existing key-value separation optimization schemes for CPU-centric architectures, such as HashKV, DiffKV, and FenceKV, while reducing the cost of Vlog GC to some extent, still rely heavily on the host side for GC. This results in frequent transfers of large amounts of value data between the host and storage devices, and the host side also bears a heavy burden of data filtering, validity judgment, and migration management. Therefore, in large-scale data scenarios, the host side still faces significant bandwidth and computational bottlenecks. To reduce the data migration and computational burden on the host side, Near-Data Processing (NDP) architecture has been applied to the architecture of traditional host and storage devices. Solid-state drives (SSDs) based on near-data computing architecture refer to SSDs that add computing units within traditional SSDs, enabling them to have computing capabilities in addition to storage. We call SSDs with near-data computing architecture near-data computing drives (NDP-based SSDs, referred to as NDP SSDs in this patent). Introducing NDP SSDs into key-value separation systems allows background maintenance tasks such as garbage collection (GC) to be executed as close as possible to the flash storage medium within the NDP SSD, thereby reducing data transfer overhead between the traditional SSD and the host and improving the processing efficiency of the key-value storage system. However, a single NDP SSD device has inherent limitations in terms of computing power, storage capacity, and parallel processing capabilities, making it difficult to meet the system requirements of high-throughput and high-concurrency scenarios. Therefore, it is necessary to further introduce a multi-NDP SSD architecture into the key-value separation system, where multiple NDP SSD devices collaboratively undertake value log storage and garbage collection tasks, thereby improving the system's parallel processing capabilities and overall scalability.

[0005] Besides the design of the storage engine itself, the organization of the underlying storage media also profoundly affects the overall performance of key-value storage systems. Traditional NDP SSDs rely on the Flash Translation Layer (FTL) to handle address mapping, garbage collection, and wear leveling, but these internal FTL operations are usually transparent to the NDP compute module. In a multi-NDP architecture oriented towards key-value separation, if the NDP SSD still uses the traditional FTL management method, the Vlog GC or data compaction operations performed in the NDP SSD device may overlap with the FTL's garbage collection behavior, causing repeated data migration, triggering additional write amplification, and thus causing performance jitter and diminishing the benefits of near-data processing. Zoned Namespace (ZNS) is an important technology in the NVMe specification, designed to fundamentally change the way flash memory is managed, allowing the host and storage device to work together more efficiently. In a zoned namespace, flash memory is divided into several zones, each supporting only sequential writes, and requiring an explicit reset operation when reclaiming space. By delegating more responsibility for data placement, write order control, and space reclamation to the host side, this partition management approach eliminates the uncontrollable data migration inherent in traditional FTL, reduces the overhead of garbage data migration during flash memory reclamation, significantly reduces overall write amplification, and improves flash media utilization efficiency and device lifespan. Therefore, introducing partition namespaces into multi-NDP key-value separation systems is expected to eliminate the interference of NDP SSDs on GC within NDP SSD devices, enabling NDP devices to perform Vlog management and garbage collection tasks more efficiently, thereby further improving system throughput and stability.

[0006] Therefore, existing technologies face challenges such as write amplification issues in LSM-trees, host-side bottlenecks caused by Vlog GC in key-value separation schemes, resource limitations of single NDP SSD devices, and the combined effects of internal FTL and upper-level GC in traditional SSDs. A new storage system is needed that can integrate multi-NDP SSD collaboration, efficient key-value separation, and partition namespace management features. Summary of the Invention

[0007] To address the aforementioned problems, the present invention aims to provide a multi-NDPSSD device key-value separation collaborative storage system that combines partitioned namespaces. This system solves the technical problems of high write amplification in LSM-trees, host-side performance bottlenecks in key-value separation Vlog GC, limited resources of a single NDP SSD device, and the performance degradation caused by the combined effects of internal FTL and upper-layer GC in traditional SSDs. The invention combines a multi-NDP architecture, a key-value separation mechanism, and the zone management features of partitioned namespaces. While retaining the efficient key indexing capabilities of LSM-trees, it writes data to different zones on each NDP SSD device and manages the zone space on the NDP device side, completing zone allocation and reclamation operations. The method of this invention effectively reduces the write amplification and data migration overhead caused by traditional compaction and Vlog GC, and alleviates the limitations of FTL in NDPSSDs on system optimization, thereby improving the throughput and stability of the key-value storage system in high-concurrency, write-intensive scenarios.

[0008] The objective of this invention can be achieved through the following technical solution: a multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces, comprising:

[0009] On the host side, in response to user read and write requests, the write requests are processed using differentiated write paths based on the value size. Small value data is generated into a regular SST file, while large value data is determined as hot or cold and generated into an indexed SST file, a hot Vlog file, or a cold Vlog file. The target NDP SSD device for the read and write request is dynamically selected, and the regular SST file, indexed SST file, hot Vlog file, or cold Vlog file is written to the corresponding NDP SSD device to complete the data persistence.

[0010] The multi-NDP SSD device side includes multiple NDP SSD devices, which implement a differentiated recycling mechanism for hot or cold Vlog files, and combine partition namespaces to perform dynamic management of zone-based storage.

[0011] As a further aspect of the present invention, the host-side dynamic selection of the target NDP SSD device for read / write requests includes:

[0012] The host queries the global key mapping table to determine whether the target key in the current write request already exists in a historically owned NDP SSD device. If the target key has a corresponding historically owned NDP SSD device, the system prioritizes routing this write request to that NDP SSD device. If the target key is appearing for the first time, or its historically owned device is in an abnormal, unavailable, or overloaded state, the host selects an NDP SSD device from the candidate NDP SSD devices based on the operating status of each NDP SSD device to execute this write request.

[0013] As a further embodiment of the present invention, the host terminal includes:

[0014] The processing module uses differentiated write paths to handle write requests based on the value size. Small values ​​are handled through inline write paths, while large values ​​are handled through key-value separation write paths.

[0015] The scheduling module dynamically selects the target NDP SSD device for read / write requests based on the global key mapping table and device status information.

[0016] The hot / cold Vlog file determination module performs hot / cold Vlog file determination on the value buffer data and writes it to the hot or cold Vlog file.

[0017] The task execution unit performs a flush operation on regular SST files, indexed SST files, hot Vlog files, or cold Vlog files, and writes the data to the target NDP SSD device.

[0018] The multi-NDP SSD device includes a memory area and a flash memory area: the memory area is equipped with a GC task execution unit, a compaction task execution unit, and a Zone space management software stack, and the flash memory area is divided into several Zones.

[0019] As a further aspect of the present invention, the small value is processed via an inline write path, including:

[0020] Small value data is written as complete key-value pairs to SST_MemTable. Data in SST_MemTable that reaches the threshold is transformed into Immutable SST_MemTable and then written to a regular SST file.

[0021] Large values ​​are handled by writing to the path using key-value separation, including:

[0022] In the large value data, key data is written to VLog_MemTable, and value data is written to the value buffer. When the value buffer reaches a threshold, the VLog_MemTable data is transformed into an Immutable VLog_MemTable and the value buffer is frozen. After determining whether the value buffer data is hot or cold, it is written to the hot Vlog file or the cold Vlog file. The Immutable VLog_MemTable is updated with key and position index entries and then written to the indexed SST file.

[0023] As a further aspect of the present invention, the value buffer data is judged as hot or cold and written to a hot Vlog file or a cold Vlog file, including:

[0024] The host side combines value data status information to update the activity level and popularity assessment, and judges whether the activity is hot or cold based on the popularity assessment value. The popularity assessment score is expressed as:

[0025]

[0026] in, Indicates whether the current write to the target key is an overwrite update; Indicates the number of times the target key is updated within a preset time window; This represents the time interval between the target key and the last write; α, β, and γ are weighting parameters, and ε is a smoothing constant.

[0027] As a further aspect of the present invention, the target NDP SSD device implements a differentiated recycling mechanism for hot or cold Vlog files, including:

[0028] The system maintains file-level metadata information for each Vlog file and calculates the failure rate of each Vlog file based on the metadata information.

[0029] The target NDP SSD device adopts a delayed recycling strategy for hot Vlog files. Recycling is only triggered when the failure rate of hot Vlog files reaches a high threshold and the rate of increase of failed data in the hot Vlog files slows down within a preset time.

[0030] The target NDP SSD device employs a priority recycling strategy for cold Vlog files, and recycling is performed when the failure rate reaches a low threshold.

[0031] As a further embodiment of the present invention, the partitioned namespace is divided into SST Zone groups and VlogZone groups, and each Zone group adopts a dynamic expansion strategy, including:

[0032] A portion of Zones is reserved in the partition namespace as a shared buffer. During operation, the partition namespace continuously monitors the space occupancy, write rate, proportion of invalid data, and request pressure of Flush, Compaction, and garbage collection tasks for each Zone group. When the remaining available space of a Zone group falls below a preset threshold, or the resource request frequency of the corresponding background operation continues to increase, the partition namespace dynamically allocates new Zones from the shared buffer and expands them to the target group. When a Zone group is in a state of low utilization for a long period of time, the partition namespace returns the idle Zones to the shared buffer.

[0033] As a further embodiment of the present invention, the Vlog Zone group is divided into a hot Vlog Zone group and a cold Vlog Zone group, which respectively store Vlog files with different hot and cold attributes; the system dynamically adjusts the number of available Zones between the hot Vlog Zone group and the cold Vlog Zone group according to the change in the ratio of hot data to cold data in the Vlog Zone group.

[0034] As a further embodiment of the present invention, the partitioned namespace performs wear-aware zone allocation and reclamation decisions within the SST Zone group and the Vlog Zone group.

[0035] As a further embodiment of the present invention, the wear perception priority of Zone allocation and recycling decisions is expressed as follows:

[0036]

[0037] in, This represents the proportion of invalid data in the i-th zone. This represents the number of times the i-th zone has been erased. This indicates the maximum number of Zone erases in the current partition namespace. This represents the effective amount of data that needs to be migrated when reclaiming the i-th Zone. This represents the idle time of the i-th Zone since its most recent write or update, where α, β, γ, and δ are weight parameters.

[0038] The beneficial effects of this invention are:

[0039] This invention is aimed at key-value separation storage systems for multiple NDP SSD devices. It introduces a sequential write and explicit management mechanism for partition namespaces (Zones) to collaboratively optimize the writing, organization, and recycling processes of value data.

[0040] (1) NDP SSD Device Selection Mechanism. This invention does not map keys to a specific NDP SSD device. Instead, it dynamically selects a more suitable target NDP SSD device for writing by combining the historical ownership of keys with status information such as the current load, buffer usage, number of background tasks, and available space of each device. This design can maintain the locality of the same key on the same device as much as possible, reduce cross-device migration and consistency maintenance overhead, and avoid overloading of a single device due to concentrated writes, thereby improving the load balancing capability and overall parallel processing efficiency in a multi-device environment.

[0041] (2) Parallel Flush and Vlog hot / cold tiered write mechanism based on dual-path memory table. This invention adopts differentiated write paths according to the value size: small values ​​are directly written to SST_MemTable, while large values ​​are written to Vlog_MemTable and value buffer through key-value separation, and indexed SST files and hot / cold Vlog files are formed respectively during subsequent flushes. This design takes into account the need for efficient management of small values ​​and low write amplification of large values, reduces the additional overhead caused by large values ​​participating in compaction, and reduces the mutual interference caused by the mixed storage of hot and cold data through hot / cold tiering, providing support for subsequent recycling optimization.

[0042] (3) Differentiated Vlog File Reclamation Mechanism. To address the issue of different update characteristics between hot and cold Vlog files, this invention employs delayed reclamation and priority reclamation strategies respectively. This prevents hot data from being repeatedly migrated due to premature cleanup, while allowing cold data to complete space reclamation more quickly. During the reclamation process, the system combines the global key mapping table and the latest index entry to determine the validity of records, migrating only still valid data, and can further divide data into hot and cold categories during migration. This mechanism reduces invalid migration overhead, improves value log reclamation efficiency, and enhances the overall space cleanup effect of the system.

[0043] (4) Zone Classification Storage and Dynamic Management Module. This invention divides the Zones in the partition namespace into SST Zones and Vlog Zones according to their purpose, and can further subdivide the Vlog Zones into hot Zones and cold Zones, while reserving shared reserved Zones to support dynamic expansion. The system can dynamically adjust the number of each type of Zone based on the space occupancy rate, write rate, proportion of failed data, and background task pressure at different stages. This design makes Zone resource allocation more flexible and can better adapt to the different data characteristics and changing trends of SST and Vlog, thereby improving space utilization and reducing resource waste caused by fixed partitioning.

[0044] (5) Adaptive Multi-Factor Wear Leveling Zone Allocation and Reclamation Mechanism. This invention incorporates factors such as the number of zone erases, garbage ratio, migration cost, and idle time into the zone allocation and reclamation decision, rather than relying solely on simple idle status or garbage ratio for management. The system prioritizes zones with less wear to undertake new write and data migration tasks, and comprehensively considers the reclamation benefits and the impact on lifespan during reclamation, thereby avoiding frequent erases and writes to a few zones. This design helps achieve global wear leveling, extends the lifespan of the flash memory in the NDP SSD, and balances space reclamation efficiency with system operational stability. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0046] Figure 2 This is a schematic diagram of the host structure of the system of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of the system and the multi-NDP SSD device of the present invention. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0049] Example 1:

[0050] This embodiment discloses a key-value separation collaborative storage system for multiple NDP SSD devices that combines partitioned namespaces, such as... Figure 1 As shown, this includes the host side and the multi-NDP SSD device side.

[0051] The host responds to user read / write requests, using differentiated write paths based on the value size. Small value data is processed into regular SST files, while large value data is processed by determining whether the value is hot or cold, resulting in indexed SST files, hot Vlog files, and cold Vlog files. The target NDP SSD device for the read / write request is dynamically selected, and the regular SST file, indexed SST file, hot Vlog file, or cold Vlog file is written to the corresponding NDP SSD device, completing the data persistence.

[0052] The multi-NDP SSD device side includes multiple NDP SSD devices, which implement a differentiated recycling mechanism for hot or cold Vlog files, and combine partition namespaces to perform dynamic management of zone-based storage.

[0053] Furthermore, such as Figure 2 As shown, the host side includes a processing module, a scheduling module, a hot / cold detection module, and a Flush task execution unit.

[0054] The processing module uses differentiated write paths to handle write requests based on the value size. Small values ​​are handled using inline write paths, while large values ​​are handled using key-value separation write paths.

[0055] The scheduling module dynamically selects the target NDP SSD device for read / write requests based on the global key mapping table and device status information.

[0056] The hot / cold Vlog file determination module determines whether the data in the value buffer is hot or cold and writes it to either the hot or cold Vlog file.

[0057] The Flush task execution unit performs a flush operation on regular SST files, indexed SST files, hot Vlog files, or cold Vlog files, and writes the data to the target NDP SSD device.

[0058] like Figure 3 As shown, the multi-NDP SSD device includes a memory area and a flash memory area: the memory area is equipped with a GC task execution unit, a compaction task execution unit, and a Zone space management software stack, while the flash memory area is divided into several Zones. The GC task execution unit is used to perform garbage collection of data written to disk; the compaction task execution unit is used to perform compaction of data written to disk; and the Zone space management software stack is used to classify, store, and dynamically manage the flash memory Zones.

[0059] Taking a write request as an example: After a user initiates a write request, the write request first reaches the host.

[0060] Upon receiving a write request, the host first determines the size of the value in the request. If the target value is less than a preset threshold, the data is considered suitable for inline storage, and the system directly writes the complete key-value pair into the `SST_MemTable`. This `SST_MemTable` is used to temporarily store complete key-value pairs that have not yet undergone key-value separation. Once it reaches the threshold, it is converted into a read-only `Immutable SST_MemTable` (frozen), no longer accepting new writes. Simultaneously, the system creates a brand new `SST_MemTable` for subsequent writes. A background thread then begins generating a regular `SST` file from the data in the `ImmutableSST_MemTable`.

[0061] If the target value is greater than or equal to the threshold, the key-value pair is considered suitable for key-value separation. In this case, the system writes the key and its corresponding index to the `Vlog_MemTable`, and simultaneously writes the complete value to the value buffer. The `Vlog_MemTable` primarily stores the key and its associated location descriptions, while the value buffer temporarily stores large values ​​that have not yet been written to the Vlog file. When the `Vlog_MemTable` reaches a preset threshold, it is converted to a read-only `ImmutableVlog_MemTable`, no longer accepting new writes. Simultaneously, the system creates a completely new `Vlog_MemTable` for subsequent writes, and the current value buffer becomes immutable.

[0062] The background flush thread handles data in the Immutable SST_MemTable, Immutable Vlog_MemTable, and value buffer. The host selects the target NDP SSD device for the current write request based on the historical ownership of the target key and the current operating status of each NDP SSD device, and routes the write request to the selected NDP SSD device for processing. This method avoids long-term concentration of write requests on a single device, alleviating local device overload issues and improving the overall load balancing capability in a multi-NDP architecture. For data in the value buffer, the host also needs to perform a hot / cold check. Based on the hot / cold check results, the data is sequentially written to either the hot or cold Vlog file. Simultaneously, the data in the corresponding Vlog_MemTable is written to the indexed SST file. When data needs to be written to disk, the system first uses a Zone allocation algorithm to select a suitable Zone for the SST file and Vlog file, and then performs the data writing.

[0063] Therefore, the write process in this invention forms two cooperating data paths under a multi-NDP SSD collaborative architecture: for small value data, the complete key-value pairs are directly written to the SST, and the corresponding NDP SSD device organizes them into an SST file and then sequentially writes them to the corresponding storage area in the flash memory; for large value data, a key-value separation method is adopted, where the target NDP SSD device first writes the value data to a hot or cold layer Vlog file, then writes the key and its corresponding value position index to an indexed SST file, and then sequentially writes them to the corresponding Vlog storage area and SST storage area in the flash memory. Through this design, this invention can achieve distributed processing and localized management of write requests in a multi-NDP SSD environment, and optimize the data writing process by leveraging the sequential write characteristics of ZNS; at the same time, by separating large values ​​from the LSM-tree, the write amplification caused by their participation in compaction and reorganization is reduced, and the hot and cold layered writing method reduces the problem of mixed storage of value data with different update frequencies, thereby improving the write efficiency, space utilization, and subsequent data management performance of the entire key-value separation storage system.

[0064] Take a read request as an example:

[0065] When a user initiates a read request, the request first reaches the host. Upon receiving the read request, the host first searches for the target key in its local memory structure. Specifically, the system prioritizes searching the currently active SST_MemTable and Vlog_MemTable; if not found, it continues searching the immutable memory tables; if still not found, the host determines the target NDP SSD device responsible for the key based on the global key mapping relationship and routes the read request to that device. The target NDP SSD device then searches in the regular SST file or the indexed SST file formed by Vlog_MemTableFlush. Through this hierarchical search process, the system can progressively locate the latest record corresponding to the target key. When the found record belongs to the regular SST file, it indicates that the key corresponds to small value data. Since the complete key-value pair in this path is directly stored in the memory table or SST file, the system can directly read the complete value from the corresponding location and return it to the user without accessing additional value log files. When the found record belongs to the indexed SST file, it indicates that the key corresponds to large value data. At this point, the system needs to further locate the actual value based on the location description information recorded in the index entry. If the index entry still points to a logical location in the value buffer, it means that the value has not yet been flushed to the Vlog file. In this case, the system directly reads the corresponding value from the value buffer and returns it. If the index entry points to a physical location in a Vlog file, the system reads the target value from the corresponding hot or cold Vlog file based on the file identifier, offset address, and length information recorded in the index entry and returns it.

[0066] Because this invention combines partitioned namespaces to organize data, storing regular SST files, indexed SST files, and hot / cold Vlog files in different types of zones in a sequential writing manner, the system not only performs logical index lookups from key to value during read request processing, but also further converts the logical locations into physical read locations in flash memory by combining the file-to-zone mapping relationship. In this way, this invention combines local index lookups under a multi-NDPSSD architecture with zone storage and management within partitioned namespaces, thereby achieving an efficient read path for key-value separation storage systems.

[0067] Example 2:

[0068] This embodiment optimizes the dynamic selection of the target NDP SSD device for read / write requests on the host side, based on embodiment 1, including:

[0069] After a user initiates a write request, the request first reaches the host. Unlike the traditional method of statically mapping keys to a storage node based solely on a fixed hash result, this invention designs an NDP SSD device selection mechanism oriented towards write requests in multi-NDP SSD scenarios. This mechanism selects a more suitable target NDP SSD device for the current write operation based on the current system state, thereby reducing the risk of local device overload, improving the parallel utilization of multiple devices, and providing a more balanced execution environment for subsequent MemTable management, file flushing, and Vlog recycling. Specifically, the host first queries the global key mapping table to determine whether the target key in the current write request already has a historically owned device. If the key already has a corresponding historically owned NDP SSD device, the system prioritizes routing the current write request to that device. This keeps the version chain, index entries, and value log files of the same key as concentrated as possible in the same NDP SSD device, reducing the overhead of remote index updates, consistency maintenance, and value relocation caused by cross-device migration. If this key is appearing for the first time, or if its historically associated device is currently abnormal, unavailable, or overloaded, the host further considers the operating status of each NDP SSD device and selects the target device with the lowest cost from the candidate devices to perform this write operation. The operating status includes at least the following information: the number of foreground read / write requests currently on the NDP SSD device, the length of the pending request queue, the number of currently executing compaction tasks, the number of currently executing Vlog file recycling tasks, and the available zone space remaining on the device. Based on the above status information, the host calculates the comprehensive selection cost for each candidate NDP SSD device, prioritizing devices with lower read / write pressure, fewer background tasks, and more available zone space as the target device for this write request. Therefore, the system not only avoids buffer congestion on some devices due to a large number of write requests concentrated in a short period, but also reduces mutual interference between background tasks and foreground writes on the device side.

[0070] After device selection is completed on the host side, the system routes the write request to the target NDP SSD device and records the mapping relationship between the key and the target device in the global key mapping table. For keys being written for the first time, this mapping relationship is kept stable for a period of time. For keys with existing historical ownership relationships, the system does not easily change their ownership device unless the original device is overloaded or fails for a long period of time. If a modification occurs, the latest historical device needs to be recorded synchronously. Through this design, the present invention achieves both write load balancing and key locality maintenance in a multi-NDP SSD environment.

[0071] Example 3:

[0072] Based on the above embodiments, this embodiment adopts a dual-path parallel Flush and Vlog cold / hot layered write mechanism, including:

[0073] In this invention, upon receiving a write request, the host does not employ a uniform data organization path. Instead, it writes the data to two different types of memory tables based on the value size. Specifically, the system internally sets up two types of memory tables on the host: SST_MemTable and Vlog_MemTable. SST_MemTable stores complete key-value pairs without key-value separation, i.e., the key and its smaller value; Vlog_MemTable stores the key and its index after key-value separation, while the corresponding larger value is written to the device's internal hot and cold value buffer.

[0074] For small value data, when the target value is below a preset threshold, the system directly writes the complete key-value pair to the SST_MemTable. This type of data does not need to be separately entered into the value log path during subsequent flushes; instead, it is generated into a regular SST file along with the SST_MemTable.

[0075] For large value data, when the target value is greater than or equal to a preset threshold, the system uses a key-value separation approach. In this case, the host writes the key and its corresponding value location description information into the Vlog_MemTable, and simultaneously writes the complete key-value pair into the cold and hot value buffer.

[0076] When the amount of data in Vlog_MemTable reaches a preset threshold or the data residence time of either structure exceeds a preset time limit, the system triggers a Flush operation on the Vlog path.

[0077] Once triggered, the current active Vlog_MemTable and the current value buffer are simultaneously frozen into an immutable state. Newly arriving large value write requests will be imported into the new active Vlog_MemTable and the new value buffer to ensure that foreground writes are not blocked during the flush.

[0078] During the flush process, the system first sequentially scans the large value records in the value buffer. Based on the hot or cold attribute of each record, it appends it to the hot or cold Vlog file respectively, and records the position of the record in the new Vlog file. Subsequently, the system organizes the keys in the Vlog_MemTable and the updated index items into an indexed SST file. During the flush process, the host uses the NDP device selection mechanism to write the SST file and Vlog file to the corresponding NDP SSD device, thus completing the data persistence.

[0079] To reduce the mixed storage of large value data with different update frequencies in Vlog files, this invention further designs a Vlog hot / cold judgment mechanism.

[0080] This mechanism is used to determine whether a value should be written to a hot or cold Vlog file during the initial write phase of a large value. Specifically, when processing a large value write request, the host evaluates its update activity based on the key's status information. This status information includes at least: whether the key is being written for the first time, the number of updates within a preset time window, the time interval since the last write, and whether frequent overwrite updates have occurred in recent writes. If a key is being written for the first time, or the number of updates within the preset time window is less than a threshold, or the time interval between two adjacent writes is long, the system considers its update activity low and classifies the value as cold data. If a key has been updated multiple times in a short period, or is an active key that has been frequently overwritten recently, the system considers its update activity high and classifies the value as hot data. This invention can define a heat score for each large value key, which can be expressed as:

[0081]

[0082] Among them, UpdateFlag k Indicates whether the current write to this key is an overwrite update; the value is 0 for the first write and 1 for an overwrite update; RecentUpdateCount k Indicates the number of times the key is updated within a preset time window; Δt k This represents the time interval between the last write operation and the key distance; α, β, and γ are weighting parameters, and ε is a smoothing constant. When HotScore... k When the value exceeds a preset threshold, the system marks it as hot data and writes it to a hot Vlog file during the subsequent flush process; when the HotScore... kWhen the value is below a preset threshold, the system marks it as cold data and writes it to a cold Vlog file during the subsequent flush process.

[0083] Under this mechanism, hot Vlog files are primarily used to store large-value data that is frequently updated and more likely to become invalid quickly, while cold Vlog files are primarily used to store large-value data that is updated less frequently and is relatively stable. This not only reduces the contamination of cold data by hot data but also provides a basis for the differentiated recycling of Vlog files in the future.

[0084] Example 4:

[0085] Based on the above embodiments, this embodiment implements a differentiated recycling mechanism for Vlog files on the target NDP SSD device, specifically:

[0086] To address the issue of different data update characteristics between hot and cold Vlog files, this invention designs a differentiated recycling mechanism for Vlog files.

[0087] Specifically, the system first maintains file-level metadata for each Vlog file. This metadata includes at least the file type, total file size, amount of valid data, amount of invalid data, and file sealing time. Based on this information, the system calculates the invalidation rate of each Vlog file and adds files that meet the criteria to the candidate recycling queue accordingly.

[0088] For hot Vlog files, due to frequent data updates, premature recycling could lead to the migration of valid data becoming invalid again within a short period, increasing the overhead of repeated migrations. Therefore, the system employs a delayed recycling strategy for hot Vlog files, triggering recycling only when the failure rate reaches a high threshold and the rate of increase in invalid data slows down within a preset time. For cold Vlog files, since the data is relatively stable, the probability of rapid invalidation after recycling is low. Therefore, the system employs a priority recycling strategy for cold Vlog files, performing recycling as soon as the failure rate reaches a low threshold. Thus, this invention achieves differentiated processing for hot files (delayed recycling) and cold files (priority recycling).

[0089] During the actual recycling process, the target NDP SSD device sequentially scans all key-value pair records in the Vlog file to be recycled through a background thread, and determines whether each record is still valid. The validity determination is based on the position of the latest value corresponding to the key in the current index structure. In multi-NDP SSD scenarios, to avoid misjudgments caused by residual historical validity information in old devices due to key migration or device reassignment, the system first queries the global key mapping table maintained on the host side when determining the validity of key-value pair records in the candidate Vlog file to determine whether the key still belongs to the target NDP SSD device performing the recycling.

[0090] If not, the record is directly identified as invalid; if it is the current NDP SSD device, then the latest index entry corresponding to the key is searched in the current index structure of the NDP SSD device.

[0091] If the location description information of the current index entry is completely consistent with that of the latest index entry, the record is considered valid. If they are inconsistent, it means that the record has been overwritten by the updated value and should be considered invalid and discarded during recycling. After identifying valid records, the system rewrites these valid key-value pairs into a new Vlog file.

[0092] Furthermore, during the migration process, the system can again determine the key's "hot" or "cold" status based on its recent update history. If the key is still a frequently updated key, it will be migrated to a new "hot" Vlog file; if the key is currently updated less frequently, it will be migrated to a new "cold" Vlog file. Once all valid records have been migrated, the system will uniformly update the value pointers of the relevant keys to point to the new Vlog file locations and delete the old Vlog files.

[0093] Through the above methods, this invention simultaneously achieves effective data filtering, hot and cold data differentiation processing, and data re-layering during the Vlog file recycling process, thereby reducing the overhead of repeated hot data migration, improving the efficiency of cold data recycling, and providing a more regular Vlog file layout for subsequent storage space organization.

[0094] Example 5:

[0095] This embodiment, based on the above embodiments, performs categorized storage and dynamic management of Zones in the partition namespace.

[0096] Organize and manage Zone resources in the partition namespace, providing available Zone resources for operations such as Flush, Compaction, and garbage collection in each NDP SSD device.

[0097] Specifically, the system manages the Zones of each NDP SSD device in groups, specifically dividing them into SST Zones and Vlog Zones. The SST Zone stores the SST files generated after flushing and compaction, while the Vlog Zone stores Vlog data. The Vlog Zone can be further subdivided into hot Zones and cold Zones to store Vlog files with different hot / cold attributes respectively.

[0098] To adapt to the changing space requirements of SST data and value log data under different load stages, this invention further designs a dynamic expansion strategy for each Zone group. Specifically, in each NDP SSD device, in addition to the pre-divided SST Zone group and Vlog Zone group, a portion of schedulable reserved Zones are reserved as a shared buffer. During operation, the system continuously monitors the space occupancy rate, write rate, proportion of failed data, and request pressure of background tasks such as Flush, Compaction, and garbage collection for each group.

[0099] When the remaining available space of a certain type of Zone group is lower than a preset threshold, or when the resource request frequency of the corresponding background operation continues to increase, the system can dynamically allocate a new Zone from the shared buffer and expand it to the target group to improve the writing and organization capabilities of that type of data. Conversely, when a group is in a state of low utilization for a long time, the system can return the idle Zone to the shared buffer after the existing Zone has completed data migration and reclamation, so that other groups can use it as needed.

[0100] For Vlog Zones, the number of available Zones can be dynamically adjusted between hot and cold Zones based on changes in the ratio of hot to cold data. This makes the allocation of hot and cold Zone resources more in line with actual access characteristics, improves overall space utilization, and reduces resource waste caused by fixed partitioning.

[0101] Example 6:

[0102] Based on Example 5, this embodiment adopts an adaptive multi-factor wear leveling allocation and recycling mechanism for Zones in the partitioned namespace.

[0103] This mechanism is built on the aforementioned Zone classification storage and dynamic management module. It is used to perform wear-aware Zone allocation and reclamation decisions within the two types of groups, SST Zone and Vlog Zone, to alleviate the wear unevenness problem caused by traditional static allocation strategies that only rely on lifecycle or simple group matching and ignore Zone erasure status.

[0104] In actual operation, due to the differences in size, update frequency, expiration time and management strategy between each SST file and Vlog file, some zones are easily used and erased frequently, while other zones remain in a low-utilization or idle state for a long time. This eventually causes some zones to reach their life limit prematurely, thereby reducing the overall lifespan of the flash memory.

[0105] To address this, this mechanism introduces the number of erases per Zone as a core wear characteristic indicator and incorporates it into both Zone allocation and Zone reclamation, ensuring that allocation and reclamation prioritize Zones with less wear, thereby achieving global wear balance. Specifically, the system maintains an array of erase counts (Reset_count[z]), a Zone state table (Zone_state[z]), a garbage ratio array (Garbage_ratio[z]), and an empty Zone queue with wear sorting capabilities. Reset_count[z] records the cumulative number of erases for each Zone, initially set to 0; Zone_state[z] records the current state of a Zone, its group, and the currently stored file information; Garbage_ratio[z] records the proportion of invalid data in each Zone; and the elements in the empty Zone queue are organized in the form of (reset_count, zone_id), allowing the system to prioritize empty Zones with the fewest current reset counts. Because in a partitioned namespace, a zone can only be made empty again through explicit erasure after effective data migration is completed, the number of erasures can directly reflect the true wear and tear of the zone and can serve as an important basis for subsequent allocation and reclamation decisions.

[0106] When an upper-layer module requests the allocation of a new Zone, the system first determines the target Zone group based on the file type. If the file is an SST file generated after flushing or compaction, it is allocated within the SST Zone group; if the file is a Vlog file, it is allocated within the Vlog Zone group, with cold Vlog files placed in cold Zones and hot Vlog files placed in hot Zones. During Zone allocation, the system prioritizes the Zone with the fewest resets in the current group as the target Zone. For SST files, the system also considers their LSM tree level, trying to place SST files at the same LSM tree level in the same Zone. If there are insufficient directly usable Zones in the corresponding group, the system selects the empty Zone with the fewest resets from the global empty Zone queue, adds it to the corresponding group, and allocates it to the current write request. In this way, newly created files and migrated data are preferentially written to Zones with less wear, thus avoiding a few local Zones bearing high-frequency write and reset operations for a long time.

[0107] When the number of available zones in a given zone group falls below a preset threshold, or the average garbage ratio of zones within that group exceeds a preset threshold, the system triggers a zone reclamation process. Unlike traditional methods that select reclamation targets solely based on garbage ratio, this invention comprehensively considers the garbage ratio, data migration overhead, wear leveling, and zone activity level in the reclamation decision, thereby improving reclamation efficiency and reducing the impact on normal system read / write operations. Specifically, the system first filters candidate zones from the target zone group and calculates a comprehensive reclamation priority for each candidate zone. This priority is related not only to the proportion of invalid data in the zone but also to the number of erases, the amount of valid data migrated, and the zone's idle time. A higher proportion of invalid data indicates a larger free space after reclamation; fewer erases indicate less wear on the zone, making it more suitable for priority reclamation; a smaller amount of valid data migrated indicates less data needs to be moved during reclamation, reducing additional overhead; and a longer idle time indicates lower current activity in the zone, minimizing the impact of reclamation on front-end operations. Therefore, the system prioritizes zones with a high proportion of invalid data, low erasure frequency, low migration cost, and long idle time as recycling targets. The recycling priority of candidate zones can be calculated as follows:

[0108]

[0109] Among them, Garbage_ratioi Reset_count represents the proportion of invalid data in the i-th zone. i This indicates the number of erases for that Zone, Max_reset indicates the maximum number of erases for the current Zone, and Migration_cost indicates the maximum number of erases for that Zone. i Idle_time represents the effective amount of data that needs to be migrated when reclaiming this Zone. i This indicates the idle time of the Zone since its most recent write or update. α, β, γ, and δ are weighting parameters used to balance space reclamation efficiency, wear leveling effect, data migration overhead, and reclamation timing. Higher priority indicates a Zone is more suitable for current reclamation. Once a candidate Zone is determined, the system iterates through the files or data items within that Zone, identifies still valid data, and re-allocates new Zones for this valid data. The reallocation process also follows the balanced allocation principle mentioned above, prioritizing Zones with less wear within the corresponding group as migration targets. Subsequently, the system appends valid data to the new target Zone according to the Zone's write constraints, and synchronously updates the mapping relationship and Zone status information. After all valid data in the candidate Zones has been migrated, the system performs an explicit erase operation on that Zone, increments its Reset_count by 1, sets it back to an empty state, and adds any extra empty Zones to the global empty Zone queue for subsequent dynamic allocation.

[0110] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces, characterized in that, include: On the host side, in response to user read and write requests, the write requests are processed using differentiated write paths based on the value size. Small value data is generated into a regular SST file, while large value data is determined as hot or cold and generated into an indexed SST file, a hot Vlog file, or a cold Vlog file. The target NDP SSD device for the read and write request is dynamically selected, and the regular SST file, indexed SST file, hot Vlog file, or cold Vlog file is written to the corresponding NDP SSD device to complete the data persistence. The multi-NDP SSD device side includes multiple NDP SSD devices, which implement a differentiated recycling mechanism for hot or cold Vlog files, and combine partition namespaces to perform dynamic management of zone-based storage.

2. The multi-NDP SSD key-value separation collaborative storage system combining partitioned namespaces according to claim 1, characterized in that, The host terminal includes: The processing module uses differentiated write paths to handle write requests based on the value size. Small values ​​are handled through inline write paths, while large values ​​are handled through key-value separation write paths. The scheduling module dynamically selects the target NDP SSD device for read / write requests based on the global key mapping table and device status information. The hot / cold Vlog file determination module performs hot / cold Vlog file determination on the value buffer data and writes it to the hot or cold Vlog file. The task execution unit performs a flush operation on regular SST files, indexed SST files, hot Vlog files, or cold Vlog files, and writes the data to the target NDP SSD device. The multi-NDP SSD device includes a memory region and a flash memory region: The memory area includes a GC task execution unit, a compaction task execution unit, and a Zone space management software stack, while the flash memory area is divided into several Zones.

3. A multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces as described in claim 2, characterized in that, The host-side dynamic selection of target NDP SSD devices for read / write requests includes: The host queries the global key mapping table to determine whether the target key in the current write request already exists in a historically owned NDP SSD device. If the target key has a corresponding historically owned NDP SSD device, the system prioritizes routing this write request to that NDP SSD device. If the target key is appearing for the first time, or its historically owned device is in an abnormal, unavailable, or overloaded state, the host selects an NDP SSD device from the candidate NDP SSD devices based on the operating status of each NDP SSD device to execute this write request.

4. A multi-NDP SSD key-value separation collaborative storage system combining partitioned namespaces according to claim 2, characterized in that: The small value is processed through the inline write path, which includes: writing the small value data into a complete key-value pair and then writing the SST_MemTable to the SST_MemTable that has reached the threshold, converting the SST_MemTable data into an Immutable SST_MemTable, and then writing it into a regular SST file. Large values ​​are handled through a key-value separation write path, including: key data in the large value data is written to VLog_MemTable, and value data is written to the value buffer. When the value buffer reaches a threshold, the VLog_MemTable data is transformed into an Immutable VLog_MemTable and the value buffer is frozen. After determining whether the value buffer data is hot or cold, it is written to the hot Vlog file or the cold Vlog file. The Immutable VLog_MemTable is updated with key and position index entries and then written to the indexed SST file.

5. A multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces according to claim 4, characterized in that, Value buffer data is processed for hot or cold Vlog file determination and written to either hot or cold Vlog file, including: The host side combines value data status information to update the activity level and popularity assessment, and judges whether the activity is hot or cold based on the popularity assessment value. The popularity assessment score is expressed as: in, Indicates whether the current write to the target key is an overwrite update; Indicates the number of times the target key is updated within a preset time window; This represents the time interval between the target key and the last write; α, β, and γ are weighting parameters, and ε is a smoothing constant.

6. A multi-NDP SSD key-value separation collaborative storage system combining partitioned namespaces according to claim 1, characterized in that, The target NDP SSD device implements a differentiated recycling mechanism for hot or cold Vlog files, including: The system maintains file-level metadata information for each Vlog file and calculates the failure rate of each Vlog file based on the metadata information. The target NDP SSD device adopts a delayed recycling strategy for hot Vlog files. Recycling is only triggered when the failure rate of hot Vlog files reaches a high threshold and the rate of increase of failed data in the hot Vlog files slows down within a preset time. The target NDP SSD device employs a priority recycling strategy for cold Vlog files, and recycling is performed when the failure rate reaches a low threshold.

7. A multi-NDP SSD key-value separation collaborative storage system combining partitioned namespaces according to claim 1, characterized in that, The partition namespace is divided into SST Zones and Vlog Zones. Each Zone group adopts a dynamic expansion strategy, including: A portion of Zones is reserved in the partition namespace as a shared buffer. During operation, the partition namespace continuously monitors the space occupancy, write rate, proportion of invalid data, and request pressure of Flush, Compaction, and garbage collection tasks for each Zone group. When the remaining available space of a Zone group falls below a preset threshold, or the resource request frequency of the corresponding background operation continues to increase, the partition namespace dynamically allocates new Zones from the shared buffer and expands them to the target group. When a Zone group is in a state of low utilization for a long period of time, the partition namespace returns the idle Zones to the shared buffer.

8. A multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces according to claim 7, characterized in that, The Vlog Zone group is divided into a hot Vlog Zone group and a cold Vlog Zone group, which store Vlog files with different hot and cold attributes respectively. The system dynamically adjusts the number of available Zones between the hot Vlog Zone group and the cold Vlog Zone group according to the change in the ratio of hot data to cold data in the Vlog Zone group.

9. A multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces according to claim 7, characterized in that, The partitioned namespace performs wear-aware zone allocation and reclamation decisions within the SST Zone group and Vlog Zone group.

10. A multi-NDP SSD device key-value separation collaborative storage system combining partitioned namespaces according to claim 9, characterized in that, The wear perception priority for zone allocation and recycling decisions is represented as follows: in, This represents the proportion of invalid data in the i-th zone. This represents the number of times the i-th zone has been erased. This indicates the maximum number of Zone erases in the current partition namespace. This represents the effective amount of data that needs to be migrated when reclaiming the i-th Zone. This represents the idle time of the i-th Zone since its most recent write or update, where α, β, γ, and δ are weight parameters.

Images