Storage and calculation integration method and system combined with ZNS SSD
By introducing a computation description and execution engine inside or near the ZNS SSD, the problem of the lack of computing power of the ZNS SSD is solved, enabling on-site data processing, reducing data movement and system latency, and improving the efficiency of the storage system.
Patent Information
- Application Number
- CN202510761833.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-09
- Publication Date
- 2025-11-07
AI Technical Summary
Traditional ZNS SSDs, as passive data storage media, lack native computing power, leading to architectural bottlenecks when large-scale data processing demands are met. This results in high data migration costs, limited I/O bandwidth, increased system power consumption, and increased overall processing latency.
By introducing a computation description and execution engine inside or near the ZNS SSD, and by attaching computation task description metadata at the Zone level and utilizing a lightweight computation task engine, data can be processed locally, reducing data movement and improving processing efficiency.
This reduces the bandwidth consumption for data transfer between the host and the SSD, lowers the overall system latency, and improves the overall bandwidth utilization and computing efficiency of the storage system.
Smart Images
Figure CN120909496A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of storage systems and data processing technology, in particular to a storage and computing integration method and system combined with ZNS SSD. BACKGROUND
[0002] With the rapid growth of data volume, the traditional data processing architecture is facing severe challenges. A large amount of data needs to be frequently transmitted from the storage device to the host processor during processing to perform logical computing operations, resulting in high data moving cost, limited I / O bandwidth, increased system energy consumption, and significantly increased overall processing delay.
[0003] Zoned Namespace (ZNS) as the next generation of high-performance SSD interface standard effectively improves the performance and lifespan of storage devices through zone management and sequential write mechanism. However, the current mainstream ZNS SSD is still mainly used as a passive data storage medium, lacking native computing power and near-data processing support, and showing obvious architecture bottlenecks when facing large-scale data processing demands.
[0004] In the existing system, all data processing tasks almost depend on the host CPU for execution, forming a typical "compute far from data architecture mode. This architecture seriously restricts system efficiency and scalability in data-intensive scenarios such as log analysis, machine learning preprocessing, real-time monitoring, etc. For example, when using a ZNS SSD to store a large amount of sensor data, the traditional ZNS SSD analyzes the average value of the past hour, and the CPU reads all data pages from multiple zones into memory, traverses and counts. While the storage and computing integration method (based on ZNS): Zone writes data aggregation as an intermediate structure; or Zone reset runs "in-place aggregation tasks" in the storage controller to compress data into summaries. When the user reads it directly, it is data, which avoids data movement. Moreover, for ZNS SSD, its sequential write and limited state are very suitable for storage integration, because zone has a life cycle (write->full->reset) that can actively process before reset. At the same time, zone sequential write has a fixed write pointer, and data is naturally aggregated by time, making it easy to aggregate and accurately know the starting point of processing.
[0005] Therefore, it is essential to propose a method and system architecture that can perform specific data processing tasks inside or near the ZNS SSD to achieve "Near-Data Processing (NDP)," thereby effectively alleviating data transport bottlenecks, reducing energy consumption, and accelerating data processing. Summary of the Invention
[0006] The present invention provides a storage and computing integration method and system that combines ZNS SSDs. By introducing a computing description and execution engine at the Zone level, data can be processed locally, thereby reducing data movement and improving processing efficiency.
[0007] According to a method for integrating storage and computing using ZNS SSDs according to the present invention, the method includes the following steps:
[0008] I. Data writing and metadata attachment;
[0009] (1.1) The host transfers data to the SSD's Zone using standard or extended write commands, and the data begins to be written to the storage system;
[0010] (1.2) The SSD controller checks the Zone's attributes to confirm whether the Zone allows the attachment of computation task description metadata;
[0011] (1.3) If Zone allows, the host appends the computation task description metadata to Zone by writing command parameters or a separate command, and then proceeds to the next step; if Zone does not allow, proceed directly to the next step.
[0012] (1.4) Data and metadata are written to the corresponding Zone: The data and additional metadata are written to the Zone storage together, so that the data is not only stored, but also comes with processing instructions;
[0013] II. Triggering of computational tasks;
[0014] (2.1) Set up the SSD controller to check the trigger conditions;
[0015] (2.2) Determine if the triggering condition is met: If the triggering condition is met, proceed to the next step; if not, continue to wait for the next triggering condition.
[0016] (2.3) SSD controller calls lightweight computing task engine: When the triggering conditions are met, the controller starts the built-in lightweight computing task engine;
[0017] III. Execution of computational tasks;
[0018] (3.1) Lightweight computing task engine parses computing task description metadata: The engine reads the metadata in the Zone and executes the operation explicitly;
[0019] (3.2) According to the analysis result, a corresponding operation is performed: according to the meta information, a predefined computing kernel is called, a limited SRAM and a DRAM cache are used as a computing cache pool, and Zone data is processed;
[0020] (3.3) Computing result generation: a computing result is generated after processing;
[0021] Four, result output;
[0022] (4.1) Result output mode selection:
[0023] Write to Result Zone: write the result to the reserved Result Zone, which is suitable for subsequent batch processing or storage;
[0024] Return to the host through the command: use the DMA mode, and trigger the result to return to the host buffer through the interrupt, or the host directly acquires through the read command, which is suitable for the scene of processing the result in time;
[0025] (4.2) Operation end: no matter which output mode is completed, the whole computing task flow ends.
[0026] As a preferred, in step (1.2), the computing task description meta information includes SUM, AVG, and FILTER operation instructions.
[0027] As a preferred, in step (1.3), the Zone mapping table ZoneID→TaskType in the controller is controlled by relying on the task binding logic of the Zone Mapper, records the predefined processing logic bound to each Zone, is configured through the NVMe Vendor Command, and supports embedded script flexible extension of the task type.
[0028] As a preferred, in step (2.1), the SSD controller checks at three times:
[0029] Zone is full: triggered when the data written in the Zone reaches a preset threshold.
[0030] Active command: triggered actively by the host sending a special NVMe command.
[0031] Timing trigger: triggered by the internal timer or resource manager at a period.
[0032] As a preferred, in step (3.2), calling the predefined computing kernel includes: the aggregator processes SUM or AVG, and the filter processes FILTER.
[0033] As a preferred, in step (3.2), processing the Zone data includes:
[0034] Aggregates: SUM / AVG / MIN / MAX;
[0035] Filter: FILTER / SCAN;
[0036] Compression: COMPRESS.
[0037] Preferably, when the host writes to the Zone, the controller captures the event through a Hook and triggers the lightweight computing task engine to perform on-site computation.
[0038] As a preferred approach, before the Zone executes a Reset, the controller automatically calls the binding logic to process the data, and the result is returned to the host or stored in the reserved area to ensure data consistency and storage efficiency.
[0039] As a preferred approach, some Zones are mapped to host user-space mmap regions, utilizing SIMD and NUMA to accelerate parallel processing, reduce copy latency, and achieve near-memory computation optimization, making it suitable for high-frequency access tasks.
[0040] The present invention also provides a storage and computing integrated system that combines ZNS SSD, which adopts the above-mentioned storage and computing integrated method that combines ZNS SSD.
[0041] The beneficial effects of this invention are as follows:
[0042] (1) Reduced data transfer bandwidth consumption between the host and SSD: Within the ZNS SSD, each Zone can write data along with a computational task description (such as SUM, FILTER, AVG). This invention executes computational tasks (such as aggregation, filtering, and compression) directly within the Zone, thus returning only the result data, rather than the original data, to the host. When a Zone is full or the host issues an execution command, the computational engine inside the SSD controller executes the task directly on the storage side. Ultimately, only the computational result (such as a sum number) needs to be sent back to the host, greatly reducing the need for original data transmission. This reduces data traffic on the PCIe bus or NVMe protocol layer, as well as host DRAM usage and cache pressure, thereby improving the overall bandwidth utilization of the storage system.
[0043] (2) Reduce overall system latency: Aggregation, filtering and other calculations can be performed while the data is still in the Zone. The results of the calculations can be written to a specific Result Zone or returned directly as a response. The whole process does not require CPU waiting or host scheduler intervention. The controller can process the calculation tasks of multiple Zones in parallel, further shortening the processing path and thus reducing the overall system latency. Attached Figure Description
[0044] Figure 1A flow chart of a storage and computing integration method combined with a ZNS SSD in an embodiment;
[0045] Figure 2 A structural architecture diagram designed for a storage and computing integration method and system combined with a ZNS SSD in an embodiment. DETAILED DESCRIPTION
[0046] For a further understanding of the present application, reference will be made to the following description taken in conjunction with the accompanying drawings. It is to be understood that the embodiments are merely illustrative of the present application and should not be construed as limiting the present application.
[0047] EMBODIMENT
[0048] As shown in the figure, the present embodiment provides a storage and computing integration method combined with a ZNS SSD, which includes the following steps: Figure 1
[0049] I. Data writing and meta information adding
[0050] (1.1) The host transmits data to the Zone of the SSD through a standard or extended write command, and the data starts to be written to the storage system.
[0051] (1.2) The SSD controller checks the attributes of the Zone and confirms whether the Zone allows to add the computing task description meta information; the computing task description meta information includes SUM, AVG, and FILTER operation instructions.
[0052] (1.3) If the Zone allows, the host adds the computing task description meta information to the Zone through a write command parameter or a separate command, and then proceeds to the next step; if the Zone does not allow, it directly proceeds to the next step.
[0053] In step (1.3), relying on the Zone Mapper and task binding logic, the Zone mapping table (ZoneID→TaskType) in the controller records the pre-defined processing logic bound to each Zone, which is configured through the NVMe Vendor Command and supports flexible extension of the task type by embedding scripts (such as WASM / VLIW).
[0054] (1.4) Data and meta information are written to the corresponding Zone: data and the added meta information are written to the Zone storage together, so that the data not only stores but also carries processing instructions.
[0055] II. Computing task triggering
[0056] (2.1) Set up the SSD controller to check the triggering conditions; the SSD controller checks at three times:
[0057] Zone full: Triggered when the amount of data written in Zone reaches the pre-set threshold.
[0058] Active command: Triggered by the host sending a dedicated NVMe command.
[0059] Timing trigger: Triggered by an internal timer or resource manager at a periodic interval.
[0060] (2.2) Determine if the trigger condition is met: If the trigger condition is met, proceed to the next step; if not, continue to wait for the next trigger condition; this is the key branching point of the flow.
[0061] (2.3) SSD controller invokes lightweight computing task engine: When the trigger condition is met, the controller starts the built-in lightweight computing task engine; this engine is the core module for performing calculations.
[0062] III. Computing task execution
[0063] (3.1) Lightweight computing task engine parses computing task description meta-information: The engine reads the meta-information in the Zone to determine the operation (such as parsing the SUM and determining the sum of numerical fields).
[0064] (3.2) Perform corresponding operations based on the parsing results: Based on the meta-information, call pre-defined computing kernels (aggregator handles SUM / AVG, filter handles FILTER), use limited SRAM, DRAM cache as computing cache pool, process Zone data, including: aggregation: SUM / AVG / MIN / MAX; filtering: FILTER / SCAN; compression: COMPRESS, etc.
[0065] (3.3) Computing result generation: Generate computing results after processing; intermediate results during processing can be stored in controller cache or written to Result Zone asynchronously to ensure efficient and stable computing.
[0066] IV. Result output
[0067] (4.1) Result output method selection:
[0068] Write to Result Zone: Write results to a reserved Result Zone (supports many-to-one), suitable for subsequent batch processing or storage;
[0069] Return to host through command: Use DMA to transmit results back to the host buffer through interrupt triggering, or the host directly accesses it through read commands, suitable for scenarios that require immediate processing of results.
[0070] (4.2) Operation completion: Regardless of the output method, the entire computing task flow is complete.
[0071] As Figure 2 shown, it is a structure architecture diagram designed for a storage and computing integrated method and system of ZNS SSD, including Host and ZNS SSD Controller two components;
[0072] 1) Host
[0073] Mmap Zone A: Memory Mapping Area A, connected with SIMD accelerator, SIMD (Single Instruction Multiple Data) accelerator for parallel processing data, improving computing efficiency.
[0074] Mmap Zone B: Memory Mapping Area B, associated with aggregation logic, used for integrated processing of data. Through NUMA optimization, memory access efficiency can be improved.
[0075] PCIe (Peripheral Component Interconnect Express): used for data transmission between Host and ZNS SSD Controller (Zone Namespace Solid State Disk Controller).
[0076] 2) ZNS SSD Controller
[0077] Zone Manager: responsible for managing Zone-related tasks, is the core management module of the controller.
[0078] Write Hook Handler: triggers write-related logic, used to monitor and process data write operations, triggers task execution engine for in-place computing.
[0079] Reset Hook Handler: responsible for Reset computing processing related tasks, plays a role in device reset and other operations.
[0080] WASM / VLIW Executor: used to execute WASM (WebAssembly) or VLIW (Very Long Instruction Word) related instructions, process specific computing tasks.
[0081] NAND Flash Zones: physical medium area for storing data, managed by ZNS SSD Controller.
[0082] The specific implementation process is as follows:
[0083] A) Data preparation and transmission (Host side)
[0084] On Host, data is pre-processed in Mmap Zone A and Mmap Zone B respectively. Mmap Zone A is processed by SIMD accelerator for parallel computation, and Mmap Zone B is integrated by aggregation logic.
[0085] B) Data transmission to Controller The pre-processed data is transmitted to ZNS SSD Controller through PCIe bus.
[0086] C) Controller processing (ZNS SSD Controller side)
[0087] Zone Manager receives tasks from Host, and assigns and manages tasks for different Zones.
[0088] When there is a write operation, Write Hook Handler triggers the write logic to correctly write data to the corresponding NAND Flash Zones.
[0089] If there is an operation such as device reset, Reset Hook Handler is started, and Reset computation is performed by WASM / VLIW executor to ensure that the device returns to the appropriate state.
[0090] Host side (Host Side) implementation steps
[0091] I. Zone mapping to memory area (mmap Zone A / B): Use mmap() system call to map Zone area on ZNS SSD to host user space memory, such as mmap Zone A and mmap Zone B. After Zone A is mapped, it is used for SIMD accelerator; after Zone B is mapped, it is used for aggregation logic processing, such as aggregation calculation or model inference.
[0092] II. Compute offload strategy configuration: Dispatch tasks (such as data processing on a certain Zone) to the underlying SSD through user mode program or device driver; combine heat information or computing demand to determine which Zones are processed by the SSD controller and which are processed by the host SIMD.
[0093] ZNS SSD Controller implementation steps
[0094] a. Zone Manager (Zone Manager): Receives "Zone+task" instructions from the host; schedules write, reset, and computation tasks for Zones; maintains meta information (status, pointer, task marker, etc.) for each Zone.
[0095] b. Zone write / reset scheduling process:
[0096] Write trigger processing component: Monitor write requests; identify characteristics such as write amplification, high heat, and write; dynamically adjust write strategy or schedule to computing engine.
[0097] Reset timing processing component: According to Zone usage, select: delay reset; early pre-reset; phased reset; avoid unnecessary reset performance jitter.
[0098] c. WASM / VLM programming engine (optional): Provide light computing logic capabilities (such as WASM runtime or VLM) within the SSD; used to implement data filtering, compression, aggregation, etc.
[0099] PCIe communication channel
[0100] PCIe communication between host and SSD controller: Zone mmap mapping data flow; task scheduling, Zone state change control signaling; can use doorbell, DMA or shared memory.
[0101] Supplementary mechanism
[0102] Write Hook and real-time processing: When the host writes to the Zone, the controller captures the event through the Hook and triggers the light computing task engine to calculate on site (such as real-time aggregation or screening), reducing subsequent processing pressure.
[0103] Reset Hook and task cleanup: Before Zone performs Reset, the controller automatically calls the binding logic to process data (such as final aggregation, garbage collection), and the results are returned to the host or stored in the reserved area, ensuring data consistency and storage efficiency.
[0104] ZNS mmap + user space processing optimization: Some Zones are mapped to host user space mmap areas, using SIMD, NUMA to accelerate parallel processing, reduce copy delay, achieve near storage optimization, suitable for high-frequency access tasks.
[0105] The embodiment provides a storage and computing integrated system combined with a ZNS SSD, which adopts the above-mentioned storage and computing integrated method combined with a ZNS SSD.
[0106] In this embodiment, each Zone allows to attach a piece of metadata block called "compute task description meta information" in addition to storing data body. This information can be attached by standard or extended write command to instruct the SSD controller to perform certain kind of compute operation on the data in this Zone at certain time. The SSD controller has a built-in lightweight compute task engine, which is a lightweight compute execution engine integrated in the SSD controller firmware, capable of parsing and operating on Zone data block, and triggering corresponding compute logic according to task description. The engine has the following capabilities: (1) identify and parse task description information; (2) call predefined compute kernel; (3) use limited SRAM / DRAM cache as compute cache pool; can be triggered to execute at the following three times: Zone full (Zone Full) automatic execution, host triggers execution through dedicated command (active execution) internally, timer or resource manager triggers (periodic execution). Result Zone / host return: the output result of the compute task can be obtained or used by the host in the following ways. Result Zone output mode: the controller automatically writes the result into a specially reserved Zone (one-to-many). Host return mode: the result is transmitted back to the host buffer through DMA after the task is completed. Inline mode (optional): the host can directly read the task result area through the read command.
[0107] The above description of the present application and its embodiments is illustrative and not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it, without departing from the spirit of the present application, similar structural modes and embodiments can be designed without creativity, which should belong to the protection scope of the present application.
Claims
1. A method for integrating storage and computing with ZNS SSDs, characterized in that: Comprising the following steps: I. Data writing and meta information adding; (1.1) The host transmits data to the Zone of the SSD through a standard or extended write command, and the data starts to be written into the storage system; (1.2) The SSD controller checks the attributes of the Zone and confirms whether the Zone allows the addition of the meta information of the computing task description; (1.3) If the Zone allows, the host adds the meta information of the computing task description to the Zone through a write command parameter or a separate command, and then proceeds to the next step; if the Zone does not allow, it directly proceeds to the next step; (1.4) Data and meta information are written into the corresponding Zone: data and added meta information are written into the Zone storage, so that the data not only stores but also carries processing instructions; II. Computing task triggering; (2.1) Establishing the SSD controller to check the triggering conditions; (2.2) Judging whether the triggering conditions are met: if the triggering conditions are met, proceeding to the next step; if not, continuing to wait for the next triggering condition; (2.3) The SSD controller calls the lightweight computing task engine: when the triggering conditions are met, the controller starts the built-in lightweight computing task engine; III. Computing task execution; (3.1) The lightweight computing task engine parses the meta information of the computing task description: the engine reads the meta information in the Zone to clearly execute the operation; (3.2) Performing corresponding operations according to the parsing results: according to the meta information, calling the pre-defined computing kernel, using limited SRAM and DRAM cache as a computing cache pool, and processing the data in the Zone; (3.3) Computing result generation: generating the computing result after processing; IV. Result output; (4.1) Result output mode selection: Writing into the Result Zone: writing the result into the reserved Result Zone, which is suitable for subsequent batch processing or storage; Returning to the host through a command: using the DMA method, the result is transmitted back to the host buffer through an interrupt trigger, or the host directly acquires it through a read command, which is suitable for scenarios that require timely processing of the result; (4.2) Operation completion: regardless of which output mode is completed, the entire computing task flow is completed.
2. The storage and computing integration method combined with the ZNS SSD according to claim 1, wherein: In step (1.2), the meta information of the computing task description includes SUM, AVG, and FILTER operation instructions.
3. The method of claim 2, wherein: In step (1.3), relying on the Zone Mapper and task binding logic, the Zone mapping table ZoneID→TaskType in the controller records the pre-defined processing logic bound to each Zone, which is configured through the NVMe Vendor Command, and supports embedded script flexible extension of the task type.
4. The storage and computing integration method combined with the ZNS SSD according to claim 3, characterized in that: In step (2.1), the SSD controller checks at three times: Zone full: triggering when the data written into the Zone reaches a preset threshold. Active command: the host sends a special NVMe command to actively trigger. Timing trigger: the internal timer or resource manager triggers the check at a period.
5. The storage and computing integration method combined with the ZNS SSD according to claim 4, characterized in that: In step (3.2), calling the pre-defined computing kernel includes: the aggregator processes SUM or AVG, and the filter processes FILTER.
6. The storage and computing integration method combined with the ZNS SSD according to claim 5, wherein: In step (3.2), processing the data in the Zone includes: Aggregate: SUM / AVG / MIN / MAX; Filter: FILTER / SCAN; Compress: COMPRESS.
7. The storage and computing integration method combined with the ZNS SSD according to claim 6, wherein: When the host writes to the Zone, the controller captures the event through the Hook, triggers the lightweight computing task engine to calculate in place.
8. The storage and computing integration method combined with the ZNS SSD according to claim 7, wherein: Before the Zone performs the Reset, the controller automatically calls the binding logic to process the data, and the result is returned to the host or stored in the reserved area, ensuring data consistency and storage efficiency.
9. The storage and computing integration method combined with the ZNS SSD according to claim 8, wherein: Some Zones are mapped to the host user space mmap area, using SIMD and NUMA to accelerate parallel processing, reduce copy latency, optimize near-memory computing, and are suitable for high-frequency access tasks.
10. A storage and compute integrated system incorporating a ZNS SSD, characterized in that: The method for integrating storage and computing by combining ZNS SSDs according to any one of claims 1-9 is adopted.