Dynamic queue binding method and system based on NVMe over Fabrics (NVMeoF)

Abstract

The invention discloses a dynamic queue binding method and a dynamic queue binding system based on NVMe over Fabrics (NVMeoF). The dynamic queue binding method and the dynamic queue binding system are applied to a protocol processing unit on a storage device side. According to the method, a unique identifier Unique ID is allocated for each I / O command, and a mapping table of the Unique ID and a physical NVMe queue is maintained in an RAM; when an I / O command is received, if the mapping is missing, dynamically selecting a target queue according to the occupation condition of the physical queue and establishing the mapping to realize on-demand binding of the logic context and the physical queue; the system comprises a protocol conversion module, a unique identifier processing module, an RAM mapping table module, a queue scheduling module and a physical NVMe controller. The method supports multiple initiators to share limited physical queues, reduces SSD resource overhead, improves load balance and system expansibility, is compatible with an existing NVMe drive, and can be deployed in an FPGA or an intelligent network card; according to the invention, the complete decoupling of the NVMeoF queue and the physical NVMe queue is realized, and meanwhile, the protocol transparency of the Initiator is kept.

Description

Technical Field

[0001] This invention relates to the field of computer storage system technology, and in particular to a dynamic queue binding method and system based on NVMe over Fabrics (NVMeoF). Background Technology

[0002] The invention, entitled "A Hardware Offloading Device and Method for NVMeoF," with application number CN202510329574.1 and publication number CN119854364A, discloses an offloading method based on NVMeoF, which improves access performance while reducing access latency and offloads the CPU computing power of the storage board. However, when using standard NVMe-oF as middleware, it is limited by the standard protocol and cannot achieve autonomous management of each queue pair (including RDMA and NVMe queue pairs), nor can it flexibly allocate queues according to actual application conditions. Therefore, this solution is not suitable for environments with variable and complex data.

[0003] The invention, with application number CN202410160584.2 and publication number CN118101777A, entitled "An NVMe-over-RDMA Protocol Transmission Method and Electronic Device," discloses a network storage implementation method based on NVMe-oF. This method utilizes a pre-established RDMA shared queue to send modular Fabrics connection commands containing unique subsystem identifiers to the target end. However, CN118101777A's use of a pre-established RDMA shared queue does not conform to the NVMe-oF standard and cannot interface with mainstream nvmecli or libnvme libraries.

[0004] The invention, entitled "NVMeoF Response Command Transmission Method Based on RDMA Completion Event," with application number CN202411171854.6 and publication number CN118672953A, discloses a method that tightly couples data transmission and NVMeoF response command transmission to improve system request processing efficiency. However, when using this tightly coupled method, resources used for data transmission cannot be released when there are no control commands to transmit or when there is temporarily no data transmission, leading to high system resource consumption and potential memory management problems.

[0005] With the widespread deployment of NVMe SSDs in data centers, high-performance I / O subsystems based on the NVMe protocol are gradually becoming mainstream. NVMe-over-Fabrics (NVMeoF) further extends NVMe access from local PCIe to Ethernet (such as RoCEv2, TCP, etc.) environments, enabling remote high-performance storage access.

[0006] In the existing NVMeoF architecture, the initiator establishes queue pairs (QPs) via RDMA, and each NVMeoF queue (SQ / CQ) is typically bound one-to-one with the physical queues of the underlying NVMe device. However, this binding method has the following problems: Queue resources are being wasted in large quantities. In scenarios with multiple initiators and multiple connections, a large number of NVMeoF queues will be created, but SSDs can only provide a limited number of high-performance physical queues, resulting in low resource utilization.

[0007] Unable to dynamically schedule queue load Currently, the NVMeoF queue is statically bound to the NVMe queue, which cannot achieve automatic mapping based on SSD load at runtime, resulting in insufficient I / O concurrency.

[0008] Command contexts strictly depend on CID / SQID and do not have cross-layer mapping capabilities. NVMeoF and NVMe share a traditional CID, which prevents the protocol conversion layer from virtualizing queue structures and sharing NVMe physical queue resources.

[0009] Lack of efficient queue virtualization mechanism In existing technologies, the number of NVMeoF queues is strongly correlated with the number of NVMe queues, making it difficult to implement an architecture where multiple hosts share an SSD resource pool. Summary of the Invention

[0010] To overcome the problems existing in the prior art, the present invention aims to provide a dynamic queue binding method and system based on NVMeover Fabrics (NVMeoF). At the protocol conversion layer, a unique identifier (Unique ID) is assigned to each NVMeoF (NVMeover Fabrics) queue item submission, and this identifier is bound to the original queue context (SQD) and stored in a RAM mapping table. Based on the physical NVMe controller load status, a physical queue is dynamically selected, and commands are issued carrying the Unique ID. After I / O is completed, the context is restored by looking up the table using the Unique ID, and a standard completed queue item is constructed and returned to the Initiator. This invention achieves complete decoupling between the NVMeoF queue and the physical NVMe queue, supporting a large number of NVMeoF queues sharing a limited physical queue in scenarios with multiple Initiators and multiple SSDs, significantly improving SSD resource utilization and system scalability, while maintaining protocol transparency to the Initiator.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A dynamic queue binding method based on NVMe over Fabrics (NVMeoF), applied to the protocol processing unit on the storage device side, includes the following steps: Step 1: Receive I / O commands sent by at least one host initiator via the NVMeoF protocol, wherein the I / O commands carry a unique identifier (Unique ID). Step 2: Based on the Unique ID described in Step 1, search for the corresponding physical NVMe queue information in the mapping table maintained in RAM; Step 3: If there is no corresponding record in the mapping table described in Step 2, then based on the current occupancy status of the physical NVMe queues pre-created by the physical NVMe controller, a target physical NVMe queue is dynamically selected from multiple available physical NVMe queues, and a mapping relationship between the Unique ID and the target physical NVMe queue is established in the mapping table. Step 4: Encapsulate the I / O command described in Step 1 into a command conforming to the local NVMe protocol format, and submit it to the target physical NVMe queue described in Step 3 for execution; Step 5: After receiving the completion response of the I / O command, return the completion status to the corresponding host initiator via the NVMeoF protocol.

[0012] The physical NVMe queue is a queue pair consisting of a submission queue (SQ) and a corresponding completion queue (CQ).

[0013] The Unique ID is generated by the host initiator and sent with each I / O command to uniquely identify the logical context to which the I / O command belongs.

[0014] The dynamic selection of the target physical NVMe queue based on the occupancy status of the pre-created physical NVMe queues of the physical NVMe controller includes adopting any of the following scheduling strategies: minimum load priority, round-robin allocation, or allocation strategy based on historical access popularity.

[0015] When the I / O command is encapsulated in the local NVMe protocol format, it is encapsulated using a Submission Queue Descriptor (SQD). The SQD contains a command opcode, a data pointer, a command identifier (CID), and a target namespace identifier (NSID).

[0016] The command identifier (CID) is virtualized across different host initiators, allowing multiple host initiators to reuse the same CID value without conflict.

[0017] A dynamic queue binding system based on NVMe over Fabrics (NVMeoF) includes: The protocol conversion module is used to receive I / O commands sent by the host initiator via the NVMeoF protocol and convert them into the local NVMe protocol format, which is used in steps 1 and 3 respectively; The unique identifier processing module is used to extract the Unique ID from the I / O command in step 2; The RAM mapping table module, stored in RAM, is used in step 3 to maintain the mapping relationship between Unique IDs and physical NVMe queues; The queue scheduling module is used in steps 3 and 4 to dynamically allocate a target physical NVMe queue based on the occupancy status of each physical NVMe queue in the physical NVMe controller when the mapping relationship is missing, and to update the RAM mapping table. A physical NVMe controller, used in step 4, which includes multiple pre-created physical NVMe queues, to execute encapsulated I / O commands and return a completion response; The protocol conversion module is also used in step 5 to return the completion response to the corresponding host initiator via the NVMeoF protocol.

[0018] The system is deployed in an FPGA or a SmartNIC.

[0019] The physical NVMe queue is a queue pair consisting of a submission queue (SQ) and a corresponding completion queue (CQ).

[0020] The queue scheduling module selects queues using strategies such as minimum load priority, round-robin allocation, or allocation based on historical access popularity.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Significantly reduces physical queue resource overhead on the SSD side This invention eliminates the need to pre-allocate physical resources for each queue of each initiator, significantly reducing the number of physical queues; it does not require each logical queue to correspond to a physical queue, but rather multiple logical queues can reuse the same physical queue, so 100 initiators only need 8 physical queues to run efficiently (context is distinguished by unique ID).

[0022] It alleviates memory pressure on SSD controllers and reduces hardware costs, making it particularly suitable for large-scale cloud storage scenarios.

[0023] 2. Improve physical queue resource utilization and load balancing capabilities. The queue scheduling module can dynamically select the lightest physical queue based on real-time occupancy (such as queue depth and number of pending commands). It employs a minimum load priority strategy, reducing the queue load standard deviation by over 60%.

[0024] This avoids some queues being overloaded while others are idle, thus improving the overall stability of IOPS and throughput.

[0025] 3. Enhance system scalability and multi-tenancy support capabilities. Theoretically, it can support an unlimited number of initiators or logical queues. As long as the physical queue pool is not saturated, new contexts can be dynamically bound; it supports a large number of hosts accessing the same SSD concurrently. Figure 5 This demonstrates a multi-initiator + single SSD deployment scenario. It is suitable for high-density access scenarios such as hyperconverged architecture and serverless storage.

[0026] 4. Decouple the logic and physical layers to improve scheduling flexibility. The mapping between Unique IDs and physical queues can be dynamically updated, migrated, and even expired and reclaimed (e.g., binding can be released if there is no access for a long time). The mapping table can age out or be actively cleaned up; queue migration is supported to cope with failures or hotspots. This lays the foundation for subsequent implementation of advanced functions such as QoS control, fault isolation, and hot migration.

[0027] 5. Maintain protocol compatibility; no modification to the host-side driver is required. The host still sends commands (including CID) according to the standard NVMeoF protocol. The unique ID can be encapsulated in existing fields (such as Keyed SGL Data Block) or extended fields; CID avoids conflicts through virtualization.

[0028] The machine does not need to be aware of the underlying queue binding mechanism; it is compatible with existing Linux NVMe drivers. Deployment costs are low, it is easy to implement, and it has good ecosystem compatibility.

[0029] 6. Suitable for hardware-accelerated deployment, reducing latency. The entire mapping lookup and scheduling logic can be deployed on an FPGA or SmartNIC, leveraging hardware parallelism and low-latency RAM access. Figure 1 The architecture diagram clearly indicates the location of the FPGA.

[0030] Compared to pure software solutions (such as SPDK), it further reduces CPU overhead and processing latency.

[0031] In summary, this invention, by introducing Unique ID, a dynamic RAM mapping table, and an intelligent scheduling strategy, achieves dynamic, on-demand, and shared binding between NVMeoF logical queues and physical queues without altering the host-side protocol behavior. Its beneficial effects are primarily reflected in: resource savings (reducing the number of physical queues), performance improvements (load balancing and high utilization), enhanced scalability (supporting massive numbers of initiators), and deployment friendliness (compatible with the existing ecosystem and supporting hardware acceleration). Attached Figure Description

[0032] Figure 1 This is a diagram showing the relationship between the system functional modules and the workflow of the present invention.

[0033] Figure 2 This is a typical working principle diagram of the present invention.

[0034] Figure 3 This is a schematic diagram of the first typical deployment of the present invention.

[0035] Figure 4 This is a schematic diagram of the second typical deployment of the present invention.

[0036] Figure 5 This is a schematic diagram of the third typical deployment of the present invention.

[0037] Figure 6 The general NVMeoF chain establishment process. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the following is combined with... Figures 1 to 5 The present invention will be further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0039] During system initialization, the NVMe controller pre-creates multiple physical execution queues and attaches them to the protocol translation layer. The protocol translation layer initializes the RAM mapping table and starts the Unique ID generator.

[0040] like Figure 1 As shown, the dynamic queue binding system of the present invention is deployed on the storage device side, preferably implemented in an FPGA or a smart NIC. This system is located between the host initiator and the backend SSD. It receives I / O commands from the host via an NVMe over Fabrics (NVMeoF) network, converts them into local NVMe protocol commands through the MAIN layer, submits them to the physical NVMe controller for execution, and returns completion via an interrupt signal.

[0041] In typical deployment scenarios, a single initiator can be connected to a single SSD (e.g., Figure 2 As shown), it can also support multiple initiators accessing the same SSD concurrently (e.g., Figure 3 (As shown). The latter is the core application scenario of this invention—when multiple hosts simultaneously initiate I / O requests to the same SSD, traditional solutions require pre-allocating an independent physical queue for each queue of each host, while this invention, through a dynamic binding mechanism, allows multiple logical contexts to share limited physical queue resources.

[0042] like Figure 4 As shown, the dynamic queue binding method of the present invention includes the following five steps: (1) Receive I / O commands sent by the host via NVMeoF, which carry a unique identifier (Unique ID); (2) Search for the corresponding physical NVMe queue in the RAM mapping table based on the Unique ID; (3) If no match is found, an idle physical queue is dynamically selected based on the current occupancy status of the physical queue (such as queue depth and number of pending commands), and a mapping is established. (4) Encapsulate the command in local NVMe format (using SQD, including fields such as CID and NSID) and submit it to the target physical queue; (5) After CQ returns the completion status, the response is returned to the corresponding initiator through the NVMeoF protocol.

[0043] The physical NVMe queue is a queue pair consisting of a submission queue (SQ) and a corresponding completion queue (CQ).

[0044] The Unique ID is generated by the host initiator and sent with each I / O command to uniquely identify the logical context to which the I / O command belongs.

[0045] The dynamic selection of the target physical NVMe queue based on the occupancy status of the pre-created physical NVMe queues of the physical NVMe controller includes adopting any of the following scheduling strategies: minimum load priority, round-robin allocation, or allocation strategy based on historical access popularity.

[0046] When the I / O command is encapsulated in the local NVMe protocol format, it is encapsulated using a Submission Queue Descriptor (SQD). The SQD contains a command opcode, a data pointer, a command identifier (CID), and a target namespace identifier (NSID).

[0047] The command identifier (CID) is virtualized across different host initiators, allowing multiple host initiators to reuse the same CID value without conflict.

[0048] like Figure 5 As shown, the system of the present invention includes the following functional modules: Protocol conversion module: Responsible for bidirectional conversion between NVMeoF and the local NVMe protocol; Unique Identifier Processing Module: Parses and extracts the Unique ID from I / O commands; RAM mapping table module: Stored in high-speed RAM, it maintains the mapping relationship between Unique IDs and physical queues; Queue scheduling module: Executes a dynamic allocation strategy when a mapping is missing; Physical NVMe controller: Contains pre-created SQ / CQ pairs and performs actual I / O operations.

[0049] The system is deployed in an FPGA or a SmartNIC.

[0050] The physical NVMe queue is a queue pair consisting of a submission queue (SQ) and a corresponding completion queue (CQ).

[0051] The queue scheduling module selects queues using strategies such as minimum load priority, round-robin allocation, or allocation based on historical access popularity.

[0052] Figure 6 This demonstrates the internal processing flow of I / O commands, from receiving and parsing the Unique ID, looking up the table, scheduling, to submitting.

[0053] Example Reference Figure 1 When the initiator sends SQE via RDMA, the system executes steps ① to ⑤: Receive SQE and parse the command type, SQID, and CID.

[0054] Assign a unique ID through the unique ID Gen and create a mapping table entry (Unique-ID<->SQD).

[0055] Select the NVMe queue based on the arbitration strategy (such as Round-Robin, load-aware algorithm, etc.).

[0056] Submit the command to the NVMe controller for execution.

[0057] After completing the task, the NVMe controller returns a Unique ID and issues an interrupt.

[0058] The protocol conversion layer looks up a table to restore the NVMeoF context.

[0059] Construct an NVMeoF CQE and return it to the initiator via RDMA.

[0060] In the above process, the Unique-ID serves as the core association key, enabling the virtualization of the NVMeoF queue, thereby supporting dynamic binding, cross-layer mapping, and efficient load scheduling.

[0061] Physical test To evaluate the performance of the dynamic queue binding method proposed in this invention in a real system, physical testing was conducted on a real hardware platform, rather than through simulation based on a storage model. This is because the internal parallel structure, FTL mapping, and scheduling strategy of NVMe SSDs are difficult to accurately model through simulation; physical testing can more realistically reflect system performance.

[0062] The test platform includes an NVMeoF Target device, using a Xilinx VCU128 development board. This board integrates an NVMeoF protocol conversion layer, an RDMA protocol stack, and a RAID 0 storage scheduling module, and connects to two PCIe 3.0 NVMe SSDs via a PCIe interface. In the test scenario, a single NVMeoF Initiator sends I / O requests to the Target via the RDMA network.

[0063] During testing, the NVMeoF protocol conversion layer enabled the dynamic queue binding mechanism proposed in this invention, dynamically mapping NVMeoF I / O requests from the Initiator to multiple NVMe physical execution queues, and distributing the I / O requests to two NVMe SSDs for parallel execution through a RAID0 striping strategy. The host computer test used fio, numjob, and iodepth as variables, while the number and depth of queues when creating the NVMe-oF connection were 8 and 32, respectively. The actual number and depth of queues created for each NVMe SSD were 8 and 32, respectively.

[0064] Performance testing employed the standard I / O testing tool fio to continuously load the NVMeoF storage system, testing system throughput and IOPS under both sequential and random read / write scenarios. The performance changes after enabling the dynamic queue binding mechanism were recorded during the testing process.

[0065] With a single initiator and two NVMe SSDs, the dynamic queue binding method proposed in this invention can effectively improve the parallelism of the NVMe SSD array, maintaining high utilization of multiple NVMe physical queues. Compared with the traditional fixed queue binding method, the overall throughput and IOPS of the system are significantly improved in high-concurrency I / O scenarios, and there is no situation where a single NVMe queue becomes a performance bottleneck.

[0066] To verify the effectiveness of this invention, physical performance tests were conducted. The test environment used the fio tool to simulate concurrent loads from multiple initiators, comparing the performance of the traditional static binding scheme with the dynamic binding scheme of this invention. The test results are shown in Table 1, containing the following key data: project Thread Queue depth DAS single disk NVMe-oF dual-disk RAID 0 4k random 1 1 18.3 KIOPS 20 KIOPS 4k random 32 16 896 KIOPS 1545 KIOPS 1m seq 1 1 2992 MB / s 4124 MB / s 1m seq 1 8 3759 MB / s 7114 MB / s Further analysis shows that by decoupling the NVMeoF logical queue from the NVMe physical queue through the Unique ID mapping mechanism, I / O requests can be dynamically allocated to different physical execution queues according to the real-time load, thereby fully leveraging the horizontal scaling capability of the multi-SSD RAID0 architecture and verifying the effectiveness and feasibility of the invention in a real-world deployment environment.

[0067] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dynamic queue binding method based on NVMe over Fabrics (NVMeoF), applied to the protocol processing unit on the storage device side, characterized in that, Includes the following steps: Step 1: Receive I / O commands sent by at least one host initiator via the NVMeoF protocol, wherein the I / O commands carry a unique identifier (Unique ID). Step 2: Based on the Unique ID described in Step 1, search for the corresponding physical NVMe queue information in the mapping table maintained in RAM; Step 3: If there is no corresponding record in the mapping table described in Step 2, then based on the current occupancy status of the physical NVMe queues pre-created by the physical NVMe controller, a target physical NVMe queue is dynamically selected from multiple available physical NVMe queues, and a mapping relationship between the Unique ID and the target physical NVMe queue is established in the mapping table. Step 4: Encapsulate the I / O command described in Step 1 into a command conforming to the local NVMe protocol format, and submit it to the target physical NVMe queue described in Step 3 for execution; Step 5: After receiving the completion response of the I / O command, return the completion status to the corresponding host initiator via the NVMeoF protocol.

2. The dynamic queue binding method according to claim 1, characterized in that, The physical NVMe queue is a queue pair consisting of a submission queue (SQ) and a corresponding completion queue (CQ).

3. The dynamic queue binding method according to claim 1, characterized in that, The Unique ID is generated by the host initiator and sent with each I / O command to uniquely identify the logical context to which the I / O command belongs.

4. The dynamic queue binding method according to claim 1, characterized in that, The dynamic selection of the target physical NVMe queue based on the occupancy status of the pre-created physical NVMe queues of the physical NVMe controller includes adopting any of the following scheduling strategies: minimum load priority, round-robin allocation, or allocation strategy based on historical access popularity.

5. The dynamic queue binding method according to claim 1, characterized in that, When the I / O command is encapsulated in the local NVMe protocol format, it is encapsulated using a Submission Queue Descriptor (SQD). The SQD contains a command opcode, a data pointer, a command identifier (CID), and a target namespace identifier (NSID).

6. The dynamic queue binding method according to claim 5, characterized in that, The command identifier (CID) is virtualized across different host initiators, allowing multiple host initiators to reuse the same CID value without conflict.

7. A dynamic queue binding system based on NVMe over Fabrics (NVMeoF), used to execute the dynamic queue binding method according to any one of claims 1 to 6, characterized in that, include: The protocol conversion module is used to receive I / O commands sent by the host initiator via the NVMeoF protocol and convert them into the local NVMe protocol format, which is used in steps 1 and 3 respectively; The unique identifier processing module is used to extract the Unique ID from the I / O command in step 2; The RAM mapping table module is stored in RAM and is used in step 3 to maintain the mapping relationship between the Unique ID and the physical NVMe queue. The queue scheduling module is used in steps 3 and 4 to dynamically allocate a target physical NVMe queue based on the occupancy status of each physical NVMe queue in the physical NVMe controller when the mapping relationship is missing, and to update the RAM mapping table. A physical NVMe controller, used in step 4, which includes multiple pre-created physical NVMe queues, to execute encapsulated I / O commands and return a completion response; The protocol conversion module is also used in step 5 to return the completion response to the corresponding host initiator via the NVMeoF protocol.

8. The dynamic queue binding system according to claim 7, characterized in that, The system is deployed in an FPGA or a SmartNIC.

9. The dynamic queue binding system according to claim 7, characterized in that, The physical NVMe queue is a queue pair consisting of a submission queue (SQ) and a corresponding completion queue (CQ).

10. The dynamic queue binding system according to claim 7, characterized in that, The queue scheduling module selects queues using strategies such as minimum load priority, round-robin allocation, or allocation based on historical access popularity.

Images