Embedded high-performance nvme-of target system and method
By combining RDMA, NVMe-oF, and NVMe components in an embedded environment, the complex queue mapping and hardware implementation difficulties in embedded NVMe-oF target systems are solved, achieving efficient utilization of storage resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2026-06-23
- Publication Date
- 2026-07-21
AI Technical Summary
In embedded environments, the queue mapping process of existing NVMe-oF target end systems is complex, difficult to offload to dedicated hardware, and fails to effectively support unified mapping of multiple NVMe SSDs to NVMe-oF disks. The hardware implementation is complex and debugging is difficult.
By combining RDMA, NVMe-oF and NVMe components, and using arrays such as qp_attr, NVMe, IO queue, controller and global queue number, a joint mapping between queues and namespaces is achieved, simplifying the hardware offloading process.
While supporting access from multiple initiators and shared access from multiple NVMe SSDs, the mapping process has been greatly simplified, hardware implementation requirements have been reduced, and storage resource utilization has been improved.
Smart Images
Figure CN122431618A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of storage network technology and relates to an embedded high-performance NVMe-oF Target end system and method. Background Technology
[0002] NVMe (Non-Volatile Memory Express) is a high-speed transmission protocol designed specifically for flash memory storage. It connects directly to the system via the PCIe bus and offers lower latency and higher parallelism compared to traditional SATA / AHCI protocols. Storage devices using the NVMe protocol are called NVMe drives. Currently, most NVMe drives on the market are SSDs (Solid State Drives), hence they are also called NVMe SSDs.
[0003] NVMe over Fabrics (NVMe-oF) is a network-based NVMe protocol that allows the host (Initiator) to remotely access NVMe storage devices (Target) over a network. The Target typically uses mature NVMe SSD products with added NVMe-oF protocol support. The NVMe storage device provided by the Target to the Initiator is an integrated abstraction of the NVMe SSD product and may not be identical to the original NVMe SSD; it can be referred to as an NVMe-oF drive.
[0004] The NVMe-oF protocol supports various underlying network transmission methods, among which RoCEv2, based on RDMA (Remote Direct Memory Access), is widely used due to its low latency and high bandwidth. The RDMA Queue Pair (QP) is the basic unit of RDMA communication. Each QP contains a Submit Queue (SQ) and a Receive Queue (RQ) for transmitting data and commands between the Initiator and Target. In embedded environments, RDMA hardware resources are limited, typically supporting only a limited number of QPs (e.g., 32). QP0 is a special, unused QP, QP1 is a special QP used for CM (Connection Manager) connection establishment, and the remaining QPs can be used for NVMe-oF command transmission.
[0005] NVMe SSDs support multiple namespaces. The NVMe-oF Target system supports multiple NVMe SSDs and abstracts them into multiple NVMe-oF disks for the Initiator. Each NVMe-oF disk may share part of the original NVMe SSD's namespace. Namespace mapping methods are complex and varied. The NVMe-oF Target system can abstract the namespaces of multiple NVMe SSDs into a single NVMe-oF disk, or it can allow multiple NVMe-oF disks to share the same namespace of the same NVMe SSD.
[0006] The NVMe-oF I / O queues ultimately need to be mapped to the NVMe SSD I / O queues. Considering different requirements, there are various queue mapping methods. Traditional NVMe-oF target systems (such as Linux and SPDK (StoragePerformance Development Kit)) primarily use software to implement queue mapping. This mapping process is complex, relies on high-performance CPUs, is difficult to offload to dedicated hardware, and is difficult to apply to embedded environments. In embedded environments, software runs on low-performance CPUs or MCUs. To achieve a high-performance NVMe-oF target system, hardware offloading is necessary. Existing hardware offloading solutions do not achieve a unified mapping from multiple NVMe SSDs to NVMe-oF disks and mainly focus on queue sharing and load balancing within NVMe SSDs, resulting in complex hardware implementation, high hardware requirements, and difficult debugging. Summary of the Invention
[0007] To address the problems existing in the above-mentioned traditional methods, this invention proposes an embedded high-performance NVMe-oF Target end system and a processing method for the embedded high-performance NVMe-oF Target end system, which can simplify the mapping process and simplify the hardware offloading implementation.
[0008] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, an embedded high-performance NVMe-oF target end system is provided, including an RDMA component, an NVMe-oF component, and an NVMe component; The RDMA component is used to perform RDMA network transmission. It uses CM to establish a connection and creates an RDMA QP in RC mode. The RDMA QP in RC mode provides a command transmission channel for the NVMe-oF component. The RDMA software provides a qp_attr array with a configurable number of elements. The qp_attr array is used to record the RDMA connections that correspond one-to-one with the NVMe-oF queue. The NVMe component software is used to initialize and configure NVMe SSDs. The NVMe component software provides configurable NVMe arrays and IO queue arrays. The NVMe array is used to record the attribute information of each NVMe SSD and the namespaces in the NVMe SSD. Each element of the IO queue array is an IO queue in an NVMe SSD. The NVMe-oF component's software executes Admin commands. After processing the Connect command for the IO queue, the NVMe-oF component's software transfers the IO queue to the NVMe-oF component's hardware for further processing of subsequent commands. Each NVMe-oF queue occupies one RDMA QP in the RDMA component. The NVMe-oF component's software provides a configurable array of controllers and a global queue number array. Each element in the controller array represents controller data, and each controller handles a connection at the Initiator end. The global queue number array records the unique global queue number used by the IO queues under each controller during connection. The NVMe-oF component's hardware provides a configurable queue command space mapping table, which records the mapping relationship between IO queues, global queue numbers, and namespaces.
[0009] In one embodiment, the elements of the qp_attr array also record the connection status of the RDMA connection, the NVMe-oF queue number, the controller number, and the global queue number.
[0010] In one embodiment, the index of each element in the global queue number array + 1 is the global queue number; the value of each element in the global queue number array includes 0 and 1, where 0 represents that the global queue number is not occupied and 1 represents that the global queue number is occupied.
[0011] In one embodiment, the number of elements in the global queue number array is the smaller of the number of elements in the qp_attr array minus 3 and the maximum number of IO queues for the NVMe SSD.
[0012] In one embodiment, the NVMe-oF component maps the namespaces of each NVMe SSD disk to multiple namespaces of the abstract NVMe disk via a controller.
[0013] In one embodiment, when an RDMA connection is established, the Target finds an unconnected element in the qp_attr array. After receiving the Connect command sent by the Initiator through the RDMA connection, the Target uses the elements of the qp_attr array to record the queue number of the NVMe-oF queue specified by the Connect command. After the IO queue is established, the NVME-OF component allocates a global queue number to each IO queue of each controller from the global queue number array, and at the same time records the gqid attribute of each element in the qp_attr array as the global queue number.
[0014] In one embodiment, the qp_attr array, NVMe array, IO queue array, controller array, and global queue number array are all in DDR memory.
[0015] On the other hand, a processing method for an embedded high-performance NVMe-oF Target end system is also provided. Based on the aforementioned embedded high-performance NVMe-oF Target end system, the processing method includes the following steps: The Target side establishes an RDMA QP number in RC mode through CM connection establishment. The Target side allocates an RDMA QP number and the corresponding array element in the qp_attr array to establish an RDMA connection as the transmission channel for the Admin queue. The Target receives the Connect command from the Initiator from the RDMA connection and records the Admin queue number of the NVMe-oF queue specified by the Connect command into the corresponding array element in the qp_attr array; The Target finds an idle controller from the controller array, updates the corresponding controller number to the corresponding array element in the qp_attr array, and returns the corresponding controller number to the Initiator. The Target receives management commands from the Admin queue and returns information about the abstract NVMe disk provided by the controller.
[0016] In one embodiment, the above-described processing method for an embedded high-performance NVMe-oF target system further includes the following steps: The target end creates an RDMA QP number in RC mode through CM connection establishment. The target end allocates an RDMA QP number j and the corresponding array element qp_attr[j] to establish an RDMA connection as the transmission channel for the IO queue. The Target receives the Connect command from the RDMA connection and records the IO queue number and controller number of the NVMe-oF queue specified in the Connect command into the array element qp_attr[j]. The Target finds an element with a value of 0 in the global queue number array and sets it to 1, then records the global queue number in the array element qp_attr[j]. The target software writes NumS configuration entries to the queue command space mapping table and transfers control of the IO queue to the hardware. Each configuration entry includes rdma_qpn, nsid, and qid_i, where NumS is the number of elements in the NVMe array, rdma_qpn is the RDMA QP number of an IO queue on the target side, nsid is the namespace number that will be carried by subsequent IO commands, and qid_i is the index of the IO queue of the NVMe disk to be accessed in the IO queue array.
[0017] In one embodiment, the above-described processing method for an embedded high-performance NVMe-oF target system further includes the following steps: The target hardware receives the IO command from the initiator and determines the RDMA QP number based on the RDMA QP where the IO command is located. The target hardware parses I / O commands and obtains the namespace number from the I / O commands; The target hardware uses the RDMA QP number and namespace number to look up the queue command space mapping table and find the index of the IO queue array; The target hardware uses the index of the IO queue array to find the IO queue of the NVMe SSD disk that needs to be used; The target hardware sends IO commands to the IO queue of the NVMe SSD and obtains the command execution results; The target hardware sends the execution results of the IO commands to the initiator through the IO queue of the NVMe-oF component.
[0018] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned embedded high-performance NVMe-oF Target system and method improve the hardware and software functions of the embedded high-performance NVMe-oF Target by providing a new joint mapping technology for queues and namespaces of the embedded high-performance NVMe-oF Target using RDMA, NVMe-oF, and NVMe components. Considering the mapping of queues and namespaces, and combining the embedded + high-performance environment, the mapping process is greatly simplified, the hardware is simplified, and the hardware implementation requirements are reduced, while supporting access from multiple initiators and shared access from multiple NVMe SSDs. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the architecture of an embedded high-performance NVMe-oF target system in one embodiment; Figure 2 This is a schematic diagram of namespace mapping in one embodiment; Figure 3 This is a schematic diagram of queue mapping in one embodiment; Figure 4 A flowchart for creating the Admin queue in one embodiment; Figure 5 A flowchart for creating an IO queue in one embodiment; Figure 6 Here is a flowchart of the IO command processing in one embodiment; Figure 7 This is a schematic diagram of a processing flow for an embedded high-performance NVMe-oF target system in one embodiment. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0022] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.
[0023] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0024] In one embodiment, such as Figure 1 As shown, an embedded high-performance NVMe-oF target end system is provided, including an RDMA component, an NVMe-oF component, and an NVMe component. The RDMA component is used to perform RDMA network transmission, and uses CM to establish a connection to create an RC-mode RDMA QP. The RC-mode RDMA QP provides a command transmission channel for the NVMe-oF component. The RDMA software provides a configurable qp_attr array, which records the RDMA connections that correspond one-to-one with the NVMe-oF queue.
[0025] The NVMe component software is used to initialize and configure NVMe SSDs. The NVMe component software provides configurable NVMe arrays and IO queue arrays. The NVMe array is used to record the attribute information of each NVMe SSD and the namespaces in the NVMe SSD. Each element of the IO queue array is an IO queue in an NVMe SSD.
[0026] The NVMe-oF component's software executes Admin commands. After processing the Connect command for the IO queue, the NVMe-oF component's software transfers the IO queue to the NVMe-oF component's hardware for further processing of subsequent commands. Each NVMe-oF queue occupies one RDMA QP in the RDMA component. The NVMe-oF component's software provides a configurable array of controllers and a global queue number array. Each element in the controller array represents controller data, and each controller handles a connection at the Initiator end. The global queue number array records the unique global queue number used by the IO queues under each controller during connection. The NVMe-oF component's hardware provides a configurable queue command space mapping table, which records the mapping relationship between IO queues, global queue numbers, and namespaces.
[0027] In this embodiment, the embedded high-performance NVMe-oF target is configured using an MCU and a bare driver. The underlying network transmission uses RDMA (e.g., RoCEv2) protocol. The RDMA hardware supports a maximum of 32 QPs, where QP0 is a special, unused QP, QP1 is a special QP used for CM connection establishment, and the remaining 30 QPs can be used for NVMe-oF command transmission. The software handles NVMe-oF management commands (i.e., Admin commands) to handle complex management operations, while the hardware handles NVMe-oF IO commands to achieve high-speed NVMe SSD read / write access. The design focus of this embodiment is the joint mapping of queues and namespaces to simplify the mapping process and hardware offloading implementation.
[0028] like Figure 1 As shown, the left side represents two standard initiator terminals, and the NVMe-oF Target terminal can receive connections from the standard initiator terminals. The right side represents the improved embedded high-performance NVMe-oF Target terminal of this embodiment, which includes three main components: an RDMA component, an NVMe-oF component, and an NVMe component.
[0029] The embedded high-performance NVMe-oF Target can accept connections from multiple Initiator terminals (such as Initiator A and Initiator B). The underlying layer requires RDMA for network data transmission, consuming RDMA QP at both ends. Each connection creates an Admin queue and multiple IO queues. Figure 1 On the Initiator side, Admin queues (e.g., rdma qp3) are identified by circles with the number 0, and IO queues (e.g., rdma qp4 and rdma qp5) are identified by circles with other numbers. Each queue requires one RDMA QP. After the connection is established, the Initiator side can see an abstract NVME device from the Target side, containing multiple namespaces (e.g., ...). Figure 1 (NS1, NS2, NS3, and NS4 in the text).
[0030] The RDMA component is used to perform RDMA network transmissions. It uses CM (Command Controller) to establish an RDMA QP in RC (Reliable Connection) mode. The RC-mode RDMA QP provides a command transmission channel (i.e., the NVMe-oF queue in the NVMe-oF component) for the NVMe-oF component. Each NVMe-oF queue (including the IO queue and the Admin queue) occupies one RDMA QP in the RDMA component on the Target side.
[0031] The RDMA software provides a `qp_attr` array containing `NumR` elements, where `NumR` is a configurable value whose size is limited by the RDMA hardware; in embedded environments, it can be 32. The element `qp_attr[k]` represents an RDMA connection with QP number `k` on the target side. One RDMA connection corresponds to one NVMe-oF queue.
[0032] Therefore, the elements of the qp_attr array also record the connection status of the RDMA connection, the NVMe-oF queue number, the controller number, and the global queue number. Specifically, qp_attr[k] can also represent an NVMe-oF queue and record more other component-related information, such as: qp_attr[k].qpn==k is the RDMA QP number; qp_attr[k].con is the connection status, with a value of 0 indicating no connection and a value of 1 indicating a connection; qp_attr[k].qid is the corresponding NVMe-oF queue number; qp_attr[k].ctrl_id and qp_attr[k].gqid are the controller number and global queue number associated with the NVMe-oF queue, provided by the NVMe-oF component, respectively.
[0033] The software for the NVMe component is used to initialize and configure the NVMe SSD disk; the hardware of the NVMe component is the NVMe SSD disk itself. The NVMe component software provides an NVMe array containing NumS elements, each representing an NVMe SSD disk, which records various existing attribute information for each NVMe SSD disk. NumS is a configurable value, its size limited by the target hardware, typically a value of 4.
[0034] The NVMe component software also provides an IO queue array (i.e., the sqcq array), containing NumS×NumR elements. Each element is an IO queue (including a completion queue CQ and an SQ queue) for an NVMe SSD. It is configured for each NVMe SSD during initialization, and subsequent data read and write operations on the NVMe SSD are completed through these IO queues.
[0035] Each IO queue in the IO queue array needs to be configured to each NVMe SSD disk in sequence, and recorded sequentially in the queue array within each element nvme[k] of the NVMe array (e.g., ...). Figure 1 The connections between elements of the `sqcq` array and elements of each `queue` array. `nvme[k]` also contains a `namespace` attribute that records the namespaces within the NVMe SSD (e.g., ...). Figure 1 As shown in the diagram, each disk is configured with only one namespace NS1 in this specification.
[0036] The NVMe-oF component is used for processing NVMe-oF commands. The NVMe-oF component's software executes Admin commands, including all commands in the Admin queue and the first Connect command in the IO queue. After processing the Connect command in the IO queue, the NVMe-oF component's software hands the IO queue over to the NVMe-oF component's hardware, which then continues processing subsequent commands (IO commands). Each NVMe-oF queue occupies one RDMA QP in the RDMA component.
[0037] The NVMe-oF component may also include the following parts: The controller array (ctrl_info): The software-provided controller array ctrl_info for the NVMe-oF component contains NumC elements, each representing controller data. Each controller (Ctrl) handles a connection from one Initiator, supporting a maximum of NumC Initiators connecting simultaneously. Each controller corresponds to one Initiator, providing an abstract NVMe disk for the Initiator to use. Each controller contains a controller ID ctrl_id, with a value range of [0, NumC-1]. The controller records one Admin queue and NumCio IO queues, corresponding to the Admin queue and IO queue of its abstract NVMe disk, respectively. It also records various status information such as whether each queue has an Initiator connection and whether the specific IO queue is connected. NumC is a configurable value, typically 16. NumCio is also a configurable value, typically 8 or 32.
[0038] Global queue number array (G_qid): The NVMe-oF component software provides a global queue number array G_qid, containing NumGqn elements, representing NumGqn global queue numbers [1, NumGqn]. Each controller's IO queue uniquely occupies one global queue number when a connection is established, and releases the occupation of that global queue number when the connection is closed.
[0039] Furthermore, the global queue number is the index of each element in the global queue number array G_qid plus 1. The value of each element can be 0 or 1, where 0 represents that the global queue number is not occupied and 1 represents that the global queue number is occupied. During initialization, each element of the global queue number array G_qid is initialized to 0, indicating that all global queue numbers are not occupied.
[0040] Furthermore, NumGqn is a configurable value that can be the smaller of the NumR-3 and the maximum number of IO queues for the NVMe SSD (typically 63 or more), or a slightly larger value (for easier alignment and other processing).
[0041] The queue command space mapping table (RqpSsdSqp) is provided by the NVMe-oF hardware and contains NumS×NumGqn elements. Each element can be populated with three values: rdma_qpn, nsid, and qid_i. Here, rdma_qpn is the RDMA QP number of an IO queue on the Target side, nsid is the namespace number to be accessed on the Initiator side, and qid_i is the index of the IO queue of the NVMe disk to be accessed in the sqcq array. After the NVMe-oF software completes the mapping from IO queues to global queue numbers, it adds a record to the queue command space mapping table for each NVMe SSD, completing the final mapping. When the NVMe-oF component processes I / O commands, it finds the qid_i value in the queue command space mapping table based on the RDMA QP number of the received I / O command and the nsid value provided in the I / O command. It then obtains the SQ queue and CQ queue from the sqcq array through the qid_i value, and interacts with the SQ queue and CQ queue to complete data read and write.
[0042] It should be noted that the arrays provided by the software (such as the qp_attr array, NVMe array, IO queue array, controller array, and global queue number array) are all in DDR memory, which can be efficiently accessed by the hardware.
[0043] The aforementioned embedded high-performance NVMe-oF Target system improves the hardware and software functions of the embedded high-performance NVMe-oF Target by utilizing RDMA, NVMe-oF, and NVMe components to provide a new joint mapping technology for queues and namespaces in the embedded high-performance NVMe-oF Target. It also considers the mapping of queues and namespaces and combines them with the embedded + high-performance environment. Under the premise of supporting multiple initiator access and multiple NVMe SSD disks for shared access, it greatly simplifies the mapping process, simplifies the hardware, and reduces the hardware implementation requirements.
[0044] The namespace and queue mapping relationship is described below: In one embodiment, namespace mapping is as follows: Figure 2 As shown, the embedded high-performance NVMe-oF Target uses a standard NVMe SSD disk, and the NVMe component configures the NVMe SSD disk to have only one namespace NS1.
[0045] Among them, the NVMe-oF component will connect each NVMe SSD disk (such as...) Figure 2 The namespace NS1 (containing NVMe SSD0, NVMe SSD1, NVMe SSD2, and NVMe SSD3) is mapped by the controller to multiple namespaces of the abstract NVMe disk (e.g., ...). Figure 2 The NVMe-oF components are divided into NS1, NS2, NS3, and NS4 within each controller. One NVMe SSD corresponds to one namespace within one controller.
[0046] Multiple controllers of NVMe-oF components (such as Figure 2 Using Ctrl0 and Ctrl15, the above mapping process can be repeated to map an NVMe SSD to the namespaces of different controllers (e.g., ...). Figure 2 The NS1 configured for NVMe SSD1 in the NVMe component is mapped to NS2 in ctrl0 of the NVMe-oF component, and also to NS2 in ctrl15 of the NVMe-oF component.
[0047] The namespace provided in each controller is the NVMe disk namespace abstracted by the controller, and it corresponds one-to-one with the abstract NVMe disk namespace seen by the Initiator after connecting to a controller. (e.g.) Figure 2 The NS1, NS2, NS3, and NS4 terminals of the two initiators correspond to the NS1, NS2, NS3, and NS4 terminals of the connected controllers, respectively.
[0048] In one embodiment, the queue mapping is as follows: Figure 3 As shown, the mapping of NVMe-oF queues at the Initiator and Target ends is illustrated, as well as the mapping of NVMe-oF queues to RDMA components. It also shows the mapping relationship between the IO queues of the NVMe-oF queues of each controller of the NVMe-oF component at the Target end and the IO queues of the NVMe SSDs managed by the NVMe component through the global queue number.
[0049] The initiator connects to the target and creates multiple RDMA connections. The first RDMA connection is used for the Admin queue, and the remaining RDMA connections are used for the IO queue. The RDMA QP number used by each RDMA connection on both the initiator and target is allocated by each end and can be configured to different values. The NVMe-oF queue corresponding to each RDMA connection has the same queue number on both the initiator and target, which is informed to the target by the initiator in its first command (i.e., the Connect command). Each RDMA connection corresponds one-to-one with an NVMe-oF queue; this is the standard practice in the NVMe-oF protocol specification.
[0050] In this embodiment, one element of the qp_attr array in the RDMA component represents an RDMA connection and also an NVMe-oF queue. When an RDMA connection is established, the Target finds an unconnected element qp_attr[i] in the qp_attr array, where i is the index of the element and also the corresponding RDMA QP number. qp_attr[i].con is recorded as 1 to indicate a connection, and qp_attr[i].qpn is recorded as i to indicate the RDMA QP number of the RDMA connection on the Target. After receiving the first command (i.e., the Connect command) from the Initiator via the RDMA connection, the Target records qp_attr[i].qid as the value qid specified by the Connect command, representing the queue number of the NVMe-oF queue.
[0051] exist Figure 3 Three RDMA connections are established between the Initiator (A) and Target (A). For example, the connection between Initiator A's rdma qp3 and Target's rdma qp2 is used for NVMe-oF queue 0 (i.e., the Admin queue); the connection between Initiator A's rdma qp4 and Target's rdma qp3, and the connection between Initiator A's rdma qp5 and Target's rdma qp4, are used for NVMe-oF queues 1 and 2 (i.e., the IO queues). Each NVMe-oF queue occupies and exclusively uses one RDMA QP (i.e., one rdma queue). Target's rdma qp0 is not used, and rdma qp1 is used for CM connection establishment to provide information for the establishment of the preceding RDMA connections.
[0052] After the IO queue is established, the NVMe-OF component assigns a global queue number (gqid) to each IO queue of each controller from the global queue number array G_qid. Simultaneously, it records the element qp_attr[i].gqid in the qp_attr array to map the IO queues of the NVMe-oF queues in the controller to the IO queues of each NVMe SSD. The numerical element qp_attr[i] is a structure whose gqid attribute records the global queue number. Subsequent IO commands in the IO queues of the NVMe-oF queue corresponding to the element qp_attr, regardless of which namespace is accessed (in the Target implementation, this means accessing the corresponding NVMe SSD), will use the queue number qp_attr[i].gqid of that NVMe SSD for access. qp_attr[i].gqid is the global queue number associated with the NVMe-oF queue provided by the NVMe-oF component.
[0053] Figure 3 The IO queue 1 of the middle controller Ctrl0 is assigned to global queue number 1. When accessing NVMe SSD0, IO queue 1 is used. When accessing NVMe SSD1, NVMe SSD2 and NVMe SSD3, IO queue 1 is still used.
[0054] In one embodiment, the workflow for creating an Admin queue may include the following: like Figure 4 As shown, the Target end creates an RDMA QP number in RC mode through CM connection establishment. The Target end allocates an RDMA QP number i and the corresponding array element qp_attr[i] to establish an RDMA connection as the transmission channel for the Admin queue.
[0055] The Target receives the Connect command from the Initiator in the RDMA connection, records the Admin queue number (0) of the NVMe-oF queue specified by the Connect command into the array element qp_attr[i], thus completing the one-to-one mapping between the Admin queue and the RDMA QP number. The Target finds an idle controller in the controller array ctrl_info and updates the corresponding controller number ctrl_id to the array element qp_attr[i], completing the mapping between the Admin queue and the controller, and returns the corresponding controller number ctrl_id to the Initiator.
[0056] The Target receives management commands from the Admin queue and returns information about the abstract NVMe disk provided by the controller, including disk size and maximum number of IO queues.
[0057] In one embodiment, the IO queue creation process can be as follows: Figure 5 As shown, the Initiator determines the number of IO queues to be created based on the maximum number of IO queues returned by the Target and its own needs. The specific workflow includes the following: The target end creates an RDMA QP number in RC mode through CM connection establishment. The target end allocates an RDMA QP number j and the corresponding array element qp_attr[j] to establish an RDMA connection as the transmission channel for the IO queue.
[0058] The Target receives the Connect command from the RDMA connection and records the IO queue number (non-zero value qid) and controller number ctrl_id of the NVMe-oF queue specified in the Connect command into the array element qp_attr[j]. Finally, the one-to-one mapping between the IO queue and the RDMA QP number, as well as the mapping between the IO queue and the controller are completed.
[0059] The Target finds an element with a value of 0 in the global queue number array G_qid (the index of this element + 1 is the global queue number gqid), sets the value of this element with a value of 0 to 1 (i.e. occupies this global queue number), and records the global queue number gqid in the array element qp_attr[j], thus completing the mapping of the IO queue of the NVMe-oF queue to the IO queue of each NVMe SSD disk.
[0060] The target software writes NumS configuration entries to the queue command space mapping table RqpSsdSqp. Each configuration entry includes three values: rdma_qpn, nsid, and qid_i, and then transfers control of the IO queue to the hardware. Here, nsid is the namespace number nsid that subsequent IO commands will carry, and the NumS configuration entries take values {1, 2, ..., NumS}; the index qid_i is the global queue number gqid-1.
[0061] If the Initiator needs to create more IO queues, repeat the process described above.
[0062] In one embodiment, the IO command processing flow can be as follows: Figure 6 As shown, the Target receives IO commands from the Initiator, and the workflow for data reading and writing includes the following: The target hardware receives IO commands from the initiator and determines the RDMA QP number rdma_qpn based on the RDMA QP where the IO command is located.
[0063] The target hardware parses the I / O commands and obtains the namespace number nsid from the I / O commands.
[0064] The target hardware uses the RDMA QP number rdma_qpn and namespace number nsid to look up the queue command space mapping table RqpSsdSqp and find the index qid_i of the IO queue array; qid_i is the index of the IO queue of the NVMe SSD disk that the IO queue array needs to use.
[0065] The target hardware uses the index qid_i of the IO queue array to find the IO queue sqcq[qid_i] of the NVMe SSD disk that needs to be used.
[0066] The target hardware sends IO commands to the IO queue sqcq[qid_i] of the NVMe SSD and retrieves the command execution results.
[0067] The target hardware sends the execution results of the IO commands to the initiator through the IO queue of the NVMe-oF component.
[0068] It is understood that the queue and namespace mapping technology described above in this invention adopts a combination of hardware and software, with software configuration as the primary method. The processing flow includes two parts: connection establishment and IO command processing. Connection establishment includes two processes: creating an Admin queue and creating an IO queue. The creation of the IO queue is executed after the creation of the Admin queue is completed.
[0069] The aforementioned embedded high-performance NVMe-oF Target system employs a global queue identifier array for unified management of queue resources. The allocation algorithm is simple, efficient, and easy to implement in hardware. The Admin command queue is handled by software to ensure flexibility, while IO commands are handled directly by hardware for high performance. The IO queues in the NVMe-oF queues share the same IO queue number across all NVMe SSDs, simplifying the queue mapping logic and reducing hardware implementation complexity. In environments with limited RDMA QPs, the system supports simultaneous connections from as many initiators as possible and access to as many IO queues as possible through a concise mapping mechanism. Namespace mapping enables multiple initiators to share NVMe SSDs, significantly improving storage resource utilization.
[0070] In some implementations, to more intuitively illustrate the above design, a specific example of an embedded high-performance NVMe-oF target system is provided, employing an MCU plus bare driver architecture, with the underlying network transmission using the RDMA RoCEv2 protocol. A typical configuration could be: NumR=32, NumS=4, NumC=16, NumCio=8, NumGqn=32. The RDMA software provides a qp_attr array containing 32 elements.
[0071] The software of the NVMe component is responsible for initializing and configuring the NVMe SSD disk, while the hardware of the NVMe component is the NVMe SSD disk. Four NVMe SSD disks are inserted into the target end, and each SSD is configured with a single namespace.
[0072] The NVMe component software provides an NVMe array containing four elements, each representing an NVMe SSD disk, which records various attribute information such as the IO queue and namespace of each NVMe SSD disk.
[0073] The NVMe component software also provides an IO queue array (i.e., the sqcq array), containing 4×32=128 elements. Each element represents an IO queue (including a commit queue (SQ) and a completion queue (CQ)) for an NVMe SSD. It is configured for the NVMe SSD during initialization, and subsequent data read and write operations on the NVMe SSD are completed through these IO queues.
[0074] Each element nvme[k] in the NVMe array contains a namespace attribute that records the namespaces contained in the NVMe SSD (in this specification, only one namespace NS1 is configured for each disk); and a queue array that records the IO queues belonging to the k-th NVMe SSD in the IO queue array. The elements in the IO queue array need to be allocated to each NVMe SSD in sequence and recorded sequentially in the queue array of each NVMe SSD. Specifically: sqcq[0,31] in the IO queue array is assigned to element nvme[0] in turn and recorded in queue[1,32] of element nvme[0]; sqcq[32:63] in the IO queue array is assigned to element nvme[1] in turn and recorded in queue[1:32] of element nvme[1]; sqcq[64:95] in the IO queue array is assigned to element nvme[2] in turn and recorded in queue[1:32] of element nvme[2]; sqcq[96:127] in the IO queue array is assigned to element nvme[3] in turn and recorded in queue[1:32] of element nvme[3].
[0075] The NVMe-oF component is responsible for processing NVMe-oF commands. The NVMe-oF component's software handles Admin commands, including all commands in the Admin queue and Connect commands in the IO queue. After processing the Connect commands in the IO queue, the NVMe-oF component's software hands the IO queue over to the NVMe-oF component's hardware, which then continues processing subsequent commands (such as IO commands) in the IO queue.
[0076] More specifically, NVMe-oF components may include the following parts: Controller Array (ctrl_info): The NVMe-oF component software provides a controller array (ctrl_info) containing 16 elements, each representing data for a controller. Each controller handles a connection from one Initiator, supporting up to 16 Initiators simultaneously. Each controller corresponds to one Initiator, providing an abstract NVMe disk for the Initiator to use. Each controller contains a controller ID (ctrl_id) ranging from [0,15]. The controller records one Admin queue and eight IO queues, corresponding to the Admin and IO queues of its abstract NVMe disk, and records various status information such as whether an Initiator is connected and whether a specific IO queue is connected.
[0077] Global Queue Number Array (G_qid): The NVMe-oF component software provides a global queue number array G_qid, containing 32 elements representing 32 global queue numbers [1, 32]. Each controller's I / O queue uniquely occupies one global queue number when a connection is established, and releases the occupation of that global queue number when the connection is closed. Specifically, the global queue number is the index of each element in the global queue number array G_qid + 1. The value of each element can be 0 or 1, where 0 represents that the global queue number is not occupied, and 1 represents that the global queue number is occupied. During initialization, each element is initialized to 0, indicating that all global queue numbers are not occupied.
[0078] The queue command space mapping table (RqpSsdSqp) is provided by the hardware of the NVMe-oF component and contains 32 × 4 = 128 elements. Each element can be filled with three values: rdma_qpn, nsid, and qid_i. All of these software-provided arrays reside in memory (DDR) and are accessible to the hardware.
[0079] In some implementations, such as Figure 4As shown, to illustrate the above design more intuitively, one specific workflow for creating an Admin queue is also provided. When Initiator A initiates a connection, Target performs the following steps: The RDMA QP number created in RC mode through CM connection establishment is 2, that is, an unconnected array element qp_attr[2] (i.e. qp_attr[2].con==0) is allocated in the qp_attr array, and interconnected with the RDMA QP3 of Initiator A (assuming that Initiator A uses RDMA QP3) to establish an RDMA connection as the transmission channel for the Admin queue. Update the array element qp_attr[2] record: qp_attr[2].con=1; qp_attr[2].qpn=2.
[0080] The Target receives the Connect command from the Initiator A in the RDMA connection, records the Admin queue number (fixed to 0) of the NVMe-oF queue specified by the Connect command into the array element qp_attr[2] (i.e., qp_attr[2].qid=0), and finally completes the one-to-one mapping between the Admin queue and the RDMA QP number. It searches for an idle controller in the controller array ctrl_info, updates the controller number 0 of the idle controller to the array element qp_attr[2] (i.e., qp_attr[2].ctrl_id=0), completes the mapping between the Admin queue and the controller, and returns the controller number 0 to the Initiator A.
[0081] On the Initiator (A) side, management commands such as get / set feature are executed in the Admin queue. On the Target side, management commands sent from the Admin queue are received, and information about the abstract NVMe disk provided by the controller is returned, including disk size and maximum number of IO queues. The get / set feature commands are the core management commands in the NVMe protocol used to query and configure the operating characteristics of SSD disks.
[0082] In some implementations, such as Figure 5 As shown, to illustrate the above design more intuitively, one specific IO queue creation process is also provided. The Initiator determines the number of IO queues to be created based on the maximum number of IO queues (8) returned by the Target and its own needs. The workflow includes the following steps: The RDMA QP number is 3 in RC mode created by CM connection establishment. The Target end allocates an RDMA QP number 3 and the corresponding array element qp_attr[3] to establish an RDMA connection as the transmission channel of the IO queue.
[0083] The Target receives the Connect command from the RDMA connection and records the IO queue number 1 and controller number 0 of the NVMe-oF queue specified in the Connect command into the array element qp_attr[3], thus completing the one-to-one mapping between the IO queue and the RDMA QP, as well as the mapping between the IO queue and the controller.
[0084] The Target finds an element with a value of 0 in the global queue number array G_qid. Since the index of this element is 0, the global queue number gqid is 1 (i.e., 0+1). Setting G_qid[0] to 1 indicates that it is occupied. The global queue number gqid is recorded in the array element qp_attr[3] (i.e., qp_attr[3].gqid=1), completing the mapping of the IO queue of the NVMe-oF queue to the IO queue of each NVMe SSD disk.
[0085] The target writes four configuration entries to the hardware queue command space mapping table RqpSsdSqp. Each entry contains three values: rdma_qpn, nsid, and qid_i. The specific values for each configuration entry are {3,1,0}, {3,2,0}, {3,3,0}, and {3,4,0}, respectively. Control of IO queue 1 is then transferred to the hardware. The nsid values for the four configuration entries are {1,2,3,4}, and the index qid_i is the previous step's gqid-1, with a value of 0.
[0086] In some implementations, such as Figure 6 As shown, to illustrate the above design more intuitively, one specific IO command processing flow is also provided. The Target receives IO commands from the Initiator, and the data read / write workflow can be as follows: The initiator sends read / write I / O commands to the target through I / O queue 1. The hardware processing flow on the target is as follows: The target hardware receives IO commands from RDMA QP3 and determines the RDMA QP number rdma_qpn==3 based on the RDMA QP where the IO command is located.
[0087] The target hardware parses the I / O commands and obtains the namespace number nsid==1 from the I / O commands.
[0088] The target hardware uses rdma_qpn==3 and nsid==1 to look up the queue command space mapping table RqpSsdSqp and find the index qid_i=0 of the IO queue array.
[0089] The target hardware uses index qid_i to find the IO queue sqcq[0] of the NVMe SSD disk to be used from the IO queue array.
[0090] The target hardware sends IO commands to the IO queue sqcq[0] of the NVMe SSD and obtains the command execution results.
[0091] The target hardware returns the execution result of the IO command to the initiator through the IO queue of the NVMe-oF component.
[0092] Each component module in the aforementioned embedded high-performance NVMe-oF Target end system can be implemented entirely or partially through software, hardware, or a combination thereof. These components can be embedded in hardware or independently of a device with protocol data processing capabilities, or stored in software within the memory of the aforementioned device, so that the processor can call and execute the corresponding operations of each component. The aforementioned device can be, but is not limited to, various types of network computer devices already existing in the art.
[0093] In one embodiment, such as Figure 7 As shown, a processing method for an embedded high-performance NVMe-oF Target end system is also provided. Based on the above-described embedded high-performance NVMe-oF Target end system, the processing method may include the following steps: S10, the Target end creates an RDMA QP number in RC mode through CM connection establishment. The Target end allocates an RDMA QP number and the corresponding array element in the qp_attr array to establish an RDMA connection as the transmission channel for the Admin queue. S12, the Target receives the Connect command from the Initiator from the RDMA connection and records the Admin queue number of the NVMe-oF queue specified by the Connect command into the corresponding array element in the qp_attr array; S14, the Target finds an idle controller from the controller array and updates the corresponding controller number to the corresponding array element in the qp_attr array, and returns the corresponding controller number to the Initiator. S16, the Target receives management commands from the Admin queue and returns information about the abstract NVMe disk provided by the controller.
[0094] The aforementioned embedded high-performance NVMe-oF Target system employs a global queue identifier array for unified management of queue resources. The allocation algorithm is simple, efficient, and easy to implement in hardware. The Admin command queue is handled by software to ensure flexibility, while IO commands are handled directly by hardware for high performance. The IO queues in the NVMe-oF queues share the same IO queue number across all NVMe SSDs, simplifying the queue mapping logic and reducing hardware implementation complexity. In environments with limited RDMA QPs, the system supports simultaneous connections from as many initiators as possible and access to as many IO queues as possible through a concise mapping mechanism. Namespace mapping enables multiple initiators to share NVMe SSDs, significantly improving storage resource utilization.
[0095] In one embodiment, the processing method of the above-described embedded high-performance NVMe-oF target end system may further include the following steps: The target end creates an RDMA QP number in RC mode through CM connection establishment. The target end allocates an RDMA QP number j and the corresponding array element qp_attr[j] to establish an RDMA connection as the transmission channel for the IO queue. The Target receives the Connect command from the RDMA connection and records the IO queue number and controller number of the NVMe-oF queue specified in the Connect command into the array element qp_attr[j]. The Target finds an element with a value of 0 in the global queue number array and sets it to 1, then records the global queue number in the array element qp_attr[j]. The target software writes NumS configuration entries to the queue command space mapping table and transfers control of the IO queue to the hardware. Each configuration entry includes rdma_qpn, nsid, and qid_i, where NumS is the number of elements in the NVMe array, rdma_qpn is the RDMA QP number of an IO queue on the target side, nsid is the namespace number that will be carried by subsequent IO commands, and qid_i is the index of the IO queue of the NVMe disk to be accessed in the IO queue array.
[0096] In one embodiment, the processing method of the above-described embedded high-performance NVMe-oF target end system may further include the following steps: The target hardware receives the IO command from the initiator and determines the RDMA QP number based on the RDMA QP where the IO command is located. The target hardware parses I / O commands and obtains the namespace number from the I / O commands; The target hardware uses the RDMA QP number and namespace number to look up the queue command space mapping table and find the index of the IO queue array; The target hardware uses the index of the IO queue array to find the IO queue of the NVMe SSD disk that needs to be used; The target hardware sends IO commands to the IO queue of the NVMe SSD and obtains the command execution results; The target hardware sends the execution results of the IO commands to the initiator through the IO queue of the NVMe-oF component.
[0097] It is understood that the explanations of the features of the processing method of the above-mentioned embedded high-performance NVMe-oF Target terminal system can be understood by referring to the explanations of the corresponding features of the various embodiments of the above-mentioned embedded high-performance NVMe-oF Target terminal system, and will not be repeated here.
[0098] It should be understood that, although Figures 4 to 7 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Figures 4 to 7 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM). By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus type dynamic random access memory (Rambus DRAM, or RDRAM for short), and direct Rambus DRAM (DRDRAM), etc.
[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0101] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.
Claims
1. An embedded high-performance NVMe-oF target terminal system, characterized in that, Including RDMA components, NVMe-oF components, and NVMe components; The RDMA component is used to perform RDMA network transmission. It uses CM to establish a connection and creates an RDMA QP in RC mode. The RDMA QP in RC mode provides a command transmission channel for the NVMe-oF component. The RDMA software provides a qp_attr array with a configurable number of elements. The qp_attr array is used to record the RDMA connections that correspond one-to-one with the NVMe-oF queue. The NVMe component software is used to initialize and configure NVMe SSDs. The NVMe component software provides NVMe arrays and IO queue arrays with configurable number of elements. The NVMe array is used to record the attribute information of each NVMe SSD and the namespaces in the NVMe SSD. Each element of the IO queue array is an IO queue in an NVMe SSD. The NVMe-oF component's software executes Admin commands. After processing the Connect command for the IO queue, the NVMe-oF component's software transfers the IO queue to the NVMe-oF component's hardware for further processing of subsequent commands. Each NVMe-oF queue occupies one RDMA QP in the RDMA component. The NVMe-oF component's software provides a configurable array of controllers and a global queue number array. Each element in the controller array represents controller data, and each controller handles a connection at the Initiator end. The global queue number array records the unique global queue number used by the IO queues under each controller during connection. The NVMe-oF component's hardware provides a configurable queue command space mapping table, which records the mapping relationship between IO queues, global queue numbers, and namespaces.
2. The embedded high-performance NVMe-oF Target terminal system according to claim 1, characterized in that, The elements of the qp_attr array also record the connection status of the RDMA connection, the NVMe-oF queue number, the controller number, and the global queue number.
3. The embedded high-performance NVMe-oF Target terminal system according to claim 1, characterized in that, The global queue number is the index of each element in the global queue number array plus 1. The value of each element in the global queue number array includes 0 and 1, where 0 represents that the global queue number is not occupied and 1 represents that the global queue number is occupied.
4. The embedded high-performance NVMe-oF Target terminal system according to claim 3, characterized in that, The number of elements in the global queue number array is the smaller of the number of elements in the qp_attr array minus 3 and the maximum number of IO queues for the NVMe SSD.
5. The embedded high-performance NVMe-oF Target terminal system according to any one of claims 1 to 4, characterized in that, The NVMe-oF component maps the namespaces of each NVMe SSD to multiple namespaces of the abstract NVMe disk through the controller.
6. The embedded high-performance NVMe-oF Target terminal system according to claim 5, characterized in that, When an RDMA connection is established, the Target finds an unconnected element in the qp_attr array. After receiving the Connect command from the Initiator via the RDMA connection, the Target uses the elements of the qp_attr array to record the queue number of the NVMe-oF queue specified by the Connect command. After the IO queue is established, the NVME-OF component allocates a global queue number to each IO queue of each controller from the global queue number array, and at the same time records the gqid attribute of each element in the qp_attr array as the global queue number.
7. The embedded high-performance NVMe-oF Target terminal system according to claim 5, characterized in that, The qp_attr array, NVMe array, IO queue array, controller array, and global queue number array are all in DDR memory.
8. A processing method for an embedded high-performance NVMe-oF target terminal system, characterized in that, Based on the embedded high-performance NVMe-oF Target terminal system according to any one of claims 1 to 7, the processing method includes the following steps: The Target side establishes an RDMA QP number in RC mode through CM connection establishment. The Target side allocates an RDMA QP number and the corresponding array element in the qp_attr array to establish an RDMA connection as the transmission channel for the Admin queue. The Target receives the Connect command from the Initiator from the RDMA connection and records the Admin queue number of the NVMe-oF queue specified by the Connect command into the corresponding array element in the qp_attr array; The Target finds an idle controller from the controller array, updates the corresponding controller number to the corresponding array element in the qp_attr array, and returns the corresponding controller number to the Initiator. The Target receives management commands from the Admin queue and returns information about the abstract NVMe disk provided by the controller.
9. The processing method of the embedded high-performance NVMe-oF Target terminal system according to claim 8, characterized in that, It also includes the following steps: The target end creates an RDMA QP number in RC mode through CM connection establishment. The target end allocates an RDMA QP number j and the corresponding array element qp_attr[j] to establish an RDMA connection as the transmission channel for the IO queue. The Target receives the Connect command from the RDMA connection and records the IO queue number and controller number of the NVMe-oF queue specified in the Connect command into the array element qp_attr[j]. The Target finds an element with a value of 0 in the global queue number array and sets it to 1, then records the global queue number in the array element qp_attr[j]. The target software writes NumS configuration entries to the queue command space mapping table and transfers control of the IO queue to the hardware. Each configuration entry includes rdma_qpn, nsid, and qid_i, where NumS is the number of elements in the NVMe array, rdma_qpn is the RDMA QP number of an IO queue on the target side, nsid is the namespace number that will be carried by subsequent IO commands, and qid_i is the index of the IO queue of the NVMe disk to be accessed in the IO queue array.
10. The processing method of the embedded high-performance NVMe-oF Target terminal system according to claim 8, characterized in that, It also includes the following steps: The target hardware receives the IO command from the initiator and determines the RDMA QP number based on the RDMA QP where the IO command is located. The target hardware parses I / O commands and obtains the namespace number from the I / O commands; The target hardware uses the RDMA QP number and namespace number to look up the queue command space mapping table and find the index of the IO queue array; The target hardware uses the index of the IO queue array to find the IO queue of the NVMe SSD disk that needs to be used; The target hardware sends IO commands to the IO queue of the NVMe SSD and obtains the command execution results; The target hardware sends the execution results of the IO commands to the initiator through the IO queue of the NVMe-oF component.