Modify NVMe PRP list pointers and data pointers to facilitate routing PCIe memory requests

Abstract

Embodiments of the present disclosure relate to modifying NVMe PRP list pointers and data pointers to facilitate routing PCIe memory requests. A first command is received from a virtual or physical host, the virtual or physical host being associated with a storage system comprising two or more hosts. The first command includes one or more physical request page (PRP) entries associated with a non-volatile memory express (NVMe) standard. The one or more PRP entries are modified with an indication of the virtual or physical host. A second command is sent to a solid state drive (SSD) with the modified one or more PRP entries. A memory request is received from the SSD, wherein the memory request includes the modified one or more PRP entries. Based on the indication of the virtual or physical host in the modified one or more PRP entries, the memory request is routed to the virtual or physical host.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present disclosure claims the benefit of priority to U.S. Provisional Application Serial No. 62 / 716,284, filed on August 8, 2018, entitled “METHOD AND APPARATUS FOR IN-LINE PROCESSING OF AN NVMe PRP LIST DURING THE PCIe FETCH OPERATION,” and U.S. Patent Application No. 16 / 532,186, filed on August 5, 2019, entitled “MODIFYING NVMe PHYSICALREGION PAGE LIST POINTERS AND DATA POINTERS TO FACILITATE ROUTING OF PCIeMEMORY REQUESTS,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to providing storage services to two or more hosts, which may be virtual hosts and / or physical hosts, namely, to modifying physical region page (PRP) data pointers and list pointers associated with the Non-Volatile Memory Express (NVMe) standard to facilitate routing Peripheral Component Interconnect Express (PCIe) memory requests to one of the two or more hosts. Background Art

[0004] PCI Express (Peripheral Component Interconnect Express) is a high-speed serial computer expansion bus standard, which is officially abbreviated as PCIe or PCI Express. PCI Express operates as a motherboard-level interconnect between a host and peripheral components, such as an attached solid-state drive (SSD). Non-Volatile Memory Express (also known as NVMe or NVM Express) is used to manage data stored on an SSD by providing a command set for SSD access. The host sends NVMe commands to the SSD to store and retrieve data from the SSD. These commands utilize physical region page (PRP) entries to represent buffer locations in the host memory. PRP entries take the form of a data pointer to the data or a list pointer to a list of data pointers (similar to a linked list). The SSD can request data movement associated with the PRP entry when executing a command, such as a PCIe memory read request. When a single host is associated with the SSD, the routing of PCIe memory read requests is clear. PCIe memory read requests are sent to a single host. When two or more hosts (such as physical hosts and / or virtual hosts) are associated with the SSD, routing PCIe memory read requests becomes more complicated. The PRP entry does not provide an indication of which of the two or more hosts is associated with the PRP entry, which would otherwise facilitate routing to that host. Summary of the invention

[0005] The present disclosure relates to a non-volatile memory express (NVMe) switch that facilitates NVMe-based access to multiple solid-state drives (SSDs) by one of two or more hosts, where the host can be a physical host and / or a virtual host. The NVMe switch modifies a physical region page (PRP) entry (such as a data pointer or a list pointer) of an NVMe command with a host ID or a virtual function (VF) ID and a tag, where the host ID or the virtual function (VF) ID is associated with the host that sends the NVMe command to the NVMe switch, and the tag indicates whether the PRP entry is a data pointer or a list pointer. The host ID or the VF ID facilitates routing a peripheral component interconnect express (PCIe) memory request with a modified PRP data pointer or list pointer to one of the two or more hosts associated with the NVMe command to execute the PCIe memory request.

[0006] According to one aspect of the described systems and techniques, a method includes: receiving a first command from a host associated with a storage system including two or more hosts, wherein the first command includes one or more physical request page (PRP) entries associated with a non-volatile memory express (NVMe) standard; modifying the one or more PRP entries with an indication to the host; sending a second command to a solid-state drive (SSD), the second command having the modified one or more PRP entries; receiving a memory request from the SSD, wherein the memory request includes the modified one or more PRP entries; and routing the memory request to the host based on the indication to the host in the modified one or more PRP entries.

[0007] According to another aspect of the described systems and techniques, a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to at least: receive a first command from a host associated with a storage system comprising two or more hosts, wherein the first command includes one or more physical request page (PRP) entries associated with a non-volatile memory express (NVMe) standard; modify the one or more PRP entries with an indication to the host; send a second command to a solid-state drive (SSD), the second command having the modified one or more PRP entries; receive a memory request from the SSD, wherein the memory request includes the modified one or more PRP entries; and route the memory request to the host based on the indication to the host in the modified one or more PRP entries.

[0008] According to yet another aspect of the described systems and techniques, a non-volatile memory express (NVMe) switch includes a command processor and a data processor, the command processor including instructions stored in a memory of the NVMe switch, which instructions, when executed by one or more processors of the NVMe switch, cause the NVMe switch to at least perform: receiving, by the command processor, a first command from a host associated with a storage system including two or more hosts, wherein the first command includes one or more physical request page (PRP) entries associated with a non-volatile memory express (NVMe) standard; modifying, by the command processor, the one or more PRP entries with an indication to the host; and sending, by the command processor, a second command to a solid-state drive (SSD), the second command having the modified one or more PRP entries; and the data processor including instructions stored in the memory of the NVMe switch, and which instructions, when executed by one or more processors of the NVMe switch, cause the NVMe switch to at least perform: receiving, by the command processor, a memory request from the SSD, wherein the memory request includes the modified one or more PRP entries; and routing the memory request to the host based on the indication to the host in the modified one or more PRP entries.

[0009] In this regard, the NVMe switch allows routing of PCIe memory requests associated with NVMe commands generated by a host when the storage system includes two or more hosts. The PCIe memory requests are routed to the host. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A and Figure 1B An example storage system is illustrated that facilitates handling of physical region pages (PRPs) when the storage system includes two or more hosts.

[0011] Figure 2 is an example flow diagram of functionality associated with processing a PRP entry associated with a Non-Volatile Memory Express (NVMe) command received from one of two or more hosts.

[0012] Figure 3 is an example flow diagram of functionality associated with handling PCIe memory requests from an SSD.

[0013] Figure 4 Illustrated is an example process for modifying a PRP entry associated with an NVMe command, and retrieving and modifying an associated data pointer and / or list pointer associated with a list of pointers retrieved from a host based on the PRP entry.

[0014] Figure 5is a simplified block diagram of an NVMe switch.

[0015] The drawings are for the purpose of illustrating example embodiments, but it is to be understood that the embodiments are not limited to the arrangements and instrumentality shown in the drawings. DETAILED DESCRIPTION

[0016] The present disclosure relates to a non-volatile memory express (NVMe) switch that facilitates NVMe-based access to multiple solid-state drives (SSDs) by one of two or more hosts. The host mentioned herein may be a physical host or a virtual host such as a virtual machine (VM). The NVMe switch modifies a physical region page (PRP) entry (such as a data pointer or a list pointer) of an NVMe command using a host ID or a virtual function (VF) ID and a tag, the host ID or the virtual function (VF) ID being associated with the host sending the NVMe command, the tag indicating whether the PRP entry is a data pointer or a list pointer. The host ID or the VF ID facilitates routing a peripheral component interconnect express (PCIe) memory request with a modified PRP data pointer or a list pointer to one of two or more hosts associated with the NVMe command. The principles described herein may be applied to control the performance of other types of storage devices such as hard disk drives (HDDs) or hybrid SSD / HDD drives.

[0017] Figure 1A and Figure 1B An example storage system is illustrated to facilitate the processing of physical region pages (PRPs) when the storage system includes two or more hosts. For ease of illustration, the example storage system is cross- Figure 1A and Figure 1B, wherein reference numeral A interconnects the structures associated with the two figures. The example storage system 100 may include two or more hosts, shown as host 102 and host 104, one or more solid-state drives (SSDs), shown as SSDs 106-112, and NVMe switches 116. Two or more hosts 102-104 may be used as servers that provide information resources, services, and / or applications to users or other nodes on a network. In some examples, the host may be an actual physical host. In other examples, the host may be a virtual machine (VM) or a virtual host. VM or virtual hosts may be implemented on an actual physical host using PCIe single root input / output virtualization (SR-IOV) and associated with a virtual function (VF). SSD is a storage mechanism based on an integrated circuit (IC) that can store data persistently to enable the host to provide the desired functionality. Compared to a hard drive, SSD106-112 can provide higher bandwidth and lower latency. NVMe switches 116 may be positioned between hosts 102-104 and SSDs 106-112. The NVMe switch 116 controls the PCIe-based point-to-point switch connections between the hosts 102-104 and the SSDs 106-112 so that each SSD 106-112 has its own dedicated connection to the host 102-104.

[0018] NVMe switch 116 includes PCIe endpoint cluster 118, command acquisition (fetch) engine 120, system fabric (fabric) 122, command processor 124, data processor 126 and PCIe root complex (RC) cluster 128. PCIe endpoint cluster 118 can be coupled to two or more hosts 102-104. NVMe is used to manage data stored on SSD 106-112 by providing a command set for SSD access. In an example, two or more hosts 102-104 can implement queue pairs according to NVMe to facilitate SSD access. The queue pair consists of a submission queue and a completion queue. For example, the host can generate NVMe commands to obtain data stored in the SSD or store data in the host's memory to the SSD. When the SSD executes the command, the completion of the execution of the notification command can be stored in the completion queue. Command 130 is an example of a command placed on the submission queue by the host 102, and command 132 is an example of a command placed on the submission queue by the host 104. Each command may be associated with an example command type shown by an OPCODE, an example data operation size shown as "# of Logical Block Addresses (LBAs)", and include a Physical Region Page (PRP) entry with a PRP address that serves as a pointer to identify the location of the data or list in the host's memory domain. NVMe-compliant commands may utilize PRP entries to identify data associated with get and store operations.

[0019] Although only one PRP is shown in commands 130, 132, two PRP entries are typically associated with the PRP entry. The PRP entries may be identified as PRP1 and PRP2. The PRP1 entry may be a PRP data pointer that points to data stored in the host, which in some examples may be 4K data. The data pointer may have a PCIe address of a host memory in which the data is stored. The PRP2 entry may also have a PRP data pointer or a PRP list pointer. If the PRP2 entry is a data pointer, the PRP2 entry points to data stored in the host, which may be 4K data, so that the PRP1 entry and the PRP2 entry allow 8K operations to be performed. If the PRP2 entry is a list pointer, the PRP2 entry points to a list of pointers stored in the host. The list pointer may have a PCIe address of a host memory in which data pointers 2 to n or a combination of data pointers and list pointers are stored.

[0020] The PCIe connection between two or more hosts and the NVMe switch 116 may include 2, 4, 8, or 16 data transmission channels. Each transmission channel may include two pairs of wires, one pair for transmission and one pair for reception. Two or more hosts may be coupled to the NVMe switch 116 via a PCIe connection so that each of the two or more hosts can be uniquely addressable by a PCIe address. The two or more hosts may be actual physical hosts or include VMs. The command acquisition engine 120 may acquire commands from two or more hosts, and then forward the commands to the command processor 124 through the internal nodes of the NVMe switch 116, via the system structure 122, or the cross switch. For example, the command acquisition engine 120 acquires commands from the NVMe submission queue in the host memory to the local memory of the NVMe switch 116 for processing. The command acquisition engine 120 facilitates the delivery of queue entries in the host memory queue submitted by the host CPU to the local memory queue for processing by the local CPU. The command processor 124 determines to which SSD each command will be forwarded, and then the PCIe RC cluster 128 provides the command to the determined SSD. For example, commands 130 associated with host 102 may be sent to SSD 106 , while commands 132 associated with host 104 may be sent to SSD 112 .

[0021] Each SSD receiving the command may generate a PCIe memory request based on the received command. The NVMe switch 116 may receive the request to route it to the host that generates the command associated with the request. The PCIe memory request may include a PRP entry for the command. The PRP entry may include a 64-bit PCIe address to identify the location of the data or list in the memory domain of the host, but it does not typically indicate the host associated with the PRP entry. An indication of the host that generated the PRP entry is required because the NVMe switch 116 needs to route the PCIe memory request with the PRP entry to the host that generated the PRP entry to execute the PCIe memory request.

[0022] In an example, the command processor 124 embeds an indication of the host before sending the command from the host to the SSD to facilitate the routing. The indication is embedded in the PRP address associated with the command, which is sent from the command processor 124 to the SSD via the PCIe RC cluster 128. For example, the command 134 associated with the host 102 and the command 136 associated with the host 104 sent to the SSDs 106, 112 may have different PRP addresses (shown as "0A" and "0B" in the higher bits, or specifically in the upper 2 octets) to indicate that each command is associated with a different host when received by the corresponding SSD. When the SSD sends a PCIe memory request with a PRP entry to the NVMe switch 116 to retrieve data from the host associated with the PRP entry, the SSD includes an indication of the host in the PCIe memory request sent to the NVMe switch 116, so that the NVMe switch 116 has a host ID (i.e., host identifier) ​​for a physical host or a VF ID (i.e., virtual function identifier) ​​for a virtual host or VM. For example, the PCIe memory read request 138 generated by the SSD 106 and the PCIe memory read request 140 generated by the SSD 112 have a PRP entry with an indication of the host embedded by the command processor. The data processor 126 sends the PCIe memory request to the appropriate host via the PCIe EP cluster 118 to perform the operation. For example, the PCIe memory request can be routed to the host based on the host ID and / or VF ID in the PRP entry, and as shown, the PCIe memory read request 138 is routed to the host 102 and the PCIe memory read request 140 is routed to the host 104. The embedded host identification allows the data processor to know where to route the PCIe memory request in order to reach a host that can execute the PCIe memory request. In addition, the data processor 126 can tag the list of pointers obtained from the PCIe memory request with information associated with whether the pointer is a data pointer or a list pointer and the host ID and / or VF ID. In some examples, the host ID or VF ID can also be considered a tag.

[0023] Example Operation

[0024] Figure 2is an example flow chart 200 of functionality associated with processing a PRP entry associated with an NVMe command received from one of two or more hosts. The functionality facilitates subsequent routing of a PCIe memory request to the host that sent the NVMe command, wherein the PCIe memory request is received from an SSD to retrieve a list of data or pointers associated with the PRP entry in the NVMe command. The functionality may be performed via a command processor in firmware, software, hardware, or a combination of hardware, software, and / or firmware.

[0025] At 202, a storage access command may be received from a host. The storage access command may be an NVMe command for reading data from an SSD or writing data to an SSD. The NVMe command may include two PRP entries referred to as PRP1 and PRP2. PRP1 may be a data pointer, and PRP2 may be a data pointer or a list pointer. The pointer may be an address to a host memory having a list of data and / or data pointers.

[0026] At 204, the address of PRP1 is modified to include an indication associated with a data pointer and an indication of a host associated with PRP1. A tag may be inserted into PRP1 to indicate that it is associated with a data pointer. For example, the tag may be a unique set of bits indicating that PRP1 is associated with a data pointer. The NVMe command may be associated with metadata indicating which host sends the command. The metadata may be indicated by routing information for sending the NVMe command from the host to the NVMe switch. PRP1 may be modified to include a host ID, which is used to indicate the host that sends the NVMe command. In some examples, PRP1 may be modified to include a virtual function (VF) ID. The VF ID may indicate a VM or a virtual host to uniquely identify the VM or virtual host. Data tags, VF IDs, and / or host IDs may be inserted in many ways. For example, the higher address bits of the address associated with PRP1 may not be used or reserved, and are replaced by data tags and hosts associated with PRP1 (e.g., VF IDs and / or host IDs). The NVMe switch may store a copy of the replaced bits.

[0027] At 206, it is determined whether PRP2 is valid. The validity of the PRP2 entry can be based on the product of the input / output (I / O) size associated with the command, indicated in some examples by a "# of LBA" entry in the NVMe command, and the size of the LBA can be a fixed value established during host configuration. The I / O size indicates the amount of data associated with the data pointer referenced by PRP1 and PRP2. For example, if the I / O is 4K, PRP1 may be sufficient to point to the data associated with the NVMe command. PRP2 does not point to any additional data pointers or lists of data pointers because the data is identified by PRP1 and PRP2 is invalid. At 214, if PRP2 is invalid, the NVMe command with the modified PRP1 is provided to the SSD for processing.

[0028] If the I / O is greater than 4K, PRP2 is valid, and at 208, it is determined whether PRP2 is a data pointer or a list pointer. For example, if the I / O size is greater than 8K, PRP2 may be a list pointer pointing to a list of data pointers, or if the I / O size is greater than 4K but less than 8K, PRP2 may be a data pointer. If PRP2 is a data pointer, at 216, the address of PRP2 is modified to include an indication associated with the data pointer (such as a data tag), and the VF ID and / or host ID that sent the NVMe command with PRP2. If PRP2 is a list pointer, at 210, it is determined whether there is a memory offset in the host memory where the list starts. If there is no offset, at 212, the address of PRP2 is modified to include an indication associated with the list pointer (such as a PRP list tag) and the VF ID and / or host ID. For example, the higher address bits of the address associated with PRP2 may not be used or reserved, which are replaced by the PRP list tag (when PRP2 is a list pointer) and the host ID. The NVMe switch can store a copy of the replaced bits. Then, at 214, the NVMe command with the modified PRP2 is submitted to the SSD for processing.

[0029] If the offset is not equal to zero, PRP2 may indicate that the list pointer in the host memory may also reference another list pointer. If another list pointer is referenced at 218, processing continues to block 220 to retrieve another list and perform functions similar to 206-218. If no list is referenced at 218, processing continues to blocks 212, 214.

[0030] Determining whether PRP2 is associated with another list pointer may be based on the I / O size and the offset. To illustrate, the list of data pointers may be no larger than 512 data pointers, where each data pointer is 8 bits, such that the list of pointers fits into a 4K buffer associated with PRP2. If each data pointer also points to 4K data, then 2MB of data is associated with PRP2. If the I / O size associated with the command is less than or equal to the sum of 2MB and the 4K associated with PRP1, then the list pointer in PRP2 may not point to another list pointer. If the I / O size is still greater than the non-zero offset, then the list pointer in PRP2 may point to another list pointer.

[0031] The SSD may process an NVMe command received from an NVMe switch. When processing an NVMe command, the SSD may send a PCIe memory request to the NVMe switch together with a PRP entry in the NVMe command. The PCIe memory request may be used to retrieve data associated with a data pointer indicated by a PRP entry, or to retrieve a data pointer associated with a list pointer indicated by a PRP entry. The data pointer or list pointer may point to a memory in a host associated with the NVMe command being processed. In addition, the PRP entry in the PCIe memory request may include a data tag or a PRP list tag and a VF ID and / or a host ID, the data tag or the PRP list tag being used to indicate whether the PRP entry is associated with a data pointer or a list of data pointers, the VF ID and / or the host ID being associated with the NVMe command being processed.

[0032] Figure 3 is an example flow diagram of the functions associated with handling a PCIe memory request from an SSD. The process may be performed by a data processor and may be implemented in hardware, software, firmware, or a combination thereof.

[0033] The PCIe memory request provided by the SSD may have a PRP entry. The PRP entry includes a data tag or a PRP list tag, and a host ID and / or a VF ID that was previously embedded in the PRP entry by the NVMe switch when the NVMe command associated with the PCIe memory request was sent to the SSD. This information can be used to route the PCIe memory request.

[0034] At 302, it is determined whether the PRP entry has a data tag in the address indicating that the PRP entry is associated with a data pointer.

[0035] If the PRP entry has a data tag, then the PRP entry is associated with the data pointer. The host is identified from the host ID or VF ID in the PRP entry, and the bits in the PRP entry that were previously replaced by the NVMe switch are reinserted back into the PRP entry. At 310, a PCIe memory request is routed to the host to perform a PCIe memory request to retrieve the data associated with the data pointer, which is then provided to the SSD in a PCIe memory response received from the host.

[0036] If the PRP entry has a PRP list tag, the PRP entry is associated with a list pointer. The host is identified from the host ID or VF ID in the PRP entry, and the bit in the PRP entry that was previously replaced by the NVMe switch is reinserted back into the PRP entry. At 304, the PCIe memory request is routed to the host so that the host executes the PCIe memory request to retrieve a list of data pointers associated with the list pointer. In some cases, the PRP entry may also include a VF ID, which is then used to further route the PCIe memory request to the VM or virtual host. At 306, the NVMe switch waits for a PCIe memory response to the PCIe memory request, which indicates that a list of pointers is received from the host. Upon receiving this completion, at 308, the pointers in the list of pointers are modified to include data tags, PRP list tags, and / or host IDs in a manner similar to that described in blocks 204, 212, and 216 above to indicate whether each pointer in the pointer points to data or a list of pointers and indicates the host associated with the data pointer or list pointer. In some cases, the PRP entry may include the VFID. The modified list of pointers is then provided to the SSD. When the SSD sends a PCIe memory request to access a pointer, the data tag, PRP list tag, host ID, and / or VF ID are used by the NVMe switch to route the pointer to the host associated with the pointer.

[0037] In some examples, the last entry of the pointer list may be a list pointer that points to a list of another data pointer. The data processor may check every 8 bytes or some other amount of data in the pointer list of the PCI response received from the host. If the data in the list of pointers is a list pointer, the pointer is marked with a PRP list tag. A VF ID and / or host ID is included in the pointer that is associated with the list from which the pointer was retrieved to facilitate the SSD's subsequent retrieval of the data pointed to by the pointer or an additional list of pointers.

[0038] Figure 4An example process for modifying a PRP entry associated with an NVMe command, and retrieving and modifying an associated data pointer and / or list pointer, which is retrieved from the host based on the PRP entry, is illustrated. The PRP entry may be indicated by an NVMe command 402 obtained from the host, in which example, the NVMe command 402 includes a PRP1 data pointer and a PRP2 list pointer. The command processor may modify the PRP1 entry and the PRP2 entry to indicate a tag identifying the host ID / VF ID ("RTNG" or routing) and whether the PRP entry is a data pointer ("DAT") or a list pointer ("PRP"). The modified NVMe command is shown as command 404. The SSD may send a PCIe memory request to the NVMe switch to obtain a list of pointers ("PRP list") from the host. The list of pointers 406 may be obtained by the data processor from the host and modified to indicate whether the pointer is associated with a data pointer ("DAT") or a list pointer ("PRP"). The pointers in the list may also be associated with the host ID and / or VF ID (“RTNG”) from which the list of pointers was obtained. The modified list of pointers is provided as list 408 .

[0039] In some examples, the lists of pointers 406, 408 may include an entry of a list pointer (shown as "Next PRP List Pointer") that points to another list pointer in the host memory. The data processor may detect the list pointer and retrieve a list of pointers associated with the list pointer, which is shown as list of pointers 410. The data processor may modify the pointer to indicate whether the pointer is associated with a data pointer ("DAT") or a list pointer ("PRP"). In addition, the data processor may associate the pointer with a host ID and / or a VF ID ("RTNG") and send the modified list of pointers 412 to the SSD. If the modified list of pointers 412 includes additional list pointers, the data processor may further retrieve the list of pointers associated with the additional list pointers upon request of the SSD in the manner described.

[0040] Example Device

[0041] Figure 5is a simplified block diagram of an NVMe switch 116. The NVMe switch 116 includes a switch processor 502 (which may include multiple processors, multiple cores, multiple nodes, and / or implement multithreading, etc.) that coordinates operations on the NVMe switch. The NVMe switch 116 includes a memory 504. The memory 504 can be a system memory (e.g., one or more of a cache, a random access memory (RAM), a synchronous RAM (SRAM), a dynamic RAM (DRAM), a zero capacitor RAM, a two-transistor RAM, an embedded DRAM (eDRAM), an extended data output RAM (EDO RAM), a double data rate RAM (DDR RAM), an electrically erasable programmable read-only memory (EEPROM), a Nano-RAM (NRAM), a resistive RAM (RRAM), a silicon-oxide-nitride-oxide-silicon memory (SONOS), a parameter random access memory (PRAM), etc.) or any one or more other possible implementations of a non-transitory machine-readable medium / media. In an example, the memory 504 may store a PRP entry 514 having a data tag or a PRP list tag and routing information in the form of a host ID and / or a VF ID associated with an NVMe command received from the host. The memory 504 may also store data associated with a data pointer retrieved from the host and a list of pointers retrieved from the host.

[0042] NVMe switch 116 also includes a bus 506 (e.g., an Advanced Scalable Bus AXI bus). Coupled to bus 506 is an interface 508 that facilitates communication with the non-volatile memory array of the SSD and the host. In this regard, interface 508 may include a PCIe EP cluster and a PCIe RC cluster that provide communication with the host and the SSD, respectively.

[0043] The command processor 516 may be implemented for any of the following, previously described functions: marking a PRP entry associated with an NVMe command using information associated with whether it is a data pointer or a list pointer. The PRP entry may also be modified using the host ID of the physical host that sent the NVMe command and / or the VF ID of the virtual host or VM. The data processor 518 may be implemented for any of the following, previously described functions: routing data pointers and / or list pointers to the host based on the host ID and / or VF ID in the modified PRP entry. These functions may be implemented in hardware and / or software (e.g., computer code, program instructions, program code, computer instructions) stored on a non-transitory machine-readable medium / media. In some cases, the processor 502 and the memory 504 may replace the command processor 510 and the data processor 512 or be in addition to the command processor 510 and the data processor 512 to implement the functions or help implement the functions. In addition, the implementation may include Figure 5 Fewer components or additional components not shown in the figure (e.g., video card, audio card, additional network interface, peripheral devices, etc.). Processor 502 and memory 504 are coupled to bus 506. Although illustrated as coupled to bus 506, memory 504 can be coupled to processor 502.

[0044] Some embodiments have been described in detail above, and various modifications are possible. The disclosed subject matter, including the functional operations described in this specification, can be implemented in electronic circuits, computer hardware, firmware, software, or a combination thereof, such as the structural components disclosed in this specification and their structural equivalents, including programs (such as programs encoded in computer-readable media, which can be storage devices, storage devices, machine-readable storage substrates or other physical, machine-readable media, or a combination of one or more of them) that are potentially operable to cause one or more data processing devices to perform the described operations.

[0045] A program (also referred to as a computer program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file (e.g., one or more scripts stored in a markup language document) that stores other programs or data, in a single file dedicated to the program in question, or in multiple coordination files (e.g., files storing one or more modules, subroutines or code portions). A program may be deployed to execute on a single computer or on multiple computers located at a station or distributed across multiple stations and interconnected by a communication network.

[0046] Although this specification contains many details, these details should not be interpreted as limitations on the scope of protection that can be claimed, but should be interpreted as descriptions of features specific to specific embodiments. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. In addition, although the features may be described above as acting in certain combinations, and even initially claimed as such, one or more features of the claimed combination may be cut out from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variant of the sub-combination.

[0047] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments.

[0048] Unless specifically stated otherwise, the use of the phrase "at least one of" with the conjunction "and" before a list should not be viewed as an exclusive list and should not be interpreted as a list of categories with one item from each category. A clause reciting "at least one of A, B, and C" may be violated by only one of the listed items, a plurality of the listed items, and one or more of the items in the list and another item not listed.

[0049] Other implementations are within the scope of the following claims.

Claims

1. A method comprising: Receiving a first command at a switch from a host, the host being associated with a storage system including two or more hosts, wherein the first command includes a first physical region page (PRP) entry and a second PRP entry, and the first command complies with a Non-Volatile Memory Express (NVMe) standard, wherein the first PRP entry includes a first data pointer, and wherein the second PRP entry includes one of a second data pointer and a list pointer; modifying the first PRP entry with an indication of the host sending the first command; determining content to be included in the second PRP entry based on whether the second PRP entry includes the second data pointer or the list pointer; modifying the second PRP entry to include the determined content; sending a second command from the switch to a solid-state drive (SSD), the second command having a plurality of entries including the modified first PRP entry and the modified second PRP entry; receiving, at the switch, a memory request from the SSD, wherein the memory request includes the plurality of PRP entries; as well as The memory request is routed to the host sending the first command based on the indication to the host in the modified second PRP entry in the memory request.

2. The method of claim 1, wherein receiving the memory request comprises receiving a peripheral computer interface express (PCIe) read request, the peripheral computer interface express (PCIe) read request comprising the plurality of PRP entries. 3 . The method of claim 1 , wherein the indication to the host is inserted into upper 2 octets of a PCIe address of the modified first PRP entry of the first command.

4. The method according to claim 1, further comprising: When routing the first command from the host, the indication of the host is determined based on metadata, wherein the host is a virtual host or a physical host.

5. The method of claim 1 , wherein modifying the first PRP entry with the indication to the host comprises: retaining in memory one or more bits in the first PRP entry for the first command, and The one or more bits in the first PRP entry of the first command are replaced with the indication to the host. The method of claim 1 , wherein the second PRP entry comprises a second data pointer.

7. The method according to claim 1, further comprising: The second PRP entry is modified with an indication of whether the second PRP entry includes the second data pointer or the list pointer.

8. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to at least perform: Receiving a first command from a host, the host being associated with a storage system including two or more hosts, wherein the first command includes a first physical region page (PRP) entry and a second PRP entry, and the first command is associated with a Non-Volatile Memory Express (NVMe) standard, wherein the first PRP entry includes a first data pointer, and wherein the second PRP entry includes one of a second data pointer and a list pointer; modifying the first PRP entry with an indication of the host sending the first command; determining content to be included in the second PRP entry based on whether the second PRP entry includes the second data pointer or the list pointer; modifying the second PRP entry to include the determined content; sending a second command to a solid-state drive (SSD), the second command having a plurality of PRP entries, wherein the plurality of entries includes a plurality of entries of the modified first PRP entry and the modified second PRP entry; receiving a memory request from the SSD, wherein the memory request includes the plurality of PRP entries; as well as The memory request is routed to the host sending the first command based on the indication to the host in the modified second PRP entry in the memory request.

9. The non-transitory computer readable medium of claim 8, wherein the instructions to receive the memory request include instructions to receive a peripheral computer interface express (PCIe) read request, the peripheral computer interface express (PCIe) read request including the plurality of PRP entries.

10. The non-transitory computer readable medium of claim 8, wherein the indication to the host is inserted into upper 2 octets of a PCIe address of the modified first PRP entry of the first command.

11. The non-transitory computer readable medium of claim 8, further comprising instructions for: determining the indication of the host based on metadata when routing the first command from the host, wherein the host is a virtual host or a physical host.

12. The non-transitory computer readable medium of claim 8, wherein the instructions to modify the first PRP entry with the indication to the host include instructions to: retaining in memory one or more bits in the first PRP entry for the first command, and The one or more bits in the first PRP entry of the first command are replaced with the indication to the host. 13 . The non-transitory computer-readable medium of claim 8 , wherein the second PRP entry comprises the second data pointer.

14. The non-transitory computer readable medium of claim 8, further comprising instructions to modify the second PRP entry with an indication of whether the second PRP entry includes the second data pointer or the list pointer.

15. A non-volatile memory high-speed NVMe switch, comprising: A command processor comprising instructions stored in a memory of the NVMe switch, the instructions, when executed by one or more processors of the NVMe switch, causing the NVMe switch to at least: receiving, by the command processor, a first command from a host, the host being associated with a storage system including two or more hosts, wherein the first command includes a first physical region page (PRP) entry and a second PRP entry, and the first command complies with the Non-Volatile Memory Express (NVMe) standard, wherein the first PRP entry includes a first data pointer, and wherein the second PRP entry includes one of a second data pointer and a list pointer; modifying, by the command processor, the first PRP entry with an indication of the host sending the first command; determining content to be included in the second PRP entry based on whether the second PRP entry includes the second data pointer or the list pointer; modifying the second PRP entry to include the determined content; as well as sending, by the command processor, a second command to a solid-state drive (SSD), the second command having a plurality of entries including the modified first PRP entry and the modified second PRP entry; as well as A data processor comprising instructions stored in a memory of the NVMe switch, and the instructions, when executed by one or more processors of the NVMe switch, cause the NVMe switch to at least: receiving, by the command processor, a memory request from the SSD, wherein the memory request includes the plurality of PRP entries; as well as The memory request is routed to the host sending the first command based on the indication to the host in the modified second PRP entry in the memory request.

16. The NVMe switch of claim 15, wherein the instruction to receive the memory request comprises an instruction to receive a peripheral computer interface express (PCIe) read request, the peripheral computer interface express (PCIe) read request comprising the plurality of PRP entries.

17. The NVMe switch of claim 15, wherein the indication to the host is inserted into upper 2 octets of a PCIe address of the modified first PRP entry with the first command, wherein the host is a virtual host or a physical host.

18. The NVMe switch of claim 15, wherein the instruction to modify the first PRP entry with the indication to the host comprises the following instruction: retaining in memory one or more bits in the first PRP entry for the first command, and The one or more bits in the first PRP entry of the first command are replaced with the indication to the host.

19. The NVMe switch of claim 15, wherein the second PRP entry comprises the second data pointer.

20. The NVMe switch of claim 15, further comprising instructions for modifying the second PRP entry with an indication of whether the second PRP entry includes the second data pointer or the list pointer.

Images