Peripheral Component Interconnect Express over Fabric Networks

Abstract

A system and associated method for statelessly communicating information between devices over a fabric network is described, including a processing circuit. The information may be received from a first device in the form of a plurality of packets and addressed to a second device of the plurality of devices using a device identifier, such as a bus:device:function (BDF) identifier. The processing circuit maps the device identifier of the second device to a unique device address. The processing circuit encapsulates each of the plurality of packets to generate a plurality of encapsulated packets. The processing circuit communicates each of the plurality of encapsulated packets over the fabric network. The processing circuit routes the plurality of encapsulated packets to the second device using the second device's unique device address.

Description

[Technical Field]

[0001] The present disclosure relates to computer systems, storage device systems, and methods for communicating over a fabric network, and more particularly to communicating statelessly over any type of fabric network using identifiers such as bus:device:function identifiers. Summary of the Invention

[0002] Distributed and configurable systems facilitate the sharing of distributed resources. Traditional systems are often configured with dedicated resources sized for worst-case conditions, increasing the space, cost, power, and cooling requirements of each system. With a fast, efficient, and scalable fabric or fabric network and associated communication architecture on the fabric, resource sharing can be advantageous. A stateless fabric communication architecture is more scalable than a stateful fabric communication architecture because dedicated resources are required to manage stateful communication. Thus, as a system with a stateful fabric communication architecture grows in size, additional dedicated resources are required to manage the increased stateful communication.

[0003] As devices such as central processing units (CPUs), data processing units (DPUs), graphics cards and graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and solid-state drives (SSDs) improve, the transmission and reception of information between different devices can become a limiting factor in system performance. For example, a first and second device may be able to process information faster than the information can be sent and received between the devices. Therefore, a faster communication architecture or protocol may be desirable. Other applications, such as cloud computing, real-time analytics, and artificial intelligence, may use devices in different physical locations, such as two different cities. The distance between the devices may limit approaches to increasing the speed or bandwidth of information transmission and reception.

[0004] In one approach, peripheral component interconnect express (PCIe) can be used as a high-speed standard bus interface for communication between a CPU and other devices (called communication devices), such as sound cards, video cards, Ethernet cards, redundant array of inexpensive disks (RAID) cards, and solid-state drives (SSDs). Each device is assigned a device identifier, such as a bus:device:function (BDF) identifier, and communicates using the device identifier. According to the PCIe 4.0 standard, PCIe can enable a device communicating with another device to transfer information at a bandwidth of up to 32 GB / s. However, PCIe as a transport does not define a protocol governing communication between the CPU and other devices within a system. PCIe is used internally within a system, typically in a data center computer, and cannot be used for devices external to the system (e.g., outside the data center computer).

[0005] In another approach, the non-volatile memory express (NVMe) communication protocol may be used to transfer information between devices, particularly between a host CPU and a PCIe-connected storage system such as a solid-state drive (SSD). Therefore, the NVMe protocol is designed for communication with storage devices directly connected to a local and dedicated PCIe bus. The NVMe protocol is designed for use locally on a computer's PCIe bus for high-speed data transfer between a host device and a storage system. The host device and the storage system are bound by input / output (I / O) queues used to manage the transfer of information. The I / O queues are located in the host device's memory, which can reduce the cost and complexity of the storage system. However, NVMe has limitations. The I / O queues can reduce the memory available to the host device to perform other operations. Because the I / O queues are located in the host device's memory, a storage system cannot be bound to or communicate with another host device. The NVMe protocol is not designed for use in a multi-host environment or for fabric connections between a host device and a storage subsystem. For example, the NVMe protocol is not designed to manage communication between a CPU in a first city and an SSD in a second city, because the SSD may need to connect directly to the CPU through a motherboard connection (e.g., a slot or expansion slot) without a cable. The SSD may also connect to the CPU using a PCIe cable, but the PCIe cable may need to be short in length (e.g., 15 inches (38.1 centimeters), 12 inches (30.48 centimeters), or 8 inches (20.32 centimeters)) to achieve high-speed communication.

[0006] Alternatively, PCIe can be used as a fabric network for communication between host devices and storage systems. PCIe fabric provides an extension of PCIe to increase the power of data center computers and can facilitate communication within a rack or across the data center. However, PCIe as a fabric does not provide a way for different host devices to communicate (e.g., CPU-to-CPU communication) or to share devices across a native PCIe fabric network. PCIe fabric does not define I / O queues like NVMe.

[0007] Another approach uses NVMe over Fabrics (NVMeoF) in conjunction with a PCIe bus to communicate between host devices and storage systems over a fabric network. The fabric network can house devices in different locations and can include traditional fabrics such as Ethernet, Fibre Channel, and InfiniBand. Because NVMeoF uses NVMe, the host device and storage system are bound to the I / O queue, as described above. The I / O queue resides in the storage system's controller rather than the host device, which means that the storage system's drives (e.g., SSDs) require available controllers and memory to manage the transfer of information. However, NVMeoF has limitations. Because NVMeoF is defined for use across traditional fabrics, a protocol conversion from PCIe / NVMe to the traditional fabric is required. Protocol conversion typically requires a store-and-forward approach to moving information, such as data in an NVMeoF exchange. Therefore, NVMeoF has scaling issues with some devices, such as storage bridges and Just A Bunch Of Flashes (JBOF) arrays, which include SSD arrays. Scaling issues arise from the need for a stateful system to track the progress of NVMeoF exchanges and store and forward data associated with those exchanges at the small computer system interface (SCSI) exchange level. Information communicated between the host device and the storage system may be received and assembled into a staging buffer. Performance may be degraded by the staging buffer.Scalability is limited by the CPU bandwidth required to manage the stateful exchanges, including the staging buffers, and by the memory space required to hold the data, because this level of store-and-forward creates bottlenecks in large systems, requiring more CPU power and buffer memory. These issues are most pronounced with traditional fabrics, primarily due to the protocol translation between PCIe and traditional fabrics. While PCIe fabrics can be used and are not hindered by the same protocol translation, NVMeoF itself can be hindered because NVMeoF was originally defined for traditional fabric networks.

[0008] NVMeoF may use remote direct memory access (RDMA) to communicate between each device's memory without involving the CPU. Memory-to-memory communication can result in lower latency and longer response times. For the initiator of an NVMe exchange, NVMeoF with RDMA may be less latency-intensive because the initiator already has the data for the exchange in memory, and modern interface controllers such as Ethernet intelligent network interface controllers (NICs) offload much of the stateful behavior. However, for devices such as storage bridges or JBOFs that contain many SSDs and connect to many initiators, NVMeoF may be more latency-intensive. The number of simultaneous exchanges can be very large and is typically limited by the memory and CPU resources available to the storage system controller. RDMA itself is not a standard transmission control protocol (TCP) / internet protocol (IP), which may make it undesirable. NVMeoF may also be limited by TCP / IP when used over an Ethernet fabric network. TCP / IP may require more computing power from the storage system because a checksum may be calculated for each packet communicated. TCP / IP may maintain and transmit multiple copies of data to avoid packet loss at the routing level, which may result in higher latency than other NVMeoF protocols. TCP / IP may require acknowledgment packets in response to information packets, which may result in higher latency than other NVMeoF protocols.

[0009] Therefore, there is a need for a high-speed communication architecture between devices connected to a fabric network that addresses these problems and limitations. Such a solution uses a fabric network to communicate statelessly over existing fabric networks such as Ethernet, Fibre Channel, and InfiniBand, leveraging existing protocols and interfaces, such as PCIe.

[0010] To solve these problems, systems and methods are provided herein for mapping device identifiers of devices to unique addresses for communication between devices over a fabric network, where the unique addresses can be device addresses such as physical addresses or fabric addresses. [Brief explanation of the drawings]

[0011] The following description includes a discussion of figures having illustrations given as examples of implementations of embodiments of the present disclosure. The figures should be understood as examples, not limitations. As used herein, reference to one or more "embodiments" should be understood as describing particular features, structures, and / or characteristics included in at least one implementation. Thus, phrases such as "in one embodiment" or "in an alternative embodiment" appearing herein describe various embodiments and implementations, not necessarily all referring to the same embodiment, but which are not necessarily mutually exclusive.

[0012] [Figure 1] 1 illustrates an example diagram of a system for communicating information between devices over a fabric network, according to some embodiments of the present disclosure.

[0013] [Figure 2A] 1 illustrates an example diagram of a system for communicating information between devices, including subsystems of the devices, over a fabric network, in accordance with some embodiments of the present disclosure.

[0014] [Figure 2B] 2B shows an example diagram of multiple packets communicated between the devices of FIG. 2A in accordance with some embodiments of the present disclosure.

[0015] [Figure 3] 1 illustrates an example diagram of information communicated between devices using input / output (I / O) queues, according to some embodiments of the present disclosure.

[0016] [Figure 4] FIG. 10 illustrates another example diagram of information communicated between devices using I / O queues, according to some embodiments of the present disclosure.

[0017] [Figure 5] 1 illustrates a flowchart for communicating information over a fabric network according to some embodiments of the present disclosure.

[0018] [Figure 6] 1 illustrates an example of system processing circuitry, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019] According to the present disclosure, systems and methods are provided for improving communication over a fabric network, and in particular for providing stateless communication between devices over a fabric network. In one approach, information may be received from a first device, such as by a processing circuit. The information may be in the form of multiple packets, such as PCIe packets, and may be received through a Peripheral Component Interconnect Express (PCIe) bus interface. The received packets may be addressed to a second device using a device identifier of the second device. The device identifier is used to identify a specific device (e.g., the second device). The device identifier may be an enumerated identifier assigned based on a query to a connected device, such as by sending a request inquiring about the presence of the device in a slot of the processing circuit and receiving an acknowledgment from the device connected to the slot. The device identifier may be assigned by the device manufacturer or supplier or may be hard-coded. The device identifier may be assigned based on the device's function, such that a single physical device may have a device identifier for each function it performs. Alternatively, a device may have a single device identifier for multiple functions and internally route received information to the appropriate function. The device identifier may be a slot identifier, such that the device identifier is based on the slot to which the device is connected. In some embodiments, the device identifier may be a bus:device:function (BDF) identifier, which is used as an example for discussion below. However, it will be understood that other identifiers, such as the examples above, may be used. For example, in some embodiments, the device identifier is not limited to including bus, device, and function components; for example, the device identifier may be modified by rearranging, changing, adding, and / or removing components.

[0020] The first device and the second device may be connected by a fabric network, such as Ethernet, Fibre Channel, or InfiniBand. A plurality of packets may be transmitted to the second device via the fabric network. The processing circuitry may encapsulate each packet of the plurality of packets before transmitting it via the fabric network. The encapsulated packets may be decapsulated by the processing circuitry before transmitting it to the second device, and the decapsulated packets may be transmitted using a PCIe bus interface. The encapsulated packets may be transmitted and received via the fabric network according to a particular protocol. The fabric protocol may require transmitting information using a unique device address of the second device, which may be different from the BDF identifier. The BDF identifier may be mapped to the unique device address, and the unique device address is used to communicate information over the network.

[0021] Each of the multiple packets may be encapsulated using a unique device address of the second device and communicated over the fabric network. Multiple packets may result in faster transfer speeds because there is no need to translate entire transactions, such as small computer system interface (SCSI) transactions or non-volatile memory express (NVMe) transactions. The encapsulation does not require additional state information to link or associate individual packets of the multiple packets. Mapping BDF identifiers to unique device addresses may allow the encapsulated packets to be communicated and flow statelessly between the first and second devices. Scalability is possible because no CPU bandwidth is required from the first and second devices, and no memory space is required to hold information from the first and second devices. There are no stateful exchanges to manage and no store-and-forward to implement. The encapsulated packets flow between devices without staging buffers. No fabric protocol translation, such as RDMA over TCP / IP over Ethernet, is required.

[0022] In another approach, a first device may communicate with multiple second devices over a fabric network. The first device and each second device may have a BDF identifier and may use the BDF identifier to address information communicated to another device. The BDF identifier may be resolved into a unique device address according to the fabric network for transmitting information over the fabric network.

[0023] In another approach, the I / O queue is located in the memory of the processing circuit. For example, the I / O queue may reside in the processing circuit associated with the second device. Locating the I / O queue in the processing circuit may free up memory in the first and second devices for processing other tasks. The I / O queue may allow multiple devices to connect to the second device.

[0024] The term "communicate" and variations thereof, unless expressly specified otherwise, may include transferring information, sending information, and receiving information.

[0025] The term "information" and variations thereof may include, to name a few, data, payload, header, footer, metadata, PCIe transaction layer protocol packets (TLPs), PCIe data link layer packets (DLLPs), bits, bytes, and datagrams, unless expressly specified otherwise.

[0026] In some embodiments, the systems and methods of the present disclosure may refer to an SSD storage system that may include an SSD pipeline accelerator and storage controller, or a pipeline processor and network controller for a transport layer protocol (e.g., PCIe).

[0027] An SSD is a data storage device that uses integrated circuit assemblies as memory to persistently store data. SSDs have no moving mechanical parts, which distinguishes them from traditional electromechanical magnetic disks, such as hard disk drives (HDDs) or floppy disks, which contain rotating disks and moving read / write heads. Compared to electromechanical disks, SSDs are typically more resistant to physical shock, operate quieter, and have faster access times and less latency.

[0028] Many types of SSDs use NAND-based flash memory, which retains data without power and includes some type of non-volatile storage technology. SSD Quality of Service (QoS) can refer to predictably low latency and consistently high input / output operations per second (IOPS) while processing read / write input / output (I / O) workloads. This means that latency or I / O command completion times must be within a specific range without unexpected outliers. Throughput or I / O speeds may also need to be tightly controlled without causing sudden drops in performance levels.

[0029] In some embodiments, the systems and methods of the present disclosure may refer to an HDD storage system that may include an HDD controller and a network controller for a transport layer protocol (eg, PCIe).

[0030] 1 illustrates an example diagram of a system 100 for communicating information between devices over a fabric network 112, according to some embodiments of the present disclosure. A first device 102 communicates with a second device 104.

[0031] The system 100 includes processing circuits, such as a first processing circuit 110A and a second processing circuit 110B. The first processing circuit 110A is part of a first device 102, and the second processing circuit 110B is part of a second device 104. The first device 102 and the second device 104 communicate with other devices over a fabric network 112 through the first processing circuit 110A and the second processing circuit 110B. In the illustrated embodiment, the first processing circuit 110A includes an initiator 103, such as a CPU, and a first PCIe bridge 111A. The second processing circuit 110B includes a second PCIe bridge 111B. The first PCIe bridge 111A may receive information from the initiator 103 and communicate the information to the second PCIe bridge 111B. The information is addressed to a device identifier, such as a bus:device:function (BDF) identifier, of a second device 104, such as a target 105. The target 105 may be memory or storage of the second device 104. The first processing circuit 110A communicates the information to the second device 104 over a fabric network 112, which may be Ethernet, Fibre Channel (FC), or InfiniBand, to name a few. The first and / or second processing circuits 110A and 110B map the BDF identifier of the target 105 to a unique device address corresponding to the fabric network 112. For example, if the fabric network 112 is an Ethernet network, the unique device address may be an Internet Protocol (IP) address.

[0032] The second processing circuit 110B may similarly use the BDF identifier of the first device 102, such as the first PCIe bridge 111A, to communicate information from the second device 104 to the first device 102 over the fabric network 112. Once the BDF identifier is mapped to a unique device address, information may flow between the first device 102 and the second device 104 without managing an information exchange or staging buffer.

[0033] In some embodiments, the first and second PCIe bridges 111A and 111B may be PCIe chips.

[0034] In some embodiments, the first device 102 and the second device 104 may each be part of a fabric node (e.g., a node of the fabric network 112). In some embodiments, the first device 102 and the second device 104 may each be a fabric node.

[0035] In some embodiments, the first device 102 may be a host device, and the second device 104 may be a storage device. In some embodiments, the first device 102 may be a first host device, and the second device 104 may be a second host device. The first and second processing circuits 110A and 110B may enable communication between the host devices over the fabric network 112. In such embodiments, the target 105 may be the initiator 105 of the second device 104. The initiators 103 and 105 may result in conflicting BDF identifiers (e.g., both may be associated with 0:0:0, such as through the first and second processing circuits 110A and 110B), which may be resolved as discussed in connection with FIG. 2A . In one example, the BDF identifier of the first processing circuit 110A may be translated to 1:0:0, and the BDF identifier of the second processing circuit 110B may be translated to 2:0:0.

[0036] FIG. 2A illustrates an exemplary diagram of a system 200 for communicating information (e.g., information 230 of FIG. 2B ) between devices, including subsystems of devices, via a fabric network 212, in accordance with some embodiments of the present disclosure. Specifically, the system 200 of FIG. 2A may communicate information between a first device (e.g., a host device 202) and a subsystem of devices (e.g., a storage array 204). The storage array 204 includes a second device (e.g., a first SSD 206A), a third device (e.g., a second SSD 206B), and a fourth device (e.g., a third SSD 206C), collectively referred to as SSDs 206A-C. While three SSD devices (206A, 206B, and 206C) are shown in FIG. 2A , any suitable number of SSD devices may be used in some embodiments.

[0037] The system 200 includes a first processing circuit 210A and a second processing circuit 210B that communicate via a fabric network 212. In the illustrated embodiment, the first processing circuit 210A includes an initiator 203, such as a CPU, and a first PCIe bridge 211A. The second processing circuit 210B includes a second PCIe bridge 211B, a storage controller 207, and a third PCIe bridge 211C. The first PCIe bridge 211A receives information from the initiator 203 and communicates the information to the second PCIe bridge 211B. The second PCIe bridge communicates the information to the third PCIe bridge 211C through the storage controller 207. The storage controller 207 may process data services for the SSDs 206A-C. The third PCIe bridge 211C communicates information to the storage array 204, specifically, to the SSDs 206A-C.

[0038] Each of the SSDs 206A-C may have a BDF identifier and may communicate with each other using the BDF identifier. The initiator 203 may connect to the first PCIe bridge 211A through the first PCIe bus 220. The first SSD 206A, the second SSD 206B, and the third SSD 206C may connect to the second processing circuit 210B, specifically the third PCIe bridge 211C, through the second PCIe bus 226A, the third PCIe bus 226B, and the fourth PCIe bus 226C, respectively. The first processing circuit 210A maps the first PCIe bus 220 to a first unique device address 222 associated with the host device 202. The second processing circuit 210B maps the second PCIe bus 226A, the third PCIe bus 226B, and the fourth PCIe bus 226C to a second unique device address 224A, a third unique device address 224B, and a fourth unique device address 224C, respectively. The second, third, and fourth unique device addresses 224A, 224B, and 224C are associated with the first, second, and third SSDs 206A, 206B, and 206C, respectively. The unique device addresses 222 and 224A-C may be used to communicate and route information over the fabric network 212. The fabric network 212 may be any fabric network, such as Ethernet, FC, or InfiniBand, to name a few.

[0039] The initiator 203 may discover devices that can have a BDF identifier. When the system 200 is initialized, the initiator 203 may probe the hierarchy of all devices connected to the system 200 and discover a first processing circuit 210A, including a first PCIe bridge device 211A, that provides a path to a subset of the hierarchy. The initiator 203 configures the first processing circuit 210A as a bridge and assigns it a bus number of the BDF identifier. Each device connected to the system 200 may have a PCIe interface (e.g., a PCIe bridge or PCIe chip) that responds to probe inquiries and identifies downstream devices connected to the PCIe interface. For example, the host device 202 may have a PCIe chip as a root complex. The initiator 203 may initialize the PCIe chip of the host device 202 and enumerate it with a BDF of 0:0:0, where the first "0" is the bus number of the BDF identifier. Downstream devices connected to the PCIe chip of host device 202 may be enumerated with different device and function numbers, but the bus number will be the same (i.e., 0). Downstream device types may include other bridges with connected devices assigned different bus numbers. Enumeration continues through the hierarchy of downstream devices. For example, the second PCIe bridge 211B may be assigned a BDF of 1:0:0, and the third PCIe bridge 211C may be assigned a BDF of 2:0:0. The first, second, and third SSDs 206A, 206B, and 206C may be assigned BDFs of 2:1:0, 2:2:0, and 2:3:0, respectively.

[0040] Initiator 203 probes to identify other PCIe chips in other devices. Each PCIe chip is enumerated with a different bus number (e.g., 1, 2, etc.), and downstream devices are enumerated with different device and function numbers associated with the PCIe chip's bus number. Initiator 203 may probe through fabric network 212 to identify devices connected through fabric network 212 (e.g., PCIe bridges 211B and 211C and SSDs 206A-C). Each PCIe chip connected to fabric network 212 may have an independent peripheral component interconnect (PCI) domain and may be enumerated by initiator 203 with different bus numbers. Bus number conflicts may occur, such as when there are multiple host devices 202 with independent domains (e.g., host devices 350A-C in FIG. 3). In a PCIe network, independent PCI domains may be addressed using a non-transparent bridge (NTB), which may be used to interconnect the independent PCI domains. The NTB may perform BDF translation to address conflicting bus numbers between domains. In some embodiments, the first and second processing circuits 210A and 210B may be NTBs that perform BDF translation. In some embodiments, either the first processing circuit 210A or the second processing circuit 210B may be a PCIe over Fabric (NVMe over Fabrics (PCIeoF)) bridge. In some embodiments, PCIeoF bridges of independent PCI domains may need to communicate with each other to resolve address translation. In some embodiments, a multicast address recognized by all devices may be used, allowing the fabric network 212 to deterministically locate participating devices. The multicast address may be used in the Ethernet fabric network 212. In some embodiments, each node in the fabric network 212 may be registered with a “name server.”A designator may be added to the name server so that all devices are recognized. Name servers may be used in FC fabric networks 212. Ethernet and InfiniBand fabric networks 212 may use a similar approach with name servers.

[0041] The first processing circuit 210A and the second processing circuit 210B may each include a bridge chip. The bridge chip may be used for conversion or translation between BDF identifiers and unique device addresses. The bridge chip may be used to encapsulate and decapsulate multiple packets 234.

[0042] In some embodiments, the second processing circuit 210B may be used to discover devices having a BDF identifier.

[0043] In some embodiments, the unique device address is a media access control (MAC) address. In some embodiments, the BDF identifier is mapped to the MAC address. In one embodiment, the BDF identifier is used as the last three bytes of the MAC address. The MAC address may be used in the Ethernet fabric network 212.

[0044] In some embodiments, the unique device address is an IP address. In some embodiments, the BDF identifier is mapped to an IP address. In one embodiment, the BDF identifier is used as three bytes of the IP address. IP addresses can be used on Ethernet and InfiniBand fabric networks 212.

[0045] In some embodiments, the unique device address is a 24-bit FC identifier. In some embodiments, a BDF identifier is mapped to a 24-bit FC identifier. In one embodiment, the BDF identifier is used as the 24-bit FC identifier. The FC identifier may be used by FC fabric network 212.

[0046] In some embodiments, the unique device address is a local identifier (LID). In some embodiments, a BDF identifier is mapped to the LID. In one embodiment, the BDF identifier is used as the LID. The LID may be used in the InfiniBand fabric network 212.

[0047] In the illustrated embodiment, the connections between first PCIe bus 220, first unique device address 222, the lines shown through fabric network 212, second through fourth unique device addresses 224A-C, and second through fourth PCIe buses 226A-C are each shown as a single line, but each connection may include multiple lines or lanes. In some embodiments, first PCIe bus 220, first unique device address 222, and the connections through fabric network 212 may include a line for each endpoint (e.g., SSDs 206A-C) connected to host device 202. In some embodiments, the number of lines per each connection may depend on the amount of lanes in the PCIe slots of host device 202 or SSDs 206A-C. For example, a line may be laid for each lane.

[0048] 2B shows an example diagram of multiple packets 234 communicated between the devices of FIG. 2A, in accordance with some embodiments of the present disclosure. In the embodiment shown in FIG. 2B, the host device 202 sends information 230 to the first SSD 206A. The information 230 may include data, a header, and a PCIe TLP or Data Link Layer Packet (DLLP), to name a few.

[0049] The initiator 203 communicates information 230 using the BDF identifier of the first SSD 206A. In the illustrated embodiment, the information 230 includes a plurality of packets 234. The initiator 203 transmits the information 230 to the first processing circuit 210A over the first PCIe bus 220. The first processing circuit 210A encapsulates each of the plurality of packets 234 to generate a plurality of encapsulated packets 236. The first processing circuit 210A transmits each of the plurality of encapsulated packets 236 to the second processing circuit 210B over the fabric network 212 using a unique device address of the second processing circuit 210B (e.g., the second unique device address 224A in FIG. 2A ). The second processing circuit 210B decapsulates each of the plurality of encapsulated packets 236 to generate a plurality of packets 234. The second processing circuit 210B transmits the plurality of packets 234 to the first SSD 206A over the second PCIe bus 226A. In some embodiments, the first SSD 206A may decapsulate each of the plurality of encapsulated packets 236 on behalf of the second processing circuit 210B.

[0050] In some embodiments, each of the multiple packets is 2 kilobytes (KB) or less, such as 1 KB or less, such as 1.5 KB or less. The Ethernet fabric network 212 may have frames that can accommodate a payload of up to 1.5 KB bytes. In some embodiments, Ethernet may use jumbo frames that can accommodate a payload of up to 9 KB bytes. The FC fabric network 212 may accommodate a payload of up to 2 KB bytes. The InfiniBand fabric network 212 may accommodate a payload of up to 4 KB bytes.

[0051] In some embodiments, the plurality of packets 234 may be PCIe packets. In some embodiments, the plurality of packets 234 may be encapsulated as TCP / IP / Ethernet (TIE) packets. In some embodiments, the plurality of packets 234 may be encapsulated as user datagram protocol (UDP) / IP / Ethernet (UIE) packets. TIE and UIE packets may be used within the Ethernet fabric network 212. UIE packets are preferred because they lack state information such as acknowledgments and checksums. In some embodiments, the plurality of packets 234 may be encapsulated as FC packets. FC packets may be used within the FC fabric network 212. FC packets may be class 1, 2, or 3 packets. FC class 3 packets are preferred because they lack state information such as acknowledgments and checksums. In some embodiments, the plurality of packets 234 may be encapsulated as InfiniBand packets. InfiniBand packets may be used within the InfiniBand fabric network 212.

[0052] In stateless communication embodiments, the concept of packets may be unnecessary, and each individual packet within the plurality of packets may be considered an atomic unit of communication across the fabric network 212.

[0053] In some embodiments, information 230 may be transmitted between host device 202 and other devices, such as second SSD 206B and / or third SSD 206C. In some embodiments, the other devices may not be SSDs. For example, host device 202 may communicate with a central processing unit (CPU), a data processing unit (DPU), a graphics card and graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a sound card, an Ethernet card, and a redundant array of inexpensive disks (RAID) card, to name a few. In such embodiments, second processing circuit 210B may be connected to or reside within the other devices.

[0054] 2B assumes that first and second processing circuits 210A and 210B are used to transmit information 230 from host device 202 to first SSD 206A, but in some embodiments, this process may be reversed. First SSD device 206A may transmit information 230 to host device 202. Second processing circuit 210B may generate multiple packets 234 and multiple encapsulated packets 236. Second processing circuit 210B may transmit encapsulated packet 236 to first processing circuit 210A. First processing circuit 210A may decapsulate encapsulated packet 236 to generate multiple packets 234 before transmitting to host device 202.

[0055] In some embodiments, processing circuits 210A and 210B each include a chip, such as a bridge chip, to facilitate communication over fabric network 212. The bridge chip may be used to encapsulate and decapsulate packets 234. The bridge chip may perform the translation or conversion between BDF identifiers and unique device addresses. For example, the bridge chip may perform the translation after receiving packets 234 from PCIe bus 220 or 226A and before communicating the encapsulated packets 236 over fabric network 212, or after receiving encapsulated packets 236 from fabric network 212 and before sending the packets 234 to PCIe bus 220 or 226A.

[0056] 3 shows an example diagram of information (e.g., information 230 of FIG. 2B) communicated between devices using input / output (I / O) queues 340 according to some embodiments of the present disclosure. The devices may include host devices (e.g., a first host 350A, a second host 350B, and up to an "mth" host 350C, etc.) and storage devices (e.g., a first SSD 352A, a second SSD 352B, and up to an "nth" SSD 352C, etc.).

[0057] The I / O queue 340 resides in the memory 338 of the processing circuit 310, which may be similar to the second processing circuit 210B discussed in connection with FIGS. 2A and 2B. The I / O queue 340 includes multiple queue pairs 342. Each of the queue pairs 342 includes a send queue (SQ) 344 and a completion queue (CQ) 346. Each of the host devices 350A-C and storage devices 352A-C is bound to a queue pair 342. The I / O queues 340 may be assigned to the storage devices 352A-C using an "admin" command from the host devices 350A-C. The processing circuit 310 may complete the creation of the I / O queues 340 by providing a local 64-bit PCIe address for each I / O queue 340 in response to the administrative command from the host devices 350A-C. In the illustrated embodiment, each of storage devices 352A-C has a quantity of queue pairs 342 equal to the quantity of host devices 350A-C, i.e., "m" hosts. The quantity of queue pairs 342 allows each of host devices 350A-C to communicate with each of storage devices 350A-C. Once I / O queues 340 are established, information can flow between hosts and storage devices without requiring host and storage device CPU bandwidth to manage information exchanges or staging buffers.

[0058] "I / O queue" 340 is an NVMe structure, not a PCIe structure. I / O queues are used to send and complete NVMe commands. An NVMe command describes the information to be transferred for the command, including the length and location of the information. When a host writes a command to an I / O queue across the fabric, it is done by sending and receiving multiple PCIe packets. In the present invention, these PCIe packets are addressed, encapsulated, sent, received, de-encapsulated, and transferred onto the destination PCIe bus just like any other packet.

[0059] The first host 350A may communicate with the first SSD 352A by writing a command as an entry (called an SQ entry) to the SQ 344. This command describes the information to be transferred between the first host 350A and the first SSD 352A. As discussed in FIG. 2B, the information 230 may be sent in a packet (e.g., packet 234 or encapsulated packet 236). The first SSD 352A obtains the command from the SQ 344 and initiates an information transfer request to send or receive the information 230. Once all the information 230 has been transferred, the first SSD 352A writes an entry (called a CQ entry) to the CQ 346 to indicate that the command associated with the SQ entry is complete and the information has been transferred. The first host 350A processes the CQ entry. The host may write to a doorbell register (not shown) to indicate that a new command has been written to the SQ 344. The first SSD 352A may write the doorbell register to notify the CQ entry, such as after the information 230 is transferred.

[0060] Each queue in the I / O queues 340 has a queue identifier. The queue identifier for each SQ 344 is not explicitly specified in the NVMe command. The queue identifier for each SQ 344 can be inferred from the SQ 344 in which the queue identifier is entered. A doorbell register can be accessed via a PCIe address, and the associated SQ identifier for the doorbell register can be inferred. The SQ identifiers are virtualized, with one value exposed to the first host 350A and a potentially different value exposed to the first SSD 352A. The CQ 346 has a queue identifier. The processing circuit 310 can intercept the completion of the I / O command and modify the CQ identifier before passing the modified CQ identifier to the first host 350A. The CQ identifiers for the “abort” process and “get error log” commands are exceptions to the CQ modification because each of these SQ identifiers is explicitly specified and must be properly mapped before being sent to the first host 350A.

[0061] Although the communication between the first host 350A and the first SSD 352A has been discussed, the communication described above can occur between any of the host devices 350A - C and the storage devices 350A - C.

[0062] In some embodiments, the number of storage devices 352A - C is greater than the number of host devices (i.e., n > m), or vice versa (i.e., n < m). In some embodiments, the amount of storage devices 352A - C and host devices 350A - C is the same (i.e., n = m). In some embodiments, the amount of queue pairs 342 may not be based on the total amount of host devices 350A - C. For example, some of the storage devices 352A - C may not be connected to all of the host devices 350A - C.

[0063] In some embodiments, the processing circuit 310 can be similar to the processing circuit 110 discussed in relation to FIG. 1. In some embodiments, the processing circuit 310 can be similar to the first processing circuit 210A discussed in relation to FIGS. 2A and 2B. In some embodiments, the processing circuit 310 can be similar to the first and second processing circuits 210A and 210B discussed in relation to FIGS. 2A and 2B. In some embodiments, the processing circuit 310 can be similar to a storage control subsystem as discussed in relation to FIG. 2A.

[0064] FIG. 4 shows another exemplary diagram of information (e.g., information 230 of FIG. 2B) communicated between devices using an I / O queue 440, according to some embodiments of the present disclosure. The devices can include a first host 350A and a second host 350B (collectively referred to as host devices 350A and 350B), and a first SSD 352A and a second SSD 352B (collectively referred to as storage devices 352A and 352B).

[0065] The processing circuit 410 includes a bridge chip 437, memory 438, and circuit logic 439. The circuit logic 429 may include a controller, a central processing unit (CPU) 439, or code, to name a few. The circuit logic 439 may discover the first and second SSDs 352A and 352B and set up an I / O queue 440. The I / O queue 440 resides in memory 438 and includes an I / O queue pair 442 and an I / O queue group 443. The I / O queue pair 442 includes an SQ 444 and a CQ 446. The I / O queue group 443 includes an SQ 444, a first CQ 466A, and a second CQ 466B. The I / O queue 440 functions similarly to the I / O queue 340 discussed in connection with FIG. 3, unless otherwise noted. The first CQ 466A is specific to a storage device (e.g., the first SSD 352A or the second SSD 352B), and the second CQ 466B is specific to a corresponding host device (e.g., the first host 350A or the second host 350B). In one example, the first SSD 352A writes an entry to the first CQ 446A (called a storage device CQ entry). The CPU 439 processes and translates the storage device CQ entry and moves the entry to the second CQ 446B (called a host device CQ entry). The first host 350A processes the host device CQ entry. The host devices 350A and 350B may use doorbell registers as discussed in connection with FIG. 3. The first SSD 352A and the second SSD 352B may use doorbell registers as discussed in connection with FIG. 3.

[0066] In some embodiments, the processing circuit 410 may be part of or connected to a storage system, such as the storage array 204 discussed in connection with FIG. 2A. The first and second CQs 446A and 446B may be required in embodiments having the processing circuit 410 part of or connected to a host device and a storage device, such as the first and second processing circuits 210A and 210B discussed in connection with FIG. 2A. For example, the first CQ 446A may identify the host devices 350A and 350B using a different BDF identifier than the second CQ 446B, and the second CQ 446B may identify the storage devices 352A and 352B using a different BDF identifier than the first CQ 446A. In such embodiments, the circuit 439 may translate the BDF identifiers.

[0067] In some embodiments, the storage device's CQ conversion may be offloaded to field programmable gate array (FPGA) or application specific integrated circuit (ASIC) logic, and duplicate CQs 466A and 466B may not be necessary or may be converted without CPU intervention.

[0068] Although two host devices 350A and 350B and two storage devices 352A and 352B are shown in FIG. 4, other embodiments may use more or fewer host devices and storage devices.

[0069] 3 and 4 discuss communication between host devices 350A and 350B and storage devices 352A and 352B, in some embodiments, communication occurs between host devices 350A and 350B, such as between a CPU and any of the CPU, DPU, graphics card or GPU, FPGA, ASIC, and sound card, to name a few. In such embodiments, I / O queues 340 or 440 may be part of or connected to host devices 350A and 350B.

[0070] FIG. 5 illustrates a method 500 for communicating information (eg, information 230 of FIG. 2B) over a fabric network according to some embodiments of the present disclosure.

[0071] Method 500 begins with operation 502, in which a processing circuit (e.g., processing circuit 110, 210A and / or 210B, 310, or 410 of FIGS. 1, 2A and 2B, 3, and 4, respectively) receives a plurality of packets from a first device (e.g., first device 102 of FIG. 1, host device 202 of FIGS. 2A and 2B, and host devices 350A-C of FIGS. 3 and 4), as described above with respect to FIGS. 1-4. In some embodiments of method 500, the plurality of packets are addressed to a second device of the plurality of devices (e.g., second through fourth devices 106A-C of FIG. 1, first through third SSDs 206A-C of FIGS. 2A and 2B, and storage devices 352A-C of FIGS. 3 and 4) using a device identifier.

[0072] In operation 504, the processing circuitry maps the device identifier of the second device to a unique device address, as described above with respect to FIGS. 1-2B.

[0073] At operation 506, the processing circuit encapsulates each of the plurality of packets to generate a plurality of encapsulated packets (e.g., encapsulated packet 236 of FIG. 2B), as described above with respect to FIGS. 2B and 3.

[0074] At operation 508, the processing circuit communicates each of the plurality of encapsulated packets over the fabric network, as described above with respect to Figures 2B-4. In some embodiments of method 500, the plurality of encapsulated packets is routed to the second device using a unique device address of the second device.

[0075] In some embodiments, the device identifier of the second device is a bus:device:function identifier. In some embodiments, the first device is a host device and the second device is a storage device. In some embodiments, the second device is a just bunch of flash (JBOF) device. In some embodiments, the first device is a storage device and the second device is a host device.

[0076] Some embodiments further include receiving information from the second device. The information is addressed to the first device using the device identifier. Some embodiments further include mapping the device identifier of the first device to a unique device address. Some embodiments further include generating a plurality of packets from the information and encapsulating each of the plurality of packets to generate a plurality of encapsulated packets. Some embodiments further include communicating each of the plurality of encapsulated packets over a fabric network. Route the plurality of encapsulated packets to the first device using the unique device address of the first device.

[0077] In some embodiments, the first device and the second device are configured to use a Peripheral Component Interconnect Express (PCIe) bus interface to send and receive information.

[0078] Some embodiments further include establishing an input / output (I / O) queue pair (e.g., queue pair 342 and 442 in FIGS. 3 and 4, or in some embodiments, queue group in FIG. 4) and mapping the I / O queue pair to the first device and the second device.

[0079] In some embodiments, communicating each of the plurality of encapsulated packets over the fabric network is initiated by sending a command describing the information to an I / O queue pair.

[0080] In some embodiments, the packets are PCIe packets, and each of the packets is encapsulated as a plurality of UDP / IP / Ethernet (UIE) packets.

[0081] In some embodiments, the plurality of packets are PCIe packets, and each of the plurality of packets is encapsulated as a plurality of TCP / IP / Ethernet (TIE) packets.

[0082] In some embodiments, the plurality of packets are PCIe packets, and each of the plurality of packets is encapsulated as a plurality of Fibre Channel (FC) packets.

[0083] In some embodiments, the plurality of packets are PCIe packets, and each of the plurality of packets is encapsulated as a plurality of InfiniBand packets.

[0084] In some embodiments, each of the plurality of packets is 2 kilobytes (KB) or less.

[0085] In some embodiments, the unique device address is a media access control (MAC) address. Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the MAC address by using the device identifier as the least significant three bytes of the MAC address.

[0086] In some embodiments, the unique device address is an Internet Protocol (IP) address. Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the IP address by using the device identifier as three bytes of the IP address.

[0087] In some embodiments, the unique device address is a 24-bit Fibre Channel (FC) identifier. Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the 24-bit FC identifier by using the device identifier as the 24-bit FC identifier.

[0088] In some embodiments, the unique device address is a local identifier (LID). Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the LID by using the LID address of the device identifier.

[0089] It should be noted that FIG. 5 is merely one example of a method, and that other methods including fewer, additional, or alternative steps are possible consistent with this disclosure.

[0090] FIG. 6 illustrates an example of a system processing circuit 600 according to some embodiments of the present disclosure.

[0091] The system processing circuit 600 includes a first processing circuit 604 and a second processing circuit 654. The first processing circuit 604 connects to an I / O device 606 and a network interface 608. The first processing circuit 604 includes a controller, such as storage 610, memory 612, and a CPU 614. The CPU 614 may include any of the storage controller 207 discussed in connection with FIG. 2A and the circuit logic 429 discussed in connection with FIG. 4. The CPU 614 is configured to process computer-executable instructions stored, for example, in the memory 612 or the storage 610, to cause the system processing circuit 600 to perform the methods and processes described herein, for example, with respect to FIG. 5.

[0092] CPU 614 is shown as representative of a single CPU, multiple CPUs, a single CPU with multiple processing cores, and other types of processing architectures capable of executing computer-executable instructions.

[0093] The I / O device 606 includes a first device 616, which may include any of the first device 102 discussed in connection with FIG. 1, the host device 202 discussed in connection with FIGS. 2A and 2B, and the host devices 350A-C discussed in connection with FIGS. 3 and 4.

[0094] The network interface 608 provides the first processing circuit 604 with access to an external network, such as a fabric network 640. The bridge chip 437 discussed in connection with FIG. 4 may include the network interface 608. The fabric network 640 may include the fabric network 212 discussed in connection with FIGS. 2A and 2B. In some implementations, the network interface 608 may include one or more of a receiver, a transmitter, or a transceiver. The fabric network 640 may be a storage area network (SAN), a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a satellite communication network, and the like, to name a few, and may communicate according to Ethernet, FC, or InfiniBand protocols.

[0095] The second processing circuit 654 connects to the I / O device 656 and the network interface 658. The second processing circuit 654 includes storage 660, memory 662, and a processor, such as a CPU 664. The CPU 664 and the network interface 658 may be configured similarly to the CPU 614 and the network interface 608, respectively.

[0096] The I / O devices 656 include a second device 656, which may include any of the second to fourth devices 106A-C discussed in connection with FIG. 1, the SSDs 206A-C discussed in connection with FIGS. 2A and 2B, and the storage devices 352A-C discussed in connection with FIGS. 3 and 4.

[0097] The network interface 658 connects the second processing circuit 654 to the first processing circuit 604 through the fabric network 640, allowing the first and second devices 616 and 666 to communicate.

[0098] The terms "an embodiment," "embodiment," "embodiments," "the embodiment," "the embodiment," "the embodiment," "one or more embodiments," "some embodiments," and "one embodiment" mean "one or more (but not all) embodiments," unless expressly specified otherwise.

[0099] The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

[0100] Listed items do not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise.

[0101] The terms "a," "an," and "the" mean "one or more" unless expressly specified otherwise.

[0102] Devices that are in communication with each other need not be in continuous communication with each other unless explicitly specified otherwise. Furthermore, devices that are in communication with each other may communicate directly or indirectly through one or more intermediaries.

[0103] A description of an embodiment having multiple components in communication with each other does not imply that all such components are required. On the contrary, various optional components are described to illustrate the wide variety of possible embodiments. Furthermore, while process steps, method steps, algorithms, or the like may be described in a sequential manner, such processes, methods, and algorithms may be configured to operate in an alternating order. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. Steps of processes described herein may be performed in any order that is practical. Furthermore, some steps may be performed simultaneously.

[0104] Where a single device or article is described herein, it will be readily apparent that more than one device / article (whether they cooperate or not) may be used in place of the single device / article. Similarly, where more than one device or article (whether they cooperate or not) is described herein, it will be readily apparent that a single device / article may be used in place of the more than one device or article, or that a different number of devices / articles may be used in place of the number of devices or programs shown. The functionality and / or features of a device may alternatively be embodied by one or more other devices not explicitly described as having such functionality / features. Thus, other embodiments need not include the device itself.

[0105] At least some operations may be depicted in figures that show some events occurring in a certain order. In alternative embodiments, some operations may be performed in a different order, modified, or removed. Furthermore, steps may be added to the logic above and still be compatible with the described embodiment. Furthermore, operations described herein may occur sequentially, or some operations may be processed in parallel. Still further, operations may be performed by a single processing unit or by distributed processing units.

[0106] The foregoing description of various embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or to be limited to the precise form disclosed. Many modifications and variations are possible in light of the above teaching.

Claims

1. 1. A communication method comprising: receiving a first plurality of packets from a first device, wherein the first plurality of packets are addressed to a second device of the plurality of devices using a device identifier; mapping the device identifier of the second device to a unique device address; encapsulating each of the first plurality of packets to generate a first plurality of encapsulated packets; and communicating each of the first plurality of encapsulated packets over a fabric network, wherein the unique device address of the second device is used to route the first plurality of encapsulated packets to the second device; A method for providing the above.

2. 2. The method of claim 1, wherein the device identifier of the second device is a bus:device:function identifier, the first device is a host device, and the second device is a storage device.

3. The method of claim 2 , wherein the second device is a just bunch of flash (JBOF) device.

4. 2. The method of claim 1, wherein the device identifier of the second device is a bus:device:function identifier, the first device is a storage device, and the second device is a host device.

5. receiving a second plurality of packets from the second device, wherein the second plurality of packets are addressed to the first device using a device identifier; mapping the device identifier of the first device to a unique device address of the first device; encapsulating each of the second plurality of packets received from the second device to generate a second plurality of encapsulated packets; and communicating each of the second plurality of encapsulated packets over the fabric network, wherein the unique device address of the first device is used to route the second plurality of encapsulated packets to the first device; The method of claim 1 further comprising:

6. the device identifier of the second device is a bus:device:function identifier; and 10. The method of claim 1, wherein the first device and the second device are configured to use a Peripheral Component Interconnect Express (PCIe) bus interface for transmitting and receiving information.

7. The method of claim 1 , wherein each of the first plurality of encapsulated packets is communicated statelessly over the fabric network.

8. establishing an input / output (I / O) queue; and The method of claim 1 , further comprising mapping the I / O queue to the first device and the second device.

9. 9. The method of claim 8, wherein communicating each of the first plurality of encapsulated packets over the fabric network comprises initiating by writing a command to the I / O queue to send the first plurality of encapsulated packets to the second device.

10. the device identifier of the second device is a bus:device:function identifier; the first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and The method of claim 1 , wherein each of the first plurality of packets is encapsulated as a plurality of TCP / IP / Ethernet (TIE) packets.

11. the device identifier of the second device is a bus:device:function identifier; the first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and The method of claim 1 , wherein each of the first plurality of packets is encapsulated as a plurality of UDP / IP / Ethernet (UIE) packets.

12. the device identifier of the second device is a bus:device:function identifier; the first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and The method of claim 1 , wherein each of the first plurality of packets is encapsulated as a plurality of Fibre Channel (FC) packets.

13. the device identifier of the second device is a bus:device:function identifier; the first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and 2. The method of claim 1, wherein each of the first plurality of packets is encapsulated as a packet of a plurality of InfiniBand packets.

14. 2. The method of claim 1, wherein each of the first plurality of packets is 2 kilobytes (KB) or less.

15. the unique device address is a Media Access Control (MAC) address; and 2. The method of claim 1, wherein mapping the device identifier of the second device to the unique device address comprises mapping the device identifier of the second device to the MAC address by using the device identifier as the least significant three bytes of the MAC address.

16. the unique device address is an Internet Protocol (IP) address; and 2. The method of claim 1, wherein mapping the device identifier of the second device to the unique device address comprises mapping the device identifier of the second device to the IP address by using the device identifier as three bytes of the IP address.

17. the unique device address is a 24-bit Fibre Channel (FC) identifier; and 2. The method of claim 1, wherein mapping the device identifier of the second device to the unique device address comprises mapping the device identifier of the second device to the 24-bit FC identifier by using the device identifier as the 24-bit FC identifier.

18. The unique device address is a local identifier (LID); and 2. The method of claim 1, wherein mapping the device identifier of the second device to the unique device address comprises mapping the device identifier of the second device to the LID by using the device identifier as a LID address.

19. 1. A system comprising processing circuitry configured to perform a method, the method comprising: receiving information from a first device, wherein said information is addressed to a second device of the plurality of devices using a device identifier; mapping a BDF identifier of the second device to a unique device address; generating a plurality of packets from said information; encapsulating each of the plurality of packets to generate a plurality of encapsulated packets; and communicating each of the plurality of encapsulated packets over a fabric network, wherein the unique device address of the second device is used to route the plurality of encapsulated packets to the second device; A system comprising:

20. The processor receiving information from a first device, wherein said information is addressed to a second device of the plurality of devices using a device identifier; mapping a BDF identifier of the second device to a unique device address; generating a plurality of packets from said information; encapsulating each of the plurality of packets to generate a plurality of encapsulated packets; and communicating each of the plurality of encapsulated packets over a fabric network, wherein the unique device address of the second device is used to route the plurality of encapsulated packets to the second device; A computer program for executing

Images