Peripheral component interconnect high speed over fabric network

Abstract

Systems and related methods are described for stateless transfer of information between devices over a fabric network, the system including processing circuitry. Information may be received from a first device in the form of a plurality of packets and addressed to a second device of the plurality of devices using a device identifier, such as a bus: device: function (BDF) identifier. The processing circuitry maps a device identifier of the second device to a unique device address. The processing circuitry encapsulates each of the plurality of packets to generate a plurality of encapsulated packets. The processing circuitry communicates each of the plurality of encapsulated packets over the fabric network. The unique device address of the second device is used to route the plurality of encapsulated packets to the second device.

Description

Technical Field

[0001] The present disclosure relates to computer systems, storage device systems, and methods for communicating over a fabric network, and more particularly to using identifiers for stateless communication over any type of fabric network, such as bus:device:function identifiers. Summary of the Invention

[0002] Decoupled and composable systems facilitate the sharing of distributed resources. Traditional systems are often configured with dedicated resources that are sized for worst-case conditions, which increases the space, cost, power, and cooling requirements for each system. Given a fast, efficient, and scalable fabric or fabric network and an associated communication architecture over the fabric, sharing resources can be beneficial. Stateless fabric communication architectures are more scalable than stateful fabric communication architectures because dedicated resources are required to manage stateful communication. Thus, if the size of a system with a stateful fabric communication architecture increases, additional dedicated resources are required to manage the increased stateful communication.

[0003] As devices such as central processing units (CPUs), data processing units (DPUs), graphics cards and graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and solid state drives (SSDs) are improved, sending and receiving information between different devices can be a limiting factor in system performance. For example, a first device and a second device may be able to process information faster than they can send and receive information between each other. Thus, a faster communication architecture or protocol may be desirable. Other applications, such as cloud computing, real-time analytics, and artificial intelligence, may use devices in different physical locations (such as two different cities). The distance between the devices may limit the methods for increasing the speed or bandwidth of sending and receiving information.

[0004] In one approach, Peripheral Component Interconnect Express (PCIe) can be used as a high-speed standard bus interface for communicating between a CPU and other devices (referred to as devices in communication), such as sound cards, video cards, Ethernet cards, redundant array of inexpensive disks (RAID) cards, and solid state drives (SSDs). Each of the devices is assigned a device identifier, such as a bus:device:function (BDF) identifier, and communication is performed using that device identifier. Per PCIe 4.0 standard, PCIe can allow devices to transfer information with another device at a bandwidth of up to 32 GB / s. However, as a transport, PCIe does not define a protocol to manage communication between a CPU and devices in a separate system. PCIe is used within a system, typically a computer in a data center, and may not be used for devices external to the system (e.g., outside of a computer in a data center).

[0005] In another approach, the Non-Volatile Memory Express (NVMe) communication protocol can be used to transfer information between devices, and particularly between a host CPU and a PCIe-attached storage system such as a Solid State Drive (SSD). Thus, the NVMe protocol is designed to communicate with storage devices directly attached to a local dedicated PCIe bus. The NVMe protocol is designed for local use over a computer's PCIe bus for high-speed data transfer between a host device and a storage system. The host device and the storage system are bound to an Input / Output (I / O) queue for managing information transfer. The I / O queue is placed in the memory of the host device, which can reduce the cost and complexity of the storage system. However, NVMe has limitations. The I / O queue may reduce the memory available for the host device to perform other operations. Since the I / O queue is located in the memory of the host device, the storage system may not be bound to or communicate with another host device. The NVMe protocol is not designed to be used in a multi-host environment, nor is it designed for fabric connectivity between a host device and a storage subsystem. For example, the NVMe protocol is not designed to manage communication between a CPU in a first city and an SSD in a second city because the SSD may need to be directly connected to the CPU via a motherboard connection (e.g., a slot or an expansion slot) rather than a cable. The SSD can also be connected to the CPU using a PCIe cable, but the PCIe cable may require a short length (e.g., 15, 12, or 8 inches) to achieve high-speed communication.

[0006] In another approach, PCIe can be used as a fabric network for communication between a host device and a storage system. The PCIe fabric can extend PCIe beyond the computers in a data center to facilitate in-rack or cross-data center communication. However, PCIe as a fabric does not provide a means for communication between different host devices (e.g., CPU-to-CPU communication), nor does it provide a means for sharing devices across a local PCIe fabric network. The PCIe fabric does not define an I / O queue as NVMe does.

[0007] In another approach, the NVMe (NVMeoF) of the fabric can be used in conjunction with the PCIe bus to communicate between a host device and a storage system over the fabric network. The fabric network allows devices to be located in different locations and can include traditional fabrics such as Ethernet, Fibre Channel, and InfiniBand. Since NVMeoF uses NVMe, the host device and the storage system are bound to I / O queues as described above. The I / O queues are placed in the controller of the storage system rather than in the host device, which requires the drives of the storage system (e.g., SSDs) to have controllers and memories available for managing the transfer of information. However, NVMeoF has limitations. Since NVMeoF is defined to be used across traditional fabrics, protocol conversion from PCIe / NVMe to the traditional fabric is required. Protocol conversion typically requires a store-and-forward approach to move the information exchanged by NVMeoF, such as data. Therefore, NVMeoF has scalability issues in some devices such as storage bridges and just a bunch of flashes (JBOF), which includes arrays of SSDs. The scalability issues arise from the need to track the progress of NVMeoF exchanges in a stateful system and the need to store and forward the data associated with those exchanges at the Small Computer System Interface (SCSI) exchange level. The information passed between the host device and the storage system can be received and assembled into a staging buffer. Performance may be degraded by the staging buffer. Scalability is limited by the CPU bandwidth required to manage the stateful exchanges (including the staging buffer) and the memory space required to hold the data, as this level of store-and-forward can create bottlenecks in larger systems, requiring more CPU power and buffer memory. These issues are most notable in traditional fabrics, mostly due to the protocol conversion between PCIe and those traditional fabrics. The PCIe fabric can be used with the same protocol conversion and is not hindered by the same protocol conversion, but may be hindered by NVMeoF itself because NVMeoF was originally defined for traditional fabric networks.

[0008] NVMeoF can use Remote Direct Memory Access (RDMA) to communicate between the memories of each device without using the CPU. Memory-to-memory communication can reduce latency and increase response time. NVMeoF with RDMA may be easier for NVMe over Fabrics initiators because the initiators already have the data for the fabric in memory, and modern interface controllers (such as Ethernet Smart Network Interface Controllers (NICs)) offload much of the stateful work. However, NVMeoF can be difficult for devices such as storage bridges or JBOFs, which include many SSDs and are connected to many initiators. The number of concurrent fabrics can be very large and is typically limited by the available memory and CPU resources of the storage system controller. RDMA itself may be undesirable because it is not a standard Transmission Control Protocol (TCP) / Internet Protocol (IP). When using NVMeoF over an Ethernet fabric network, NVMeoF can also be hindered by TCP / IP. TCP / IP may require the computing power of the storage system because a checksum may be calculated for each packet transmitted. TCP / IP may impose more latency than other NVMeoF protocols because it may maintain and transmit multiple copies of data to bypass packet loss at the routing level. TCP / IP may introduce more latency than other NVMeoF protocols because it requires acknowledgment packets in response to information packets.

[0009] Accordingly, there is a need for a high-speed communication architecture between devices connected to a fabric network that addresses these problems and limitations. This solution uses a fabric network and leverages existing protocols and interfaces (such as PCIe) to communicate statelessly over existing fabric networks (such as Ethernet, Fibre Channel, and InfiniBand).

[0010] To address these problems, the present disclosure provides systems and methods for mapping a device identifier of a device to a unique address for communication between devices over a fabric network. The unique address can be a device address, such as a physical address or a fabric address. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following description includes a discussion of the figures, which have illustrations given by way of example of implementations of embodiments of the present disclosure. The figures should be understood by way of example and not by way of limitation. As used herein, reference to one or more "embodiments" should be understood to describe at least one implementation that includes a particular feature, structure, and / or characteristic. Thus, phrases such as "in one embodiment" or "in an alternative embodiment" that appear herein describe various embodiments and implementations and do not necessarily all refer to the same embodiment. However, they are not necessarily mutually exclusive either.

[0012] Figure 1 A schematic diagram showing a system for transferring information between devices through a fabric network according to some embodiments of the present disclosure;

[0013] Figure 2A A schematic diagram showing a system for transferring information between devices including device subsystems through a fabric network according to some embodiments of the present disclosure;

[0014] Figure 2B A schematic diagram showing according to some embodiments of the present disclosure the Figure 2A multiple packets transferred between devices;

[0015] Figure 3 A schematic diagram showing information transferred between devices using an input / output (I / O) queue according to some embodiments of the present disclosure;

[0016] Figure 4 An alternative schematic diagram showing information transferred between devices using an I / O queue according to some embodiments of the present disclosure;

[0017] Figure 5 A flowchart showing a process for transferring information through a fabric network according to some embodiments of the present disclosure; and

[0018] Figure 6 An example of a system processing circuit according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] According to the present disclosure, systems and methods are provided for improving communication through a fabric network, and in particular, for providing stateless communication between devices through a fabric network. In one method, information can be received from a first device, such as by a processing circuit. The information can be in the form of a plurality of packets, such as PCIe packets, and can be received via a peripheral component interconnect express (PCIe) bus interface. The plurality of received packets can be addressed to a second device using a device identifier of the second device. The device identifier is used to identify a specific device (e.g., the second device). The device identifier can be an enumerated identifier assigned based on querying the connected device, such as by sending a request to inquire whether the device exists on a slot of the processing circuit and receiving an acknowledgement from the device connected to the slot. The device identifier can be assigned or hard-coded by the manufacturer or vendor of the device. The device identifier can be assigned based on the function of the device, such that a single physical device can have a device identifier for each function it performs. Alternatively, the device can have a single device identifier for multiple functions and internally route the received information to the appropriate function. The device identifier can be a slot identifier, such that the device identifier is based on the slot to which the device is connected. In some embodiments, the device identifier can be a bus:device:function (BDF) identifier, which is used as an example in the discussion below. However, it will be understood that other identifiers can be used, such as the examples discussed previously. For example, in some embodiments, the device identifier is not limited to including components of a bus, device, and function; the device identifier can be modified by, for example, rearranging, changing, adding, and / or removing components.

[0020] The first device and the second device can be connected through a fabric network, such as Ethernet, Fibre Channel, or InfiniBand. The plurality of packets can be sent to the second device through the fabric network. The processing circuit can encapsulate each of the plurality of packets before sending them through the fabric network. Before being sent to the second device, the encapsulated packets can be decapsulated by the processing circuit, and the decapsulated packets can be sent using the PCIe bus interface. The encapsulated packets can be sent and received through the fabric network according to a specific protocol. The fabric protocol can require using a unique device address of the second device different from the BDF identifier to send information. The BDF identifier can be mapped to the unique device address, and the unique device address is used to convey information across the network.

[0021] Each of a plurality of packets can be encapsulated and transmitted over a fabric network using a unique device address of a second device. The plurality of packets can result in faster transfer speeds because there is no need to translate an entire transaction, such as a Small Computer System Interface (SCSI) transaction or a Non-Volatile Memory Express (NVMe) transaction. Encapsulation does not require additional state information to link or associate individual packets among the plurality of packets. Mapping a BDF identifier to a unique device address can allow the encapsulated packets to be transmitted statelessly and flow between a first device and the second device. Scalability is possible because there is no need for CPU bandwidth from the first device and the second device, and there is no need for memory space from the first device and the second device to store information. There is no stateful switching to manage and no store-and-forward to implement. The encapsulated packets flow between devices without a staging buffer. There is no need for fabric protocol translation, such as RDMA over TCP / IP over Ethernet.

[0022] In another approach, a first device can communicate with a plurality of second devices over a fabric network. Each of the first device and the second devices can have a BDF identifier and can use the BDF identifier to address information passed to the other device. The BDF identifier can be translated into a unique device address according to the fabric network to send information over the fabric network.

[0023] In another approach, an I / O queue is placed in a memory of a processing circuit. For example, the I / O queue can reside in a processing circuit associated with a second device. Placing the I / O queue in the processing circuit can free up the memory of the first device and the second device to handle other tasks. The I / O queue can allow multiple devices to connect to the second device.

[0024] Unless otherwise expressly specified, the term “communicate” and its variants can include transferring information, sending information, and receiving information.

[0025] Unless otherwise expressly specified, the term “information” and its variants can include data, payload, header, footer, metadata, PCIe Transaction Layer Protocol Packet (TLP), PCIe Data Link Layer Packet (DLLP), bits, bytes, and datagrams, to name just a few examples.

[0026] In some embodiments, the systems and methods of the present disclosure can refer to an SSD storage system, which can include an SSD pipelined accelerator and a storage controller, or a pipelined processor and a network controller for a transport layer protocol (e.g., PCIe).

[0027] An SSD is a data storage device that uses integrated circuit assemblies as the memory for permanently storing data. An SSD has no moving mechanical components, and this feature differentiates an SSD from traditional electromechanical disks, such as hard disk drives (HDDs) or floppy disks, which contain rotating disks and movable read / write heads. Compared with electromechanical disks, an SSD is typically more resistant to physical shock, runs quietly, has a shorter access time, and lower latency.

[0028] Many types of SSDs use NAND-based flash memory (which does not require power to retain data) and include a type of non-volatile storage technology. When serving read / write input / output (I / O) workloads, the quality of service (QoS) of an SSD may be related to the predictability of low latency and the consistency of a high number of input / output operations per second (IOPS). This means that the latency or I / O command completion time needs to be within a specified range without having unexpected outliers. The throughput or I / O rate may also need to be tightly regulated without causing a sudden drop in the performance level.

[0029] In some embodiments, the systems and methods of the present disclosure may refer to an HDD storage system, which may include an HDD controller and a network controller for a transport layer protocol (e.g., PCIe).

[0030] Figure 1 An illustrative diagram of a system 100 for transferring information between devices via a fabric network 112 in accordance with some embodiments of the present disclosure is shown. A first device 102 communicates with a second device 104.

[0031] System 100 includes processing circuitry, such as first processing circuitry 110A and second processing circuitry 110B. The first processing circuitry 110A is part of a first device 102, and the second processing circuitry 110B is part of a second device 104. The first device 102 and the second device 104 communicate with one another via the first processing circuitry 110A and the second processing circuitry 110B over a fabric network 112. In the depicted embodiment, the first processing circuitry 110A includes an initiator 103 (such as a CPU) and a first PCIe bridge 111A. The second processing circuitry 110B includes a second PCIe bridge 111B. The first PCIe bridge 111A may receive information from the initiator 103 and pass the information to the second PCIe bridge 111B. The information is addressed to a device identifier of the second device 104 (such as a target 105), such as a bus:device:function (BDF) identifier. The target 105 may be a memory or storage device of the second device 104. The first processing circuitry 110A passes the information over the fabric network 112 to the second device 104, which may be Ethernet, Fibre Channel (FC), or InfiniBand, to name a few examples. The first processing circuitry 110A and / or the second processing circuitry 110B maps the BDF identifier of the target 105 to a unique device address compatible with the fabric network 112. For example, if the fabric network 112 is Ethernet, the unique device address may be an Internet Protocol (IP) address.

[0032] The second processing circuitry 110B may similarly use the BDF identifier of the first device 102 (such as the first PCIe bridge 111A) to pass information from the second device 104 to the first device 102 over the fabric network 112. Once the BDF identifier is mapped to a unique device address, information may flow between the first device 102 and the second device 104 without managing the information exchange or staging buffers.

[0033] In some embodiments, the first PCIe bridge 111A and the second PCIe bridge 111B may be PCIe chips.

[0034] In some embodiments, each of the first device 102 and the second device 104 may be part of a fabric node (e.g., a node of the fabric network 112). In some embodiments, each of the first device 102 and the second device 104 may be a fabric node.

[0035] In some embodiments, the first device 102 may be a host device, and the second device 104 may be a storage device. In some embodiments, the first device 102 may be a first host device, and the second device 104 may be a second host device. The first processing circuit 110A and the second processing circuit 110B may allow communication between host devices via the fabric network 112. In such an embodiment, the target 105 may be the initiator 105 of the second device 104. The initiators 103 and 105 may cause a BDF identifier conflict (e.g., both may be associated with 0:0:0, such as via the first processing circuit 110A and the second processing circuit 110B), and this conflict can be resolved as discussed with respect to Figure 2A . In one example, the BDF identifier of the first processing circuit 110A may be translated to 1:0:0, and the BDF identifier of the second processing circuit 110B may be translated to 2:0:0.

[0036] Figure 2A FIG. shows an illustrative diagram of a system 200 for transferring information (e.g., Figure 2B the information 230 therein) between devices (including subsystems of the devices) via a fabric network 212. In particular, Figure 2A the system 200 may transfer information between a first device (e.g., the host device 202) and a subsystem of a device (e.g., the storage array 204). The storage array 204 includes a second device (e.g., the first SSD 206A), a third device (e.g., the second SSD 206B), and a fourth device (e.g., the third SSD 206C), which are collectively referred to as SSDs 206A - 206C. Although three SSD devices (206A, 206B, and 206C) are shown in Figure 2A , any suitable number of SSD devices may be used in some embodiments.

[0037] System 200 includes a first processing circuit 210A and a second processing circuit 210B that communicate via a fabric network 212. In the depicted embodiment, the first processing circuit 210A includes a launcher 203 (such as a CPU) and a first PCIe bridge 211A. The second processing circuit 210B includes a second PCIe bridge 211B, a storage controller 207, and a third PCIe bridge 211C. The first PCIe bridge 211A receives information from the launcher 203 and passes the information to the second PCIe bridge 211B. The second PCIe bridge passes the information to the third PCIe bridge 211C via the storage controller 207. The storage controller 207 can process data services for SSDs 206A - 206C. The third PCIe bridge 211C passes the information to the storage array 204, and in particular, the storage array 204 has SSDs 206A - 206C.

[0038] Each of the SSDs 206A - 206C can have a BDF identifier and can communicate with each other using the BDF identifier. The launcher 203 can be connected to the first PCIe bridge 211A via a first PCIe bus 220. The first SSD 206A, the second SSD 206B, and the third SSD 206C can be connected to the second processing circuit 210B via a second PCIe bus 226A, a third PCIe bus 226B, and a fourth PCIe bus 226C respectively, and in particular, to the third PCIe bridge 211C. The first processing circuit 210A maps the first PCIe bus 220 to a first unique device address 222, which is associated with the host device 202. The second processing circuit 210B maps the second PCIe bus 226A, the third PCIe bus 226B, and the fourth PCIe bus 226C to a second unique device address 224A, a third unique device address 224B, and a fourth unique device address 224C respectively. The second unique device address 224A, the third unique device address 224B, and the fourth unique device address 224C are associated with the first SSD 206A, the second SSD 206B, and the third SSD 206C respectively. The unique device addresses 222 and 224A - 224C can be used to transfer and route information via the fabric network 212. The fabric network 212 can be any fabric network, such as Ethernet, FC, or InfiniBand, to name just a few examples.

[0039] The initiator 203 can discover devices that can have BDF identifiers. When the system 200 is initialized, the initiator 203 can probe the hierarchy of all devices connected to the system 200 and discover the first processing circuit 210A, which includes the first PCIe bridge device 211A that provides a path to a subset of the hierarchy. The initiator 203 configures the first processing circuit 210A as a bridge and assigns it the bus number of the BDF identifier. Each device connected to the system 200 can have a PCIe interface (e.g., a PCIe bridge or a PCIe chip) that responds to the probe query and identifies the downstream devices connected to the PCIe interface. For example, the host device 202 can have a PCIe chip as a root complex. The initiator 203 can initialize the PCIe chip of the host device 202 and enumerate it with a BDF of 0:0:0, where the first "0" is the bus number of the BDF identifier. The downstream devices connected to the PCIe chip of the host device 202 can be enumerated with different device and function numbers, but will have the same bus number (i.e., 0). One type of downstream device can be another bridge, and another bus number is assigned to the other bridge with devices connected to it. The enumeration continues through the hierarchy of the downstream devices. For example, the second PCIe bridge 211B can be assigned a BDF of 1:0:0, while the third PCIe bridge 211C can be assigned a BDF of 2:0:0. The first SSD 206A, the second SSD 206B, and the third SSD 206C can be assigned BDFs of 2:1:0, 2:2:0, and 2:3:0, respectively.

[0040] The initiator 203 probes other PCIe chips to identify other devices. Each PCIe chip is enumerated with a different bus number (e.g., 1, 2, etc.), and the downstream devices are enumerated with different device and function numbers associated with the bus number of the PCIe chip. The initiator 203 can probe through the fabric network 212 to identify the devices connected through the fabric network 212 (e.g., the PCIe bridges 211B and 211C and the SSDs 206A - 206C). Each PCIe chip connected to the fabric network 212 can have an independent Peripheral Component Interconnect (PCI) domain and can be enumerated by the initiator 203 with a varying bus number. Bus number conflicts can occur, such as when there are multiple host devices 202 with independent domains (e.g., Figure 3When in the host devices 350A - 350C). In a PCIe network, independent PCI domains can be addressed using a Non - Transparent Bridge (NTB), which can be used to interconnect independent PCI domains. The NTB can perform BDF translation to accommodate conflicting bus numbers between domains. In some embodiments, the first processing circuit 210A and the second processing circuit 210B can be NTBs that perform BDF translation. In some embodiments, one of the first processing circuit 210A and the second processing circuit 210B can be a PCI - over - Ethernet (PCIe oF) bridge on the fabric. In some embodiments, PCIe oF bridges of independent PCI domains may need to communicate with each other to resolve address translation. In some embodiments, a multicast address recognized by each device can be used, allowing the fabric network 212 to deterministically find the participating devices. The multicast address can be used with the Ethernet fabric network 212. In some embodiments, each node of the fabric network 212 can be registered with a "name server". Indicators can be added to the name server to ensure that each device is recognized. The name server can be used with the FC fabric network 212. Ethernet and InfiniBand fabric networks 212 can use a method similar to the name server.

[0041] Each of the first processing circuit 210A and the second processing circuit 210B can include a bridge chip. The bridge chip can be used to convert or translate between BDF identifiers and unique device addresses. The bridge chip can be used to encapsulate and decapsulate multiple packets 234.

[0042] In some embodiments, the second processing circuit 210B can be used to discover devices with BDF identifiers.

[0043] In some embodiments, the unique device address is a Media Access Control (MAC) address. In some embodiments, the BDF identifier is mapped to the MAC address. In one embodiment, the BDF identifier is used as the lower three bytes of the MAC address. The Ethernet fabric network 212 can use the MAC address.

[0044] In some embodiments, the unique device address is an IP address. In some embodiments, the BDF identifier is mapped to the IP address. In one embodiment, the BDF identifier is used as three bytes of the IP address. Ethernet and InfiniBand fabric networks 212 can use the IP address.

[0045] In some embodiments, the unique device address is a 24 - bit FC identifier. In some embodiments, the BDF identifier is mapped to the 24 - bit FC identifier. In one embodiment, the BDF identifier is used as the 24 - bit FC identifier. The FC fabric network 212 can use the FC identifier.

[0046] In some embodiments, the unique device address is a local identifier (LID). In some embodiments, the BDF identifier is mapped to the LID. In one embodiment, the BDF identifier is used as the LID. The InfiniBand fabric network 212 can use the LID.

[0047] Although in the depicted embodiments, the first PCIe bus 220, the first unique device address 222, the line shown passing through the fabric network 212, the second through fourth unique device addresses 224A - 224C, and the second through fourth PCIe buses 226A - 226C are each shown as a single line, each of these connections can include multiple lines or channels. In some embodiments, the first PCIe bus 220, the first unique device address 222, and the connection through the fabric network 212 can include lines for each endpoint connected to the host device 202 (e.g., SSDs 206A - 206C). In some embodiments, the number of lines for each connection can depend on the number of channels of the PCIe slot of the host device 202 or the SSDs 206A - 206C. For example, each channel can have one line.

[0048] Figure 2B An illustrative diagram of multiple packets 234 being transferred between devices in accordance with some embodiments of the present disclosure is shown. Figure 2A In the embodiments depicted Figure 2B the host device 202 sends information 230 to the first SSD 206A. The information 230 can include data, headers, and PCIe TLPs or data link layer packets (DLLPs), to name just a few examples.

[0049] The initiator 203 transfers the information 230 using the BDF identifier of the first SSD 206A. In the depicted embodiments, the information 230 includes multiple packets 234. The initiator 203 sends the information 230 through the first PCIe bus 220 to the first processing circuit 210A. The first processing circuit 210A encapsulates each of the multiple packets 234 to generate multiple encapsulated packets 236. The first processing circuit 210A uses the unique device address of the second processing circuit 210B (e.g., Figure 2A the second unique device address 224A in

[0050] In some embodiments, each of the plurality of packets is 2 kilobytes (KB) or less, such as 1.5 KB or less, such as 1 KB or less. The Ethernet fabric network 212 can have frames capable of accommodating a payload of up to 1.5 KB bytes. In some embodiments, Ethernet can use jumbo frames, which can accommodate a payload of up to 9 KB bytes. The FC fabric network 212 can accommodate a payload of up to 2 KB bytes. The InfiniBand fabric network 212 can accommodate a payload of up to 4 KB bytes.

[0051] In some embodiments, the plurality of packets 234 can be PCIe packets. In some embodiments, the plurality of packets 234 can be encapsulated as a plurality of packets of TCP / IP / Ethernet (TIE) packets. In some embodiments, the plurality of packets 234 can be encapsulated as a plurality of packets of User Datagram Protocol (UDP) / IP / Ethernet (UIE) packets. TIE and UIE packets can be used with the Ethernet fabric network 212. UIE packets are well-suited because of their lack of state information such as acknowledgments and checksums. In some embodiments, the plurality of packets 234 can be encapsulated as a plurality of FC packets. FC packets can be used with the FC fabric network 212. FC packets can be Class 1, Class 2, or Class 3 packets. FC Class 3 packets are well-suited because of their lack of state information such as acknowledgments and checksums. In some embodiments, the plurality of packets 234 can be encapsulated as a plurality of InfiniBand packets. InfiniBand packets can be used with the InfiniBand fabric network 212.

[0052] In a stateless communication embodiment, the concept of a plurality of packets can be dispensed with, and each individual packet among the plurality of packets can be considered an atomic unit of communication across the fabric network 212.

[0053] In some embodiments, the information 230 can be sent between the host device 202 and other devices such as a second SSD 206B and / or a third SSD 206C. In some embodiments, the other devices can be other than SSDs. For example, the host device 202 can communicate with: a central processing unit (CPU), a data processing unit (DPU), a graphics card and a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a sound card, an Ethernet card, and a redundant array of inexpensive disks (RAID) card, to name just a few examples. In such embodiments, the second processing circuit 210B can be connected to or reside in the other devices.

[0054] Although with respect to Figure 2BThe discussion envisioned using first processing circuitry 210A and second processing circuitry 210B to send information 230 from host device 202 to first SSD 206A, but in some embodiments, the process can be reversed. First SSD device 206A can send information 230 to host device 202. Second processing circuitry 210B can generate a plurality of packets 234 and a plurality of encapsulated packets 236. Second processing circuitry 210B can send the encapsulated packets 236 to first processing circuitry 210A. First processing circuitry can de - encapsulate the encapsulated packets 236 to generate a plurality of packets 234 before sending them to host device 202.

[0055] In some embodiments, each of processing circuitry 210A and processing circuitry 210B has a chip, such as a bridge chip, to facilitate communication across fabric network 212. The bridge chip can be used to encapsulate and de - encapsulate a plurality of packets 234. The bridge chip can translate or convert between a BDF identifier and a unique device address. For example, the bridge chip can perform the translation after receiving a plurality of packets 234 from PCIe bus 220 or 226A and before passing the encapsulated packets 236 across fabric network 212, or after receiving the encapsulated packets 236 from fabric network 212 and before sending the plurality of packets 234 to PCIe bus 220 or 226A.

[0056] Figure 3 A diagram illustrating information (e.g., Figure 2B information 230 in ) being passed between devices using an input / output (I / O) queue 340 is shown in accordance with some embodiments of the present disclosure. The devices can include host devices (e.g., first host 350A, second host 350B, and so on up to “mth” host 350C) and storage devices (e.g., first SSD 352A, second SSD 352B, and so on up to “nth” SSD 352C).

[0057] I / O queue 340 resides in the memory 338 of processing circuitry 310, and processing circuitry 310 can be similar to that regarding Figure 2A and Figure 2BThe second processing circuit 210B under discussion. The I / O queue 340 includes a plurality of queue pairs 342. Each of the queue pairs 342 in the queue pair 342 includes a submission queue (SQ) 344 and a completion queue (CQ) 346. Each of the host devices 350A - 350C and the storage devices 352A - 352C is bound to a queue pair 342. The I / O queue 340 can be assigned to the storage devices 352A - 352C using "management" commands from the host devices 350A - 350C. The processing circuit 310 can respond to management commands from the host devices 350A - 350C to complete the creation of the I / O queue 340 by providing the local 64 - bit PCIe address of each I / O queue 340. In the depicted embodiment, each of the storage devices 352A - 352C has a number of queue pairs 342 equal to the number of host devices 350A - 350C, and the number is "m" hosts. The number of queue pairs 342 allows each of the host devices 350A - 350C to communicate with each of the storage devices 350A - 350C. Once the I / O queue 340 is established, information can flow between the host device and the storage device without requiring CPU bandwidth from the host device and the storage device to manage the information exchange or staging buffer.

[0058] The "I / O queue" 340 is an NVMe construct, not a PCIe construct. The I / O queue is used to submit and complete NVMe commands. The NVMe command describes the information to be transferred for that command, including the length and location of the information. When the host writes a command to the I / O queue across the fabric, this is done by transmitting and receiving multiple PCIe packets. In the present invention, these PCIe packets are addressed, encapsulated, transmitted, received, decapsulated, and forwarded to the destination PCIe bus, just like any other packet.

[0059] The first host 350A can communicate with the first SSD 352A by writing a command as an entry to the SQ 344 (referred to as an SQ entry). The command describes the information to be transferred between the first host 350A and the first SSD 352A. As Figure 2BAs discussed, the information 230 can be sent in a packet (e.g., packet 234 or encapsulated packet 236). The first SSD 352A obtains a command from SQ 344 and initiates an information transfer request to send or receive the information 230. When all the information 230 has been transferred, the first SSD 352A writes an entry (referred to as a CQ entry) to CQ 346 to indicate that the command associated with the SQ entry has been completed and the information has been transferred. The first host 350A processes the CQ entry. The host can also write to a doorbell register (not shown) to flag that a new command has been written to SQ 344. The first SSD 352A can write to the doorbell register to flag the CQ entry, such as after the information 230 has been transferred.

[0060] Each queue in the I / O queue 340 has a queue identifier. The queue identifier of each SQ 344 is not explicitly specified in the NVMe command. The queue identifier of each SQ 344 can be inferred from the SQ 344 filled with the queue identifier. The doorbell register can be accessed via the PCIe address, and the associated SQ identifier of the doorbell register can be inferred. The SQ identifier can be virtualized, exposing one value to the first host 350A and a potentially different value to the first SSD 352A. CQ 346 has a queue identifier. The processing circuit 310 can intercept the I / O command completion and change the CQ identifier, and then forward the changed CQ identifier to the first host 350A. The CQ identifiers for the "abort" process and the "get error log" command can be exceptions to the CQ change because the SQ identifier for each of them is explicitly specified and must be correctly mapped before it is sent to the first host 350A.

[0061] Although the communication between the first host 350A and the first SSD 352A has been discussed, the above communication can occur between any host devices 350A - 350C and storage devices 350A - 350C.

[0062] In some embodiments, there are more storage devices 352A - 352C than host devices (i.e., n > m), or vice versa (i.e., n < m). In some embodiments, there are the same number of storage devices 352A - 352C and host devices 350A - 350C (i.e., n = m). In some embodiments, the number of queue pairs 342 may not be based on the total number of host devices 350A - 350C. For example, some storage devices 352A - 352C may not be connected to all of the host devices 350A - 350C.

[0063] In some embodiments, the processing circuit 310 can be similar with respect to Figure 1The processing circuit 110 under discussion. In some embodiments, the processing circuit 310 may be similar to the first processing circuit 210A discussed with respect to Figure 2A and Figure 2B The first processing circuit 210A discussed with respect to Figure 2A and Figure 2B The first processing circuit 210A and the second processing circuit 210B discussed with respect to Figure 2A The storage control subsystem discussed with respect to

[0064] Figure 4 An alternative illustrative diagram showing information (e.g., the information 230 in Figure 2B ) transferred between devices using the I / O queue 440 in accordance with some embodiments of the present disclosure. The devices may include a first host 350A and a second host 350B (collectively referred to as host devices 350A and 350B) and a first SSD 352A and a second SSD 352B (collectively referred to as storage devices 352A and 352B).

[0065] The processing circuit 410 includes a bridge chip 437, a memory 438, and circuit logic 439. The circuit logic 429 may include, by way of example only, a controller, a central processing unit (CPU) 439, or code. The circuit logic 439 may discover the first SSD 352A and the second SSD 352B and set up the I / O queue 440. The I / O queue 440 resides in the memory 438 and includes an I / O queue pair 442 and an I / O queue group 443. The I / O queue pair 442 includes an SQ 444 and a CQ 446. The I / O queue group 443 includes an SQ 444, a first CQ 466A, and a second CQ 466B. Unless otherwise noted, the function of the I / O queue 440 is similar to the I / O queue 340 discussed with respect to Figure 3 The first CQ 466A is dedicated to a storage device (e.g., the first SSD 352A or the second SSD 352B), while the second CQ 466B is dedicated to the corresponding host device (e.g., the first host 350A or the second host 350B). In one example, the first SSD 352A writes an entry (referred to as a storage device CQ entry) to the first CQ 446A. The CPU 439 processes and transforms the storage device CQ entry and moves the entry to the second CQ 446B (referred to as a host device CQ entry). The first host 350A processes the host device CQ entry. The host devices 350A and 350B may use doorbell registers as discussed with respect to Figure 3 The first SSD 352A and the second SSD 352B may use doorbell registers as discussed with respect to Figure 3 The doorbell registers discussed with respect to

[0066] In some embodiments, the processing circuit 410 may be part of or attached to a storage system, such as the storage array 204 discussed with respect to Figure 2A For embodiments with a processing circuit 410, a first CQ 446A and a second CQ 446B may be required, where the processing circuit 410 is part of or attached to a host device and a storage device, such as the first processing circuit 210A and the second processing circuit 210B discussed with respect to Figure 2A For example, the first CQ 446A may use a different BDF identifier than the second CQ 446B to identify the host devices 350A and 350B, and the second CQ 446B may use a different BDF identifier than the first CQ 446A to identify the storage devices 352A and 352B. In such embodiments, the circuit 439 may translate the BDF identifier.

[0067] In some embodiments, storage device CQ translation may be offloaded to a field programmable gate array (FPGA) or application specific integrated circuit (ASIC) logic, where duplicate CQs 466A and 466B may not be required or duplicate CQs 466A and 466B may be translated without CPU intervention.

[0068] Although Figure 4 two host devices 350A and 350B and two storage devices 352A and 352B are shown, other embodiments may use more or fewer host devices and storage devices.

[0069] Although Figure 3 and Figure 4 discuss communication between the host devices 350A and 350B and the storage devices 352A and 352B, in some embodiments, communication may occur between the host devices 350A and 350B. For example, between a CPU and any CPU, DPU, graphics card or GPU, FPGA, ASIC, and sound card, to name just a few examples. In such embodiments, the I / O queue 340 or 440 may be part of or attached to the host devices 350A and 350B.

[0070] Figure 5 illustrates a method 500 for transmitting information (e.g., Figure 2B the information 230 in

[0071] Method 500 begins at operation 502, where a processing circuit (e.g., respectively in Figure 1 、 Figure 2A and Figure 2B 、 Figure 3and Figure 4 the processing circuits 110, 210A and / or 210B, 310 or 410) receive a plurality of packets from a first device (e.g., Figure 1 the first device 102 in Figure 2A and Figure 2B the host device 202 in Figure 3 and Figure 4 the host devices 350A - 350C in, as described above with respect to Figures 1 - 4 In some embodiments of method 500, a plurality of packets are addressed to a second device (e.g., Figure 1 the second to fourth devices 106A - 106C in Figure 2A and Figure 2B the first to third SSDs 206A - 206C in, and Figure 3 and Figure 4 the storage devices 352A - 352C in) among a plurality of devices using a device identifier.

[0072] At operation 504, the processing circuit maps the device identifier of the second device to a unique device address, as described above with respect to Figures 1 - 2B

[0073] At operation 506, the processing circuit encapsulates each of the plurality of packets to generate a plurality of encapsulated packets (e.g., Figure 2B the encapsulated packet 236 in Figure 2B and Figure 3 ), as described above with respect to

[0074] At operation 508, the processing circuit transmits each of the plurality of encapsulated packets over a fabric network, as described above with respect to Figures 2B - 4 . In some embodiments of method 500, the unique device address of the second device is used to route the plurality of encapsulated packets to the second device.

[0075] In some embodiments, the device identifier of the second device is a bus:device:function identifier. In some embodiments, the first device is a host device and the second device is a storage device. In some embodiments, the second device is a Just a Bunch of Flash (JBOF) device. In some embodiments, the first device is a storage device and the second device is a host device.

[0076] ​Some embodiments further include receiving information from a second device. The information is addressed to the first device using a device identifier. Some embodiments further include mapping the device identifier of the first device to a unique device address. Some embodiments further include generating a plurality of packets based on the information and encapsulating each of the plurality of packets to generate a plurality of encapsulated packets. Some embodiments further include transmitting each of the plurality of encapsulated packets over a fabric network. The unique device address of the first device is used to route the plurality of encapsulated packets to the first device.

[0077] In some embodiments, the first device and the second device are configured to use a Peripheral Component Interconnect Express (PCIe) bus interface for transmitting and receiving information.

[0078] Some embodiments further include establishing an input / output (I / O) queue pair (e.g., Figure 3 and Figure 4 queue pairs 342 and 442 in Figure 4 or, in some embodiments, a queue group in

[0079] and mapping the I / O queue pair to the first device and to the second device.

[0080] In some embodiments, the plurality of packets are PCIe packets. Each of the plurality of packets is encapsulated as a plurality of UDP / IP / Ethernet (UIE) packets.

[0081] In some embodiments, the plurality of packets are PCIe packets. Each of the plurality of packets is encapsulated as a plurality of TCP / IP / Ethernet (TIE) packets.

[0082] In some embodiments, the plurality of packets are PCIe packets. Each of the plurality of packets is encapsulated as a plurality of Fibre Channel (FC) packets.

[0083] In some embodiments, the plurality of packets are PCIe packets. Each of the plurality of packets is encapsulated as a plurality of InfiniBand packets.

[0084] In some embodiments, each of the plurality of packets is 2 kilobytes (KB) or smaller.

[0085] In some embodiments, the unique device address is a Media Access Control (MAC) address. Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the MAC address by using the device identifier as the lower three bytes of the MAC address.

[0086] In some embodiments, the unique device address is an Internet Protocol (IP) address. Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the IP address by using the device identifier as three bytes of the IP address.

[0087] In some embodiments, the unique device address is a 24-bit Fibre Channel (FC) identifier. Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the 24-bit FC identifier by using the device identifier as the 24-bit FC identifier.

[0088] In some embodiments, the unique device address is a Local Identifier (LID). Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the LID by using the device identifier as the LID address.

[0089] Note that Figure 5 is just an example of a method, and other methods including fewer, additional, or alternative steps may be consistent with the present disclosure.

[0090] Figure 6 An example of a system processing circuit 600 according to some embodiments of the present disclosure is shown.

[0091] The system processing circuit 600 includes a first processing circuit 604 and a second processing circuit 654. The first processing circuit 604 is connected to an I / O device 606 and a network interface 608. The first processing circuit 604 includes a storage device 610, a memory 612, and a controller, such as a CPU 614. The CPU 614 may include any of the storage controllers 207 discussed with respect to Figure 2A and any of the circuit logics 429 discussed with respect to Figure 4 . The CPU 614 is configured to process computer-executable instructions, such as instructions stored in the memory 612 or the storage device 610, and cause the system processing circuit 600 to perform the methods and processes described herein (e.g., with respect to Figure 5 ).

[0092] The CPU 614 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, and other forms of processing architectures capable of executing computer-executable instructions.

[0093] The I / O device 606 includes a first device 616, which may include the first device 102 discussed with respect to Figure 1 , the host device 202 discussed in Figure 2A and Figure 2B , and the devices discussed with respect to Figure 3 and Figure 4Any of the host devices 350A - 350C discussed.

[0094] The network interface 608 provides access to an external network, such as the fabric network 640, to the first processing circuit 604. Regarding Figure 4 The bridge chip 437 discussed may include the network interface 608. The fabric network 640 may include regarding Figure 2A and Figure 2B the fabric network 212 discussed. In some embodiments, the network interface 608 may include one or more of a receiver, a transmitter, or a transceiver. The fabric network 640 may be a storage area network (SAN), a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a satellite communication network, etc., and communicate according to Ethernet, FC, or InfiniBand protocol, to name just a few examples.

[0095] The second processing circuit 654 is connected to the I / O device 656 and the network interface 658. The second processing circuit 654 includes a storage device 660, a memory 662, and a processor, such as the CPU 664. The CPU 664 and the network interface 658 may be configured similarly to the CPU 614 and the network interface 608, respectively.

[0096] The I / O device 656 includes a second device 656, which may include regarding Figure 1 the second to fourth devices 106A - 106C discussed, the Figure 2A and Figure 2B SSD 206A - 206C discussed in Figure 3 and Figure 4 any of the storage devices 352A - 352C discussed.

[0097] The network interface 658 connects the second processing circuit 654 to the first processing circuit 604 via the fabric network 640, allowing the first device 616 and the second device 666 to communicate.

[0098] Unless otherwise explicitly specified, the terms "one embodiment", "an embodiment", "multiple embodiments", "the embodiment", "the multiple embodiments", "one or more embodiments", "some embodiments", and "an embodiment" mean "one or more (but not all) embodiments".

[0099] Unless otherwise explicitly specified, the terms "comprising", "including", "having", and their variants mean "including but not limited to".

[0100] Unless otherwise explicitly specified, an enumeration of items does not imply that any or all of the items are mutually exclusive.

[0101] Unless otherwise expressly specified, the terms "a", "an", and "the" mean "one or more".

[0102] Unless otherwise expressly specified, devices that communicate with each other do not need to communicate with each other continuously. In addition, devices that communicate with each other can communicate directly or indirectly through one or more intermediaries.

[0103] The description of embodiments with several components that communicate with each other does not imply a requirement for all such components. Instead, a wide variety of alternative components are described to illustrate a wide variety of possible embodiments. Further, although process steps, method steps, algorithms, etc. may be described in a sequential order, these processes, methods, and algorithms can be configured to work in an alternative order. In other words, any order or sequence of steps that may be described does not necessarily indicate a requirement to execute the steps in that order. The process steps described herein can be executed in any practical order. Further, some steps can be executed simultaneously.

[0104] When a single device or article is described herein, it will be apparent that more than one device / article (whether or not they cooperate) can be used in place of the single device / article. Similarly, in cases where more than one device or article is described herein (whether or not they cooperate), it will be apparent that a single device / article can be used in place of the more than one device or article, or that a different number of devices / articles can be used in place of the number of devices or programs shown. The functions and / or features of a device can alternatively be embodied by one or more other devices not expressly described as having such functions / features. Accordingly, other embodiments need not include the device itself.

[0105] In the figures, at least some of the operations that may have been illustrated show certain events occurring in a certain order. In alternative embodiments, certain operations can be performed, modified, or removed in a different order. In addition, steps can be added to the foregoing logic and still conform to the described embodiments. Further, the operations described herein can occur in sequence, or certain operations can be processed in parallel. Still further, the operations can be performed by a single processing unit or by distributed processing units.

[0106] For purposes of illustration and description, the foregoing description of various embodiments has been presented. It is not intended to be exhaustive or to limit the precise forms disclosed. Many modifications and variations are possible in light of the above teachings.

Claims

1. A method for communication, comprising: Receiving a first plurality of packets from a first device, wherein the first plurality of packets are addressed to a second device among a plurality of devices using a device identifier; Mapping the device identifier of the second device to a unique device address; Encapsulating each of the plurality of packets to generate a first plurality of encapsulated packets; And Transmitting each of the first plurality of encapsulated packets through a fabric network, wherein the unique device address of the second device is used to route the first plurality of encapsulated packets to the second device.

2. The method according to claim 1, wherein the device identifier of the second device is a bus:device:function identifier, the first device is a host device, and the second device is a storage device.

3. The method according to claim 2, wherein the second device is a Just a Bunch of Flash (JBOF) device.

4. The method according to claim 1, wherein the device identifier of the second device is a bus:device:function identifier, the first device is a storage device, and the second device is a host device.

5. The method according to claim 1, further comprising: Receiving a second plurality of packets from the second device, wherein the second plurality of packets are addressed to the first device using a device identifier; Mapping the device identifier of the first device to a unique device address of the first device; Encapsulating each of the second plurality of packets received from the second device to generate a second plurality of encapsulated packets; And Transmitting each of the second plurality of encapsulated packets through the fabric network, wherein the unique device address of the first device is used to route the second plurality of encapsulated packets to the first device.

6. The method according to claim 1, wherein: The device identifier of the second device is a bus:device:function identifier; And The first device and the second device are configured to use a Peripheral Component Interconnect Express (PCIe) bus interface for sending and receiving information.

7. The method according to claim 1, wherein each of the first plurality of encapsulated packets is transmitted statelessly through the fabric network.

8. The method according to claim 1, further comprising: Establishing an input / output (I / O) queue; And Mapping the I / O queue to the first device and the second device.

9. The method according to claim 8, wherein transmitting each of the first plurality of encapsulated packets through the fabric network is initiated by writing a command to the I / O queue and includes sending the first plurality of encapsulated packets to the second device.

10. The method according to claim 1, wherein: The device identifier of the second device is a bus:device:function identifier; The first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and Each of the first plurality of packets is encapsulated into a plurality of packets of a Transmission Control Protocol / Internet Protocol / Ethernet (TIE) packet.

11. The method according to claim 1, wherein: The device identifier of the second device is a bus:device:function identifier; The first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and Each of the first plurality of packets is encapsulated as a plurality of UDP / IP / Ethernet (UIE) packets.

12. The method according to claim 1, wherein: The device identifier of the second device is a bus:device:function identifier; The first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; And Each of the first plurality of packets is encapsulated as a plurality of Fibre Channel (FC) packets.

13. The method according to claim 1, wherein: The device identifier of the second device is a bus:device:function identifier; The first plurality of packets are Peripheral Component Interconnect Express (PCIe) packets; and Each of the first plurality of packets is encapsulated as a plurality of InfiniBand packets.

14. The method according to claim 1, wherein each of the first plurality of packets is 2 kilobytes (KB) or smaller.

15. The method according to claim 1, wherein: The unique device address is a Media Access Control (MAC) address; and Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the MAC address by using the device identifier as the lower three bytes of the MAC address.

16. The method according to claim 1, wherein: The unique device address is an Internet Protocol (IP) address; And Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the IP address by using the device identifier as three bytes of the IP address.

17. The method according to claim 1, wherein: The unique device address is a 24-bit Fibre Channel (FC) identifier; and Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the 24-bit FC identifier by using the device identifier as the 24-bit FC identifier.

18. The method according to claim 1, wherein: The unique device address is a Local Identifier (LID); and Mapping the device identifier of the second device to the unique device address includes mapping the device identifier of the second device to the LID by using the device identifier as the LID address.

19. A system comprising a processing circuit configured to execute a method, the method comprising: Receiving information from a first device, wherein the information is addressed to a second device among a plurality of devices using a device identifier; Mapping the BDF identifier of the second device to a unique device address; Generating a plurality of packets based on the information; Encapsulating each of the plurality of packets to generate a plurality of encapsulated packets; And Transmitting each of the plurality of encapsulated packets over a fabric network, wherein the unique device address of the second device is used to route the plurality of encapsulated packets to the second device.

20. A non-transitory computer-readable storage medium including computer-executable instructions that, when executed by a processor, cause the processor to perform operations including: Receiving information from a first device, wherein the information is addressed to a second device among a plurality of devices using a device identifier; Mapping a BDF identifier of the second device to a unique device address; Generating a plurality of packets based on the information; Encapsulating each of the plurality of packets to generate a plurality of encapsulated packets; And Transmitting each of the plurality of encapsulated packets through a fabric network, wherein the unique device address of the second device is used to route the plurality of encapsulated packets to the second device.

Images