Local installation of remote peripheral devices

Abstract

Improved apparatus, systems, and techniques are presented for coupling a network-linked peripheral device to a host computing device. The method includes obtaining, via a network interface of the host device, an indication of a peripheral device available for association with the host device. Based on the indication, the method includes initiating instantiation of the peripheral device into a Peripheral Component Interconnect Express (PCIe) subsystem of the host device by at least emulating the behavior of the peripheral device via the network interface as a PCIe peripheral device locally coupled to the host system.

Description

[Background technology]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of and priority to U.S. patent application Ser. No. 17 / 581,605, entitled "LOCAL INSTANTIATION OF REMOTE PERIPHERAL DEVICES," filed on January 21, 2022. Clustered computer systems are becoming more prevalent due to the growing demand for data storage, data processing, and communication processing. Data centers typically include large rack-mounted and network-coupled data storage and data processing systems. These data centers may receive data for storage from external users over network links as well as data generated from applications running on processing elements within the data center. Often, data centers and associated computer equipment may be utilized to execute jobs for multiple simultaneous users or applications. Jobs include running jobs that may utilize the resources of the data center to process data using a central processing unit (CPU) or a graphics processing unit (GPU) as well as routing data associated with these resources between temporary and long-term storage or between various network locations. For example, GPU-based processing is becoming more popular for use in artificial intelligence (AI) and machine learning regimes. In these regimes, computer systems such as blade servers may include one or more GPUs along with associated CPUs for processing large data sets.

[0002] However, density limitations may occur in large computer clusters. Specifically, blade servers typically include a fixed arrangement between CPUs, GPUs, and storage elements housed in a common enclosure or chassis. When an incoming job is deployed in a data center, the granularity of computer resources is limited to individual servers. Thus, a deployed job typically occupies one or more servers along with all of the corresponding CPUs, GPUs, and storage elements of each server, regardless of whether the entire resources of the server are actually required to execute the job. To compensate, data center operators typically deploy a continuously increasing number of servers to accommodate the increasing traffic from the jobs. This strategy may face barriers due to large space and cost requirements in addition to the physical space required for rack-mounted servers. Summary of the Invention [Means for solving the problem]

[0003] Improved apparatus, systems, and techniques are presented for coupling remotely located communication fabric coupled peripheral devices to a host computer system as local devices via a communication network link. These improvements may provide peripheral devices such as GPUs, FPGAs, or ASICs (called co-processing units (CoPUs), or data processing accelerators) that are utilized on demand by the host computer system via the network link. These peripheral devices may be arbitrarily associated and disassociated with various host devices (such as servers or other computer systems) as if they were local peripheral devices plugged into the host device. Thus, the host device may add a larger number of peripheral devices for use in processing workloads or user data than would normally be feasible when compared to physically plugging the peripheral devices into the host device's motherboard.

[0004] In one example implementation, a method includes obtaining, via a network interface of the host device, an indication of a peripheral device available for association with the host device. Based on the indication, the method includes initiating instantiation of the peripheral device to a local peripheral interconnect interface of the host device by at least emulating a behavior of the peripheral device coupled via the network interface as a local peripheral device coupled to the peripheral interconnect interface of the host system.

[0005] In another exemplary implementation, an apparatus includes one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media. When executed by a processing system, the program instructions instruct the processing system to at least obtain, via a network interface of the host device, an indication of a peripheral device available via the network interface for association with the host device. Based on the indication, the program instructions instruct the processing system to initiate instantiation of the peripheral device to the local peripheral interconnect interface of the host device by at least emulating a behavior of the peripheral device coupled via the network interface as a local peripheral device coupled to the peripheral interconnect interface of the host system.

[0006] In yet another exemplary implementation, the software element comprises a local interface configured to communicate with a local Peripheral Component Interconnect interface of the host device and a remote interface configured to communicate with a network stack of a network interface of the host device. An interaction unit is included and configured to obtain, via the remote interface, an indication of a peripheral device available via the network interface for association with the host device. Based on the indication, the interaction unit is configured to initiate instantiation of the peripheral device onto the local Peripheral Component Interconnect interface of the host device by at least emulating a behavior of the peripheral device coupled via the network interface as a local peripheral device coupled to the host system.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the technical disclosure. It should be understood that this Summary is not intended to identify key features or essential features of the claimed subject matter, nor should it be used to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0008] Many aspects of the present disclosure can be best understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, like reference numbers indicate corresponding parts throughout the several views. Although several embodiments have been described in connection with these drawings, the present disclosure is not limited to the embodiments disclosed herein. On the contrary, the intention is to cover all alternatives, modifications, and equivalents.

[0009] [Figure 1] FIG. 1 illustrates a computer system in one implementation.

[0010] [Diagram 2]1 is a flow diagram illustrating an example of the operation of a computer system in one implementation.

[0011] [Diagram 3] FIG. 2 illustrates a host device in one implementation.

[0012] [Figure 4] FIG. 1 illustrates a fabric control system in one implementation.

[0013] [Diagram 5] FIG. 1 illustrates a computer system in one implementation.

[0014] [Figure 6] FIG. 1 illustrates a control system in one implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Data centers with associated computer equipment may be utilized to process running jobs and process data, as well as transfer data associated with running jobs between temporary and long-term storage, or between various network destinations. Data processing elements may include central processing units (CPUs) along with various types of peripheral devices. These peripheral devices may include co-processing units (CoPUs) or data processing accelerators, such as graphic processing units (GPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), or application specific integrated circuits (ASICs). Data processing of host device CPUs enhanced by CoPUs is gaining popularity for use in artificial intelligence (AI), machine learning systems, cryptocurrency mining and processing, advanced graphical visualization, biological systems modeling, autonomous vehicle systems, and various other tasks. In the examples herein, limitations of host computer systems may be overcome using distributed peripheral devices that may be dynamically attached to the host system while having no local coupling to the data bus of such host system. Instead of having a fixed arrangement between host devices and peripheral devices housed in a common enclosure or chassis, examples herein can flexibly include any number of peripheral devices across any number of enclosures / chassis, dynamically formed into a logical arrangement over a communication network. Thus, computer systems can better utilize resources by not having idle or wasted portions of blade servers that are not needed for a particular task, or for a particular portion of a task. Data center operators can achieve much higher utilization levels of their data centers than could be achieved using fixed-location servers, and augment existing servers with additional capabilities over existing network connections.

[0016] The systems and operations described herein provide dynamic allocation of peripheral resources, such as computer resources (CPU), graphics processing resources (GPU), network resources (NIC), storage resources (SSD), programmable logic resources (FPGA), and memory resources (RAM), to host devices, even if the peripheral resources are not locally coupled to the host device. The peripheral resources are distributed and reside in a pool of unused, unallocated, or free peripheral resources until they are allocated (configured) to a host device. A management processor, fabric controller, or fabric control system controls compositing and decompositing and may provide an interface to external users, job management software, or orchestration software. Peripheral resources and other elements (graphics processing, network, storage, FPGA, RAM, or other) may be attached / detached to / from a host device on the fly, and these resources may be assigned to other host devices. In one example, a graphics processing resource may be attached to a first host device and then provide the status / results of graphics processing to another host device after being detached from the first host device. In another example, when a resource fails, hangs, or is overloaded, additional peripheral resources may be introduced into the host device to supplement the existing resources.

[0017] As a first exemplary system, FIG. 1 is presented. FIG. 1 is a system diagram illustrating a computer system 100 utilizing an improved peripheral device composition technique. The system 100 includes a host device 110, a fabric control system 130, a network switch 131, a communication fabric switch 132, and a distributed peripheral device 140. The host device 110 communicates with the network switch 131 via network links 150-151. The peripheral device 140 communicates with the fabric switch 132 via communication fabric links 160-162. Although only one host device is shown in FIG. 1, it should be understood that additional host devices may be included and coupled to the network switch 131 via associated network links.

[0018] Host device 110 includes a network interface 111, a local peripheral interconnect interface 112, and a Peripheral over Fabric (PoF) system 115. Network interface 111 is coupled to a network switch 131 via a network link 150. Local interface 112 is coupled to a local peripheral device 113 via a local link (not shown). PoF system 115 is coupled to both network interface 111 and local interface 112 via software connections, such as via software interfaces to associated protocol stacks or programming interfaces of network interface 111 and local interface 112.

[0019] In operation, the host device 110 may use an on-board central processing unit (CPU) along with peripheral devices including graphics processing devices, data storage devices, memory devices, and user interface devices to execute system software and user applications for various tasks. An operator of the host device 110 may wish to add additional peripheral devices for use by the host device. Various improved techniques and systems are presented herein to facilitate the addition of additional peripheral devices to the host system without physically plugging such peripheral devices into the host system. Peripheral devices including data processing elements (such as CoPUs) or other peripheral devices (such as data storage or memory devices) may be configured to be associated with the host device 110 even if such devices or elements are not physically local to the host device 110. Elements included in the host device 110 and fabric control system 130 allow remote peripheral devices to be added for use by the host device 110 as if the remote peripheral devices were local devices coupled via a local interface, such as a PCIe interface. This allows any association between any peripheral device 140 and any host device 110 to be created and modified on the fly. These associations are made through a network interface of the host device 110 and a communication fabric interface that couples to the peripheral device 140, as described in more detail below.

[0020] Referring now to the description of the elements of FIG. 1, the host device 110 comprises a computer system having a processing element, a data storage and memory element, and a user interface element. Examples of the host device 110 include a server, a blade server, a desktop computer, a laptop computer, a tablet computer, a smart phone, a gaming system, an element of a distributed computer system, a customer equipment, an access terminal, an Internet appliance, a media player, or other computer system. Typically, the host device 110 includes a motherboard or other system circuit board having a central processing unit (CPU) coupled together with a memory device, such as a random access memory (RAM). The CPU can be a component in a processing system formed from one or more microprocessor elements, including an Intel® microprocessor, an Apple® microprocessor, an AMD® microprocessor, an ARM® microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a tensor processor, an application specific processor, or other microprocessor or processing element. Various peripheral devices may be physically coupled locally to a peripheral interconnect interface of the host device 110 through corresponding slots, connectors, or through cabling within an enclosure or chassis of the host device 110. These peripheral devices may include graphics cards housing a graphics processing unit (GPU), data storage drives using various computer-readable media, network interface controllers (NICs) having physical layer elements for coupling to a network link (e.g., Ethernet), or other devices including user interface devices. For example, a PCIe device may be included in the host device 110, which is coupled to a PCIe slot on the motherboard of the host device 110. Such devices are referred to as local peripheral devices. The host device 110 also includes various software executed by the processing system of the host device 110.This software typically includes an operating system, user applications, device drivers, user data, hypervisor software, telemetry software, and various other software elements.

[0021] The fabric control system 130 comprises the control and interaction elements for the network switch 131 and the fabric switch 132. The fabric control system 130 may be included in a fabric management module or a top-of-rack (ToR) switch chassis. The fabric control system 130 comprises a management operating system (OS), an operator control interface, and various other elements as shown in Figures 4-7. The fabric control system 130 may comprise one or more microprocessors and other processing circuitry that retrieves and executes software from an associated storage system (not shown). The fabric control system 130 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of the fabric control system 130 include general purpose central processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof. In some examples, fabric control system 130 comprises an Intel® microprocessor, an Apple® microprocessor, an AMD® microprocessor, an ARM® microprocessor, an FPGA, an ASIC, an application specific processor, or other microprocessor or processing element. Fabric control system 130 includes at least one network interface and at least one fabric interface. The network interface comprises a network stack and associated physical layer interfaces used to communicate with network switch 131 and control elements of network switch 131 as well as devices coupled to ports 133 of network switch 131. The fabric interface comprises a communication link subsystem used to communicate with fabric switch 132 and between peripheral devices coupled to ports 134 of fabric switch 132. The fabric interface may comprise one or more PCIe interfaces or other suitable fabric interfaces.

[0022] Network switch 131 includes network ports 133 that provide switched network connections to host devices, as shown for network link 150. Network switch 131 includes various network switching circuitry for communicatively linking individual ports to other ports based on traffic patterns, addressing, or other traffic properties. In one example, network switch 131 comprises an Ethernet or Wi-Fi (802.11xx) switch that supports wired or wireless connections, which may refer to any of a variety of available network communication protocol standards and bandwidths, such as 10BASE-T, 100BASE-TX, 1000BASE-T, 10GBASE-T (10 GB Ethernet), 40GBASE-T (40 GB Ethernet), Gigabit (GbE), Terabit (TbE), 200GbE, 400GbE, 800GbE, or various other wired and wireless formats and speeds.

[0023] Fabric switch 132 includes fabric ports 134. Fabric ports 134 couple to peripheral devices 140 via associated fabric links, which typically comprise point-to-point multi-lane serial links. In FIG. 1, fabric ports 134 are shown coupled to sets or groups of peripheral devices 140, which include individual peripheral devices coupled to a backplane having a shared fabric connection. For example, backplane 144 couples to fabric port 134 via link 160, backplane 145 couples to fabric port 134 via link 161, and backplane 146 couples to fabric port 134 via link 162. It should be understood that although FIG. 1 shows individual peripheral devices organized by device type, any arrangement of peripheral devices may instead be utilized and individual peripheral devices may be coupled to fabric ports 134 without the use of a shared backplane.

[0024] The fabric switch 132 comprises at least a portion of a communication fabric formed from one or more fabric switch elements. One example of a communication fabric utilizes a PCIe fabric. The PCIe fabric is formed from PCIe switch circuitry, which may be referred to as a PCIe crosspoint switch. The PCIe switch circuitry may be configured to logically interconnect various PCIe links based at least on the traffic carried by each PCIe link. Domain-based PCIe signaling distribution may be included that allows for separation of PCIe ports of the PCIe switch according to operator-defined groups. The operator-defined groups may be managed by the fabric control system 130, which logically assembles and separates components into logical groups. The fabric control system 130 may control the PCIe switch circuitry via a fabric interface coupled to the PCIe fabric (i.e., to the fabric switch 132) to change the logical division or separation between the PCIe ports and thus change the configuration of the grouping of the physical components. In addition to or in the alternative to the domain-based separation, each PCIe switch port may be a non-transparent (NT) port or a transparent port. While NT ports allow some logical isolation between endpoints, like bridges, transparent ports do not allow logical isolation and have the effect of connecting endpoints in a purely switched configuration. Access via one or more NT ports may involve an additional handshake between the PCIe switch and the initiating endpoint to select a particular NT port or to enable visibility via the NT port.

[0025] Advantageously, this domain-based separation, or NT port-based separation, allows physical components (i.e., CPUs, CoPUs, data storage or memory devices, or NICs) to be coupled to a shared or common fabric, but only have immediate visibility to the components involved via separation / division into groups or units. Thus, logical division between communication fabrics may be used to achieve grouping between multiple physical components. This division is inherently scalable and may be dynamically changed as needed by the fabric control system 130 or other control elements. In some examples, each PCIe switch comprises a PLX / Broadcom / Avago PEX series chip, such as a PEX8796 24-port, 96-lane PCIe switch chip, a PEX8725 10-port, 24-lane PCIe switch chip, a PEX97xx chip, a PEX9797 chip, or other PEX87xx / PEX97xx chips.

[0026] The peripheral device 140 may include various co-processing units (CoPUs) or data processing accelerators, such as graphics processing units (GPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), or application specific integrated circuits (ASICs), as described above. The peripheral device 140 may also include data storage devices, such as solid-state storage devices (SSDs) including flash memory or other media types, hard drives (HDDs) including rotating magnetic media, magnetoresistive random access memories (MRAMs), or other data storage devices with various media and interface types. The peripheral device 140 may include fabric-coupled memory devices, such as dynamic random access memories (DRAMs), static random access memories (SRAMs), 3D XPoint memories, solid-state memory devices, MRAM devices, or various other memory devices. The peripheral device 140 may include a network interface controller (NIC) including various network interface elements, such as physical layer elements (PHYs), transport layer elements, TCP / IP traffic processing elements, routers, switches, or bridges, along with associated cable connectors.

[0027] In FIG. 1, the peripheral devices 140 are coupled to individual backplanes (144-146) that comprise circuit boards and may be included in individual chassis or enclosures (as shown in FIG. 5). These backplanes may include connectors or sockets to couple each peripheral device to a communication fabric while providing multiplexing or sufficient ports for simultaneous connections per device. Various fabric switch circuits may be included in the backplanes to couple the connected devices through shared or common fabric links (160-162). Other configurations are possible, such as individually coupled peripheral devices, multiple fabric links per backplane, or other configurations. In FIG. 1, a plurality of data processing accelerators are coupled to the backplane 144, such as GPUs, and communicate with the fabric switch 132 via fabric link 160. A second plurality of data processing accelerators are coupled to the backplane 145, such as FPGAs, and communicate with the fabric switch 132 via fabric link 161. A third plurality of devices, i.e., data storage or RAM devices, are coupled to the backplane 146 and communicate with the fabric switch 132 via fabric link 162.

[0028] Turning now to an exemplary set of operations of the elements of Figure 1, operations 200 are illustrated in Figure 2. Operations 200 may be subdivided into those performed by fabric control system 130 (labeled switch (SW) operations) and those performed by host device 110 (labeled host operations), although various operations may be performed by different devices than those listed in Figure 2.

[0029] In operation 201, the fabric control system 130 discovers fabric-bound peripheral devices. This discovery process may include applying power to the peripheral devices 140 and detecting initialization messaging provided by the peripheral devices 140, such as device identifiers (IDs), vendor IDs, memory space requests / requirements, addressing, and other information. This information may be provided automatically by the peripheral devices upon power-up or reset, or may be queried by the fabric control system 130 using various protocol-specific command or control traffic. Regardless of the technique used to discover individual peripheral devices over the links 160-162, the fabric control system 130 may establish and maintain one or more data structures that list these properties and identifiers of the peripheral devices. One exemplary data structure is shown in FIG. 1 as "peripheral devices 135." The fabric control system 130 may update this data structure over time to add or remove peripheral devices as their operational status, presence, or availability changes.

[0030] A similar technique is implemented by the fabric control system 130 for network-coupled devices in operation 202. Thus, the fabric control system 130 performs a network discovery process to discover network-coupled host devices, such as the host device 110. In some examples, the fabric control system 130 may discover many network devices, but catalogs only those host devices that support the network-coupled peripheral device mode through the use of specialized software deployed on such host devices. This specialized software, such as a driver or kernel-level module, may report a status to the fabric control system 130 that includes a mode support indication to the fabric control system 130 along with a host-related network address, network port, or network socket. A communication protocol or handshake operation may be utilized between the host device 110 and the fabric control system 130, which indicates the host IP address, port number, and login / password of the host device 110, so that the fabric control system 130 may issue an attach / detach operation to the host device 110. Regardless of the technique used to discover individual host devices over link 150, fabric control system 130 may establish and maintain one or more data structures that list these properties and identifiers of the host devices along with corresponding IP addresses, port numbers, and login / password parameters. One exemplary data structure is shown in FIG. 1 as "host devices 134." Fabric control system 130 may update this data structure over time to add or remove host devices in response to changes in the operational status, presence, or availability of the host devices.

[0031] In response to discovering a host device 110 that may support the network-coupled peripheral device mode, the fabric control system 130 presents (203) available peripheral devices to the host device 110 via the network link 150. This list may be obtained from the "peripheral devices 135" maintained by the fabric control system 130. A subset of the total number of peripheral devices in the "peripheral devices 135" may be provided based on the host device 110 requesting only certain types of devices. The PoF system 115 of the host device 110 may then make a selection (204) of the peripheral devices to be presented to the fabric control system 130 via the network interface 111. This selection indicates an identification or identifier of the peripheral device that the host device 110 wishes to couple via the network link 150, such as a device ID or vendor ID used to identify the peripheral device on the local peripheral interconnect interface 112.

[0032] In response to the selection by the host system 110, the fabric control system 130 presents one or more commands instructing the PoF system 115 to attach (205) the peripheral device to the host system 110. Considering the peripheral device 147 as an example of a selected peripheral device, in response to the command issued by the fabric control system 130, the PoF system 115 performs a hot plug or attach process to instantiate the peripheral device 147 to the local peripheral interconnect interface 112 of the host device 110. This hot plug or attach process involves emulating to the local peripheral interconnect interface 112 that a new hardware device is physically present in a slot or connector of the host device 110 and triggering a start point process when the hardware device is not actually physically present in the slot or connector of the host device 110. The PoF system 115 invokes the start point process in the local peripheral interconnect interface 112 without the use of a corresponding hardware interrupt or hardware indicator that would normally result from a physical plug-in or power-on of a peripheral device to the host device 110. This may also include modifying the local peripheral interconnect interface 112 to accept interrupts or attach commands from the PoF system 115, rather than just from hardware modifications. The PoF system 115 also emulates a hardware plug-in process by providing an address space description to the local peripheral interconnect interface 112 to access the peripheral device 147 as a local peripheral device coupled to the host system 110. In an example where the local peripheral interconnect interface 112 comprises a PCIe subsystem of the host device 110, an "attach" command may be issued by the PoF system 115 indicating a vendor ID and a device ID to the PCIe subsystem. The PCIe subsystem may also request or provide a memory addressing location for memory-mapped access to the peripheral device 147.The PoF system 115 may emulate these addressing locations and the structural properties of such locations (eg, buffer sizing and formatting) to further emulate the behavior of the peripheral device 147.

[0033] Once instantiated on the local peripheral interconnect interface 112 of the host device 110, the device driver of the host device 110 may interface with the peripheral device 147 through the local peripheral interconnect interface 112 to provide access to operating system processes, user applications, kernel resources, and various other interfaces. Any tools and libraries associated with the peripheral device driver function similarly for a local peripheral device physically coupled to the host device 110 or for a peripheral device mapped over a network link through the PoF system 115. Thus, the device driver for the peripheral device 147 is typically unaware that the peripheral device 147 is not locally connected to the host device 110. User applications, operating systems, kernel resources, hypervisors, etc. may all interface with the peripheral device 147 as they would normally do when locally connected, emulating the behavior of the local hardware interface of the peripheral device 147 to the local peripheral interconnect interface 112 via the PoF system 115 (206). This emulation may include the above behaviors for instantiation and for subsequent handling of communications between the local peripheral interconnect interface 112 and the network interface 111. These communications may include configuration traffic, command and control handshakes, input / output traffic, and read / write traffic or data transfers between the host device 110 and the peripheral device 147. Thus, the PoF system 115 acts as an interworking unit for the peripheral device 147's traffic between the local peripheral interconnect interface 112 and the network interface 111. The PoF system 115 interfaces with the network stack of the network interface 111 to send and receive traffic to and from the peripheral device 147 via the network link 150.This may include intercepting host-originated traffic for peripheral device 147 from local peripheral interconnect interface 112 and interpreting the host-originated traffic for conversion or re-bundling from a native format (e.g., PCIe frames or the like of local peripheral interconnect interface 112) to an IP packet format or an Ethernet frame format suitable for transport over network interface 111. PoF system 115 then routes the host-originated traffic in packet format through network interface 111 for delivery to peripheral device 147 over link 150.

[0034] Fabric control system 130 may receive traffic originating from a host device via link 150. Because network link 150 couples to host device 110 and fabric link 160 couples to peripheral device 147, interworking operations between the two connection types are established. Fabric control system 130 provides interworking (207) between at least network link 150 and fabric link 160. This may comprise protocol translation, in which network frames or IP packets are stripped of various network protocol specific headers, and then the payload data is repackaged or encapsulated into frames or packets (or other fabric native data crumbs) suitable for transfer over fabric link 160. For example, when fabric link 160 comprises a PCIe link, frames suitable for transfer over the PCIe link are created by fabric control system 130 based on data received over link 150. This traffic is then forwarded over link 160 for delivery to peripheral device 147. Various read, write, input / output, control, command, or other traffic may be processed in this manner to forward transactions to peripheral device 147 that originate at host device 110. Similarly, the reverse operation may occur for transactions and responses originating from peripheral device 147 for forwarding to host device 110. Specifically, frames or packets (or other fabric-native data crumbs) received over link 160 are converted into frames or packets for forwarding over link 150 by fabric control system 130.

[0035] Upon receipt by the host system 110 via link 150 and network interface 111, these network frames or packets are processed by the network stack of the network interface 111. The PoF system 115 communicates with the network stack of the network interface 111 to interpret these received network frames or packets for conversion to a native format suitable for the local peripheral interconnect interface 112. The local peripheral interconnect interface 112 provides communications in the native format to various software elements of the host device 110, such as device drivers that interface with user-level applications. Thus, the host device 110 may use the peripheral device 147 as if it were locally coupled to a local bus or connector of the host device 110.

[0036] When the peripheral device 147 is no longer needed by the host device 110, the PoF system 115 may operate upon instruction or command by the fabric control system 130 to remove the peripheral device 147 from the host device 110. This removal may be accomplished by the fabric control system 130 indicating a detach command to remove 208 the peripheral device 147 from the host device 110 and return the peripheral device 147 to the available peripheral device pool. The PoF system 115 may effect removal of the instantiation of the peripheral device 147 from the local peripheral interconnect interface 112 by at least emulating a hardware device removal process or a "hot unplug" operation to the local peripheral interconnect interface 112. This hot unplug or remove process includes emulating to the local peripheral interconnect interface 112 that the hardware device is no longer physically present in a slot or connector of the host device 110 and triggering an exit point process. The PoF system 115 invokes an end point process in the local peripheral interconnect interface 112 without a corresponding hardware interrupt or hardware indicator that would normally result from a physical unplug or power off of the peripheral device to the host device 110. This may also include modifying the local peripheral interconnect interface 112 to accept an interrupt or detach command from the PoF system 115, rather than only from a hardware change. The address space description established for the peripheral device 147 in the host device 110 may be torn down or removed in the local peripheral interconnect interface 112. In another example, a "detach" command may be issued by the PoF system 115, which indicates the PCIe vendor ID and PCIe device ID of the peripheral device 147 to the local peripheral interconnect interface 112.

[0037] As discussed above, once the peripheral device 147 is removed, it may be returned to a pool or peripheral devices where it remains in an inactive, removed state until needed by a host device or other device. The attachment process may then proceed as described above. Further description of the pool of components or peripheral devices is provided in FIG. 5. Additionally, instead of a GPU or device similar to the peripheral device 147, other fabric-coupled devices may be attached / detached as described in the operations of FIG. 2, such as FPGAs, SSDs, NICs, memory devices, user interface devices, or other peripheral devices.

[0038] FIG. 3 is presented to illustrate a detailed structure and operation of the host device. The elements of FIG. 3 may be applied to the host device 110 of FIG. 1, but it should be understood that the host device 110 may utilize other structures and operations. FIG. 3 includes a host device 300 as an example of a computer device, server, computer system, blade server, etc. The host device 300 may include one or more network links, each having an associated network address. For example, as described herein, a first network link 380 may couple the host device 300 to a network switch for coupling peripheral devices. Another network link 381 may communicate with other host devices, additional networks, or the Internet, among various other networks and endpoints. The example described with respect to FIG. 3 also utilizes TCP / IP style networking for communicating with the host device 300, and a PCIe fabric for communicating with peripheral devices.

[0039] The host device 300 includes a user space and a kernel space. The kernel space 301 is a software system that includes core operating system (OS) elements such as the OS kernel, device drivers, hardware interface subsystem, network stack, memory management subsystem, machine clock / time module, and other low-level elements that are utilized to manage the resources of the host device 300 between user-level and kernel-level software, in addition to acting as an interface between the hardware components and user applications. The user space 302 includes user applications, tools, games, graphical or command line user interface elements, and other similar elements. Typically, the user space elements interface with the device driver elements of the kernel space 301 through application programming interfaces (APIs) or other software-defined interfaces to share access to the low-level hardware elements among all the user software, such as network controllers, graphics cards, audio devices, video devices, user interface hardware, and various communication interfaces. These device driver elements receive user-level traffic and interact with the hardware elements that ultimately drive link layer communication, data transfer, data processing, logic, or other low-level functions.

[0040] Within kernel space 301, host device 300 includes network stack 330, peripheral over fabric (PoF) unit 320, PCI / PCIe module 340, and device driver 350. Other kernel space elements are omitted for clarity and to focus on kernel level elements relevant to the operation of this specification. User space 302 includes user commands 360 and user applications 361. Network stack 330 comprises a TCP / IP stack and includes various layers or modules typical of a network stack, although some elements are omitted for clarity. Ethernet driver 334 includes the functions of link layer, media access controller (MAC) addressing, Ethernet frame processing, and interfacing with a network interface controller (not shown) that handles physical layer operations and structures. IP module 333 performs packet processing, IP addressing, and internetwork operations. The TCP / UDP module 332, in addition to interfacing between the user application data structures and the IP module 333, packetizes the user data and handles error correction and retransmission, acknowledgment of transfers, etc. The socket layer 331 interfaces with the user applications and other components of the host device 300 and acts as an endpoint for packetized communication. Individual sockets can be established, each of which handles a specific communication purpose, type, protocol, or other communication separation. Several sockets can be established by the network stack, each of which can act as an endpoint for a different communication type. For TCP / UDP, a socket is typically identified by an IP address and a port number, and a host device may have many such port numbers for multiple IP addresses, each with its own set of port numbers, in addition to a single IP address. Thus, many sockets may be established, each with a specific purpose.User level applications, user processes, or even kernel level processes, modules, and elements may interface with the network stack through specific sockets.

[0041] In operation, in response to attach / detach commands forwarded by the fabric management / control entity (and directed to the sockets), the PoF unit 320 may establish these goals by invoking hot plug / unplug functions of the PCI / PCIe module 340 and emulating the hardware behavior for these functions. The PoF unit 320 interfaces with the socket layer 331 to forward and receive packets carrying traffic associated with peripheral devices that may be remotely located from the host device 310. Instead of directly interfacing with the socket layer 331, the PoF unit 320 may instead use a TCP Offload Engine (TOE) stack and remote direct memory access (RDMA) for a particular network interface controller vendor type. The socket layer 331, or equivalent as described above, is identified by an IP address and port number and is typically dedicated to traffic associated with a particular peripheral device or all remote peripheral devices for the host device 310. A username / password or other security credentials may be passed along with packets received by the socket layer 331. The PoF unit 320 has "hooks" or software interface functions to communicate with the socket layer 331. Packets arrive from peripheral devices through the network stack 303 and are interpreted by the PoF unit 320, which then converts the communications into a format suitable for the PCI / PCIe module 340. The packets received by the interworking unit 320 may include PCIe device state information of the peripheral devices. The PCI / PCIe module 340 receives these communications from the PoF unit 320 as if they originated from local peripheral devices of the host device 110. Thus, the PoF unit 320 emulates the behavior of local peripheral devices to the PCI / PCIe module 340, and such peripheral devices appear local to the PCI / PCIe module 340.The device drivers, device tools or toolsets, and device-centric libraries function similarly for locally connected or remote PCIe devices mapped through the PoF unit 320. To achieve this emulation of local devices, the PoF unit 320 may establish several functions or libraries that present targets for the PCI / PCIe module 340 to communicate with for I / O transactions, configuration transactions, reads / writes, and various other communications.

[0042] Advantageously, user applications may interact with peripheral devices located remotely from the host device 110 using standard device drivers 350 that interface with the PCI / PCIe module 340. Communications issued by the PCI / PCIe module 340 that are normally intended for local hardware devices are intercepted by the PoF unit 320 and interpreted for forwarding via the network stack 330 and network link 380. When a GPU peripheral device is utilized, in one example, a graphics driver may be utilized without modification by a user application, such as a machine learning, deep learning, artificial intelligence, or gaming application, even though the GPU is located remotely with respect to the host device 300.

[0043] FIG. 4 is presented to illustrate a detailed structure and operation of the fabric control system. The elements of FIG. 4 may be applied to the fabric control system 130 of FIG. 1, but it should be understood that the fabric control system 130 may utilize other structures and operations. FIG. 4 includes a fabric control system 400 as an example of a management node, computer device, server, computer system, blade server, etc. The fabric control system 400 may include one or more network links, each having an associated network address. For example, as described herein, a first network link 480 may couple to a network switch, which in turn couples to a host device for coupling peripheral devices. The example described with respect to FIG. 4 also utilizes TCP / IP style networking for communicating with the fabric control system 400 and a PCIe fabric for communicating with a pool of peripheral devices.

[0044] Fabric control system 400 includes a user space and a kernel space. Kernel space 401 is a software system that includes core operating system (OS) elements such as the OS kernel, device drivers, hardware interface subsystem, network stack, memory management subsystem, machine clock / time module, and other low-level elements utilized to manage resources of fabric control system 400 between user-level and kernel-level software in addition to acting as an interface between hardware components and user applications. User space 402 includes user applications, tools, telemetry, event handlers, user interface elements, and other similar elements. Typically, user space elements interface with device driver elements of kernel space 401 through application programming interfaces (APIs) or other software-defined interfaces to share access to low-level hardware elements among all user software, such as network controllers, fabric interfaces, sideband communication / control interfaces, maintenance interfaces, user interface hardware, and various communication interfaces. These device driver elements receive user-level traffic and interact with hardware elements that ultimately drive link layer communication, data transfer, data processing, logic, or other low-level functions.

[0045] Within kernel space 401, fabric control system 400 includes network stack 430, PCI / PCIe module 440, fabric device driver 450, and fabric interface 460. Other kernel space elements are omitted for clarity and to focus on kernel level elements relevant to the operation of this specification. Network stack 430 comprises a TCP / IP stack and includes various layers or modules typical of a network stack, although some elements have been omitted for clarity. Ethernet driver 434 includes the functions of link layer, media access controller (MAC) addressing, Ethernet frame processing, and interfacing with a network interface controller (not shown) that handles physical layer operations and structures. IP module 433 performs packet processing, IP addressing, and internetwork operations. TCP / UDP module 432 packetizes user data and handles error correction and retransmission, acknowledgement of transfers, etc., in addition to interfacing between user application data structures and IP module 433. The socket layer 431 interfaces with user applications and other components of the fabric control system 400 and acts as an endpoint for packetized communication. Individual sockets can be established, each of which handles a specific communication purpose, type, protocol, or other communication separation. Several sockets can be established by the network stack, each of which can act as an endpoint for a different communication type. In the case of TCP / UDP, a socket is typically identified by an IP address and a port number, and a host device may have many such port numbers for multiple IP addresses, in addition to a single IP address, each with its own set of port numbers. Thus, many sockets may be established, each with a specific purpose. User-level applications, user processes, or even kernel-level processes, modules, and elements may interface with the network stack through specific sockets.

[0046] PCI / PCIe module 440 comprises a PCI / PCIe subsystem that includes the equivalent of a protocol stack for a PCI / PCIe link. PCI / PCIe module 440 interfaces with physical layer elements, such as fabric interface 440, and also presents a software / programming interface for configuration handler 412. Fabric device driver 450 provides functionality similar to PCI / PCIe module 440, and interfaces user space elements (e.g., command processor 414) with fabric interface 412. Fabric device driver 450 provides a path for command and control of the fabric itself, such as for logical partitioning / isolation of peripheral devices, for handling of peripheral device reads, writes, configuration, I / O related traffic by PCI / PCIe module 440. Fabric interface 460 comprises a fabric chip or fabric switch circuit that may provide one or more physical fabric links for coupling to a pool of peripheral devices coupled via PCI / PCIe module 440, fabric device driver 450, and associated PCIe links, or to further fabric chips or fabric switch circuits that provide portions of the fabric and further fabric links.

[0047] The user space 402 includes a fabric control element 410 further comprising a monitor 411, a configuration handler 412, an event handler 413, a command processor 414, and a user interface 415. The command processor 414 communicates with the fabric driver 450 and controls the communication fabric utilized to establish logical partitioning between peripheral devices coupled to the fabric, isolating the selected peripheral device from other peripheral devices. For example, when a peripheral device is selected to be attached to a host device, the command processor 414 can isolate the selected peripheral device from other peripheral devices, such that a logical partition in the fabric includes the selected peripheral device to provide routing of communication / traffic to and from the selected peripheral device without interfering with the rest of the peripheral devices. When the selected peripheral device is removed, the logical isolation is removed or discarded and the peripheral device can be placed in a pool of unused devices. The user interface 415 receives operator commands to control the addition or removal of peripheral devices to and from the host device. The user interface 415 can display or display a list of hosts and peripheral devices along with their associated status or telemetry. The user interface 415 may display or indicate which peripheral devices are associated with which host devices. The user interface 415 may display or indicate traffic histograms, logs, faults, alerts, and various other telemetry and status. The user interface 415 may comprise a terminal interface, an application programming interface (API), a representational state transfer (REST) ​​interface, or a RestAPI, a web interface, a WebSocket interface, among other types of user interfaces, including those that are software, hardware, virtualized, or transmitted over various intermediate links. The event handler 413 initiates attach / detach and device discovery operations to the host device.The configuration handler 412 interacts with traffic between fabric-coupled peripheral devices and network-coupled host devices. The configuration handler 412 interfaces with PCI / PCIe modules 440 for fabric communication and with network stack 430 for network communication. The configuration handler 412 may interact with frame or packet formats, sizes, and types for transporting fabric communication over network links and network communication over fabric links. The configuration handler 412 interfaces with network stack 430 through socket layer 431 through specific sockets indicated by at least IP addresses and port numbers. The monitor 414 monitors various telemetry, operations, logs, and status of the fabric control system 400. The monitor 414 may maintain data structures indicating indicators or identities of peripheral devices in addition to indicating host devices and associated sockets (IP addresses and ports). The monitor 414 may maintain logs and data structures in a computer-readable medium.

[0048] During operation, the event handler 413 may initiate operations to discover compatible host devices coupled through the network interface and to discover peripheral devices coupled to the communications fabric. The event handler 413 may instruct the command processor 414 to discover peripheral devices through the fabric driver 450 and the fabric interface 460. The event handler 413 may instruct the configuration handler 412 to discover host devices through the network stack 430. Socket information of compatible host devices may be determined and stored for later use. The fabric interface 460 scans the communications fabric to determine which peripheral devices and resources are available. The command processor 414 may form a pool of free peripheral devices and an indication of assigned peripheral devices, store the device / vendor IDs of the peripheral devices, and store an indication of the PCIe addressing and buffer structure or characteristics of each peripheral device.

[0049] A request to add a peripheral device to a host device is received, such as via a user interface 415, and the command processor 414, via the fabric driver 450, may isolate the selected peripheral device to a logical partition of the communications fabric. This may trigger the event handler 413 to initiate an attach command with the peripheral device information (e.g., vendor ID and device ID) to the affected host device, which then attaches the peripheral device as described herein. From here, communications between the host device and the peripheral device are mediated using the configuration handler 412, which interprets and exchanges traffic between the socket layer 431 and the PCI / PCIe module 440. At some point, it may be desirable for an attached peripheral device to be removed or detached from a particular host device. The event handler 413 may detect these detach events, such as by the user interface 415 or received from the host device. The event handler 413 then issues a detach command through the configuration handler 412 to detach the affected peripheral device, as described herein. The command processor 414 may remove the logical partitioning of the removed peripheral device and return the removed peripheral device to an inactive state or to a free pool of peripheral devices for later use.

[0050] FIG. 5 is a system diagram illustrating a computer system 500. The computer system 500 may comprise elements of the computer system 100 of FIG. 1, the host device 300 of FIG. 3, or the fabric control system 400 of FIG. 4, although variations are possible. The computer system 500 comprises a rack-mounted arrangement of multiple modular chassis. One or more physical enclosures, such as modular chassis, may further be included in a shelf or rack unit. The chassis 510, 520, 530, 540, and 550 are included in the computer system 500 and may be mounted in a common rack-mounted arrangement in one or more data centers or may span multiple rack-mounted arrangements. Within each chassis, modules are mounted to a shared PCIe switch along with various power system, structural support, and connector elements. A certain number of components of the computer system 500 may be inserted or installed in a physical enclosure, such as a modular framework in which modules may be inserted and removed as needed by a particular end user. An enclosed modular system may include a physical support structure and enclosure that contains the circuits, printed circuit boards, semiconductor systems, and structural elements. Modules comprising components of computer system 500 are insertable into and removable from rack-mounted enclosures. In some examples, the elements of FIG. 5 are included in a "U" style chassis for mounting within a larger rack-mounted environment. It should be understood that the components of FIG. 5 can be included in any physical mounting environment and need not include associated enclosures or rack-mounted elements.

[0051] The chassis 510 comprises a management module or a top-of-rack (ToR) switch chassis, and comprises a management processor 511, an Ethernet switch 516, and a PCIe switch 560. The management processor 511 comprises a management operating system (OS) 512, a user interface 513, and an interaction unit 514. The management processor 511 is coupled to the PCIe switch 560 via one or more PCIe links comprising one or more PCIe lanes. The management processor 511 is coupled to the Ethernet switch 516 via one or more network links via a network interface controller.

[0052] PCIe switch 560 is coupled to PCIe switches 561-564 in other chassis in computer system 500 via one or more PCIe links. These one or more PCIe links are represented by PCIe inter-module connections 565. PCIe switches 560-564 and PCIe inter-module connections 565 form a PCIe fabric that communicatively couples all of the various physical computer elements of FIG. 5. In some examples, management processor 511 may communicate with elements of the PCIe fabric via special management PCIe links or sideband signaling (not shown), such as an inter-integrated circuit (I2C) interface, to control the operation and partitioning of the PCIe fabric. These control operations may include configuring and disassembling peripheral devices, changing logical partitioning within the PCIe fabric, monitoring telemetry of the PCIe fabric, controlling power up / down operation of modules on the PCIe fabric, updating firmware of various circuits that comprise the PCIe fabric, and other operations.

[0053] The Ethernet switch 516 includes network ports that provide switched network connections to attached devices, as shown for network link 566. The Ethernet switch 516 includes various network switching circuitry for communicatively linking individual ports to other ports based on traffic patterns, addressing, or other traffic properties. In one example, the Ethernet switch 516 comprises an Ethernet or Wi-Fi (802.11xx) switch that hosts wired or wireless connections, which may refer to any of the various network communication protocol standards and bandwidths available, such as 10BASE-T, 100BASE-TX, 1000BASE-T, 10GBASE-T (10 GB Ethernet), 40GBASE-T (40 GB Ethernet), Gigabit (GbE), Terabit (TbE), 200GbE, 400GbE, 800GbE, or various other wired and wireless formats and speeds.

[0054] Chassis 520 includes a number of CPUs 521-525, each coupled to a PCIe fabric via a PCIe switch 561 and associated PCIe links (not shown). Chassis 530 includes a number of GPUs 531-535, each coupled to a PCIe fabric via a PCIe switch 562 and associated PCIe links (not shown). Chassis 540 includes a number of SSDs 541-545, each coupled to a PCIe fabric via a PCIe switch 563 and associated PCIe links (not shown). Chassis 550 includes a number of NICs 551-555, each coupled to a PCIe fabric via a PCIe switch 564 and associated PCIe links (not shown). Each chassis 520, 530, 540, and 550 may include various modular bays for mounting modules comprising a corresponding element of each CPU, GPU, SSD, or NIC. Power systems, monitoring elements, internal / external ports, loading / removal hardware, and other associated functionality may be included in each chassis. Further descriptions of the individual elements of chassis 520, 530, 540, and 550 are included below. In addition to the component types discussed above, other component types may be included as well, such as FPGAs, CoPUs, RAM, memory devices, or other components.

[0055] Once the various CPU, GPU, SSD, or NIC components of computer system 500 are installed in their associated chassis or enclosures, the components may be coupled via a PCIe fabric and logically isolated into any number of separate, arbitrarily defined arrangements and attached to a host device. These arrangements may consist of a selected number of CPUs, GPUs, SSDs, and NICs, including zero of any type of module. An example of a host device 501 is shown in FIG. 5, which includes an attached remote peripheral GPU 531. GPU 531 is attached to host device 501 via a network link, shown for link 566, in addition to using logical partitioning within the PCIe fabric, shown by logical domains 570. The PCIe fabric may be configured by management processor 511 to selectively route traffic between selected peripheral devices for a particular host device, while maintaining logical isolation between components not included in the particular host device. In this manner, a distributed and flexible "bare metal" configuration may be established between components of computer system 100. Individual peripheral devices may be arranged according to a particular user identity, host device identity, running job, or usage policy.

[0056] In some examples, the management processor 511 may provide installation or removal of peripheral devices and host devices through one or more user interfaces or job interfaces. For example, the management processor 511 may provide a user interface 513 that may present an indication of peripheral components to be installed, an indication of available host devices, and software and configuration information. In some examples, the creation user interface may provide a template for installing a predefined arrangement of peripheral devices to a host device based on a use case or usage category. For example, the user interface may provide suggested templates or configurations for a game server unit, an artificial intelligence learning computation unit, a data analysis unit, and a storage server unit. For example, a game server unit or an artificial intelligence processing template may specify additional graphics processing resources when compared to a storage server unit template. Furthermore, the user interface may provide customization of the template or arrangement configuration and options for a user to create an arrangement template from a component type arbitrarily selected from a list or category of components.

[0057] In additional examples, the management processor 511 may provide policy-based dynamic adjustments to placement during operation. In some examples, the user interface 513 allows a user to define policies for adjustments to the host device's configuration information during operation, as well as adjustments to peripheral devices assigned to the host device. In one example, during operation, the management processor 511 may analyze telemetry data to determine current resource utilization by the host device. Based on the current utilization, the dynamic adjustment policy may specify allocating or removing general processing resources, graphics processing resources, storage resources, networking resources, memory resources, and the like, to or from the host device. For example, the telemetry data may indicate that the current utilization level of the host device's allocated storage resources is approaching a threshold level, and additional storage devices may be allocated to the host device.

[0058] Although a PCIe fabric is described in the context of FIG. 5, the management processor 511 may provide control and management of multiple protocol communication fabrics and communication fabrics other than PCIe. For example, the management processor 511 and PCIe switch devices of the PCIe fabric may provide communication coupling of physical components using multiple different implementations or versions of PCIe and similar protocols. For example, different PCIe versions (e.g., 3.0, 4.0, 5.0, and later) may be utilized for different physical components in the same PCIe fabric. Additionally, next generation interfaces may be utilized, e.g., Gen-Z, CCIX, CXL, OpenCAPI, or wireless interfaces, including Wi-Fi interfaces or cellular wireless interfaces. Also, while FIG. 5 uses PCIe, it should be understood that PCIe may not be present and a different communication link or bus may be utilized instead, such as NVMe, Ethernet, SAS, FibreChannel, Thunderbolt, SATA Express, among other interconnect, network, and link interfaces.

[0059] Referring now to the description of the components of computer system 500, management processor 511 may comprise one or more microprocessors and other processing circuitry that retrieves and executes software, such as management operating system 512, user interface 513, and interaction unit 514, from associated storage systems. Management processor 511 may be implemented in a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of management processor 511 include general-purpose central processing units, application-specific processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof. In some examples, management processor 511 comprises an Intel®, AMD® microprocessor, Apple® microprocessor, ARM® microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), application-specific processor, or other microprocessor or processing element.

[0060] A management operating system (OS) 512 is executed by the management processor 511 and provides management of the resources of the computer system 500 as well as execution of the user interface 513 and the interaction unit 514. The management OS 512 provides the functionality and operations described herein for the management processor 511, and in particular for the user interface 513 and the interaction unit 514.

[0061] The user interface 513 may present a graphical user interface (GUI), an application programming interface (API), a REST interface, a Rest API, or a command line interface (CLI), a WebSocket interface to one or more users. The user interface 513 may be utilized by an end user or administrator to configure host devices and peripheral devices, establish configurations of host devices and peripheral devices, receive status, telemetry, and other notifications regarding the operation of the host devices and peripheral devices, and perform other operations. The user interface 513 may be utilized to manage, select, and modify templates. The user interface 513 may be utilized to manage, select, and modify policies. The user interface 513 may also provide telemetry information for the operation of the computer system 500 to the user in one or more status interfaces or status views, etc. The status of various components or elements of the computer system 500 may be monitored via the user interface 513, such as CPU status, GPU status, NIC status, SSD status, Ethernet status, PCIe switch / fabric status, among others. Various performance metrics, error status can be monitored using the user interface 513.

[0062] The interaction unit 514 provides various network-to-fabric interaction functions and operations described herein along with discovery, attachment, and removal of peripheral devices with respect to a host device. Specifically, the interaction unit 514 discovers peripheral devices among the CPUs, GPUs, SSDs, NICs, and other devices in the computer system 500 coupled via a communication fabric (e.g., 565) and catalogs these devices in one or more data structures illustrated by exemplary data structure 592. The interaction unit 514 also discovers host devices coupled via network links (e.g., 566) and catalogs these devices in one or more data structures illustrated by exemplary data structure 591. The cataloging may include indicating device identifiers, vendor identifiers, addresses, ports, sockets, or other properties. The interaction unit 514 may receive operator instructions via a user interface 513 from the host device to establish an arrangement between the host device and one or more peripheral devices. The instructions may identify which host device and which peripheral device or types of peripheral devices are to be coupled. In response to the command, the interaction unit 514 initiates the attachment of the peripheral device to the host device, such as by issuing one or more attach commands to the PoF unit of the host device. The attach command alerts the PoF unit of the host device to initiate instantiation of the peripheral device to the local peripheral interconnect interface of the host device by at least emulating the behavior of the peripheral device coupled through the network interface as a local peripheral device coupled to the peripheral interconnect interface of the host system. The interaction unit 514 may then interact with traffic originating from the peripheral device on the communications fabric and carried on 565 using network link 566, and with traffic originating from the host system and carried on network link 566 using the fabric communications via link 565.The interaction unit 514 may receive instructions to detach a peripheral device from a host device and issue a detach command to the PoF unit of the host device. Once detached, the peripheral device may be returned to a pool of free devices for later use by other host devices.

[0063] An example of an attachment operation is shown in FIG. 5. To attach GPU 531 to host device 501, the interaction unit may receive the command and provide an attach command and an identifier of GPU 531 to host device 501 via a corresponding network link. Table 591 shows a target host device identifier of 001234 with an IP address of 10.10.10.1 and a corresponding socket (not shown) corresponding to a port number. Table 592 shows GPU 531 with a vendor ID and device ID provided to the target host device for attachment. To enable proper exchange of communications between GPU 531 and the target host device over the communications fabric without interference with other peripheral and host devices, logical isolation of GPU 531 may be established between GPU 531 and management processor 511 in the communications fabric. This arrangement may be referred to as a logical domain 570 that includes the Ethernet links (e.g., 566), communications fabric links, fabric switch circuitry, and elements of GPU 531. GPU 531's communications may be directed to the target host device over a PCIe link using the socket corresponding to host device 501, i.e., IP address of 10.10.10.1 and associated port number. Host device 501's communications may be directed to GPU 531 over a network link using GPU 531's device ID and vendor ID, to be intercepted and interpreted by interworking unit 514.

[0064] Multiple instances of elements 511-514 may be included in computer system 500. User commands, such as those received via a GUI, may be received by any of the management instances and forwarded by the receiving management instance to a handling management instance. Each management instance may have a unique or pre-assigned identifier that may aid in the delivery of user commands to the appropriate management instance. Additionally, the management processors of each management instance may communicate with each other, such as using a mailbox process or other data exchange technique. This communication may be via a dedicated sideband interface, such as an I2C interface, or via a PCIe or Ethernet interface coupling each management processor.

[0065] A number of CPUs 521-525 are included in the chassis 520. Each CPU may comprise a CPU module including one or more CPUs or microprocessors and other processing circuitry that retrieves and executes software, such as operating systems, device drivers, and applications from an associated storage system. Each CPU may be implemented in a single processing device, but may be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of each CPU include general-purpose central processing units, application-specific processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof. In some examples, each CPU comprises an Intel® microprocessor, an Apple® microprocessor, an AMD® microprocessor, an ARM® microprocessor, a graphics processor, a computational core, a graphics core, an ASIC, an FPGA, or other microprocessor or processing element. Each CPU may also communicate with other CPUs, such as CPUs in the same storage assembly / enclosure or another storage assembly / enclosure, via one or more PCIe interfaces and a PCIe fabric.

[0066] Multiple GPUs 531-535 are included in chassis 530, which may represent any type of CoCPU. Each GPU may comprise a GPU module including one or more GPUs. Each GPU includes graphics processing resources that may be allocated to one or more host devices. A GPU may include graphics processors, shaders, pixel rendering elements, frame buffers, texture mappers, graphics cores, graphics pipelines, graphics memory, or other graphics processing and processing elements. In some examples, each GPU includes a graphics "card" that includes circuitry supporting the GPU chip. Exemplary GPU cards include nVIDIA® or AMD® graphics cards that include graphics processing elements along with various support circuits, connectors, and other elements. In further examples, other styles of graphics processing units, graphics processing assemblies, or co-processing elements may be utilized, such as machine learning processing units, tensor processing units (TPUs), FPGAs, ASICs, or other specialized processors that may include specialized processing elements to focus processing and memory resources for processing of specialized data sets.

[0067] A number of SSDs 541-545 are included in the chassis 540. Each SSD may comprise an SSD module including one or more SSDs. Each SSD includes one or more storage drives, such as a solid-state storage drive with a PCIe interface. Each SSD also includes a PCIe interface, a control processor, and power system elements. Each SSD may include a processor or control system for traffic statistics and status monitoring, among other operations. In yet another example, each SSD instead includes a different data storage medium, such as a magnetic hard disk drive (HDD), a cross-point memory (e.g., an Optane® device), a static random access memory (SRAM) device, a programmable read-only memory (PROM) device, or other magnetic, optical, or semiconductor-based storage medium, along with associated enclosures, control systems, power systems, and interface circuits.

[0068] A number of NICs 551-555 are included within chassis 550, each having an associated MAC address or Ethernet address. Each NIC may comprise a NIC module including one or more NICs. Each NIC may comprise a network interface controller card for communicating over a TCP / IP (Transmission Control Protocol (TCP) / Internet Protocol) network or for carrying user traffic, such as iSCSI (Internet Small Computer System Interface) or NVMe (NVM Express) traffic for components of an associated host device. The NICs comprise Ethernet interface equipment and may communicate over wired, optical, or wireless links. External access to components of computer system 500 may be provided over a packet network link provided by the NIC. The NICs may communicate with other components of an associated host device over associated PCIe links of a PCIe fabric. In some examples, the NICs are provided to communicate with management processor 511 over an Ethernet link. In additional examples, the NICs are provided to communicate with one or more other chassis, rack mount systems, data centers, computer platforms, communication fabrics, or other components over an Ethernet link.

[0069] In addition to the CPU, GPU, SSD, and NIC, other special purpose devices may be utilized in the computer platform. These other special purpose devices may include co-processing modules with various memory, storage, and interface components, as well as dedicated co-processing circuitry, fabric-coupled RAM devices, ASIC circuitry, or FPGA circuitry, among other circuitry. The other special purpose devices may each include a PCIe interface with one or more PCIe lanes. These PCIe interfaces may be utilized to communicate over a PCIe fabric and include other special purpose devices in one or more host devices. These other special purpose devices may include PCIe endpoint devices or PCIe host devices that may or may not have a root complex.

[0070] An FPGA device may be used as an example of another dedicated device. An FPGA device may receive processing tasks from another PCIe device, such as a CPU or GPU, and offload those processing tasks to an FPGA programmable logic circuit. An FPGA is typically initialized to a program state using configuration data, which includes various logic arrangements, memory circuits, registers, processing cores, specialized circuits, and other functions that provide specialized or application-specific circuits. In addition to changing the circuits implemented therein, an FPGA device may be reprogrammed to perform different sets of processing tasks at different times. An FPGA device may be used to perform machine learning tasks, implement artificial neural network circuits, implement custom interfaces or glue logic, perform encryption / decryption tasks, perform blockchain computation and processing tasks, or other tasks. In some examples, a CPU provides data to the FPGA to be processed by the FPGA via a PCIe interface. The FPGA may process the data to generate results and provide the results to the CPU via a PCIe interface. Two or more CPUs and / or FPGAs may be involved to parallelize tasks across two or more devices or to serially process data through two or more devices. In some examples, the FPGA arrangement may include locally stored configuration data that may be supplemented, replaced, or overwritten using configuration data stored in a configuration data storage. This configuration data may comprise firmware, programmable logic programs, bitstreams, or objects, PCIe device initial configuration data, among other configuration data described herein. The FPGA arrangement may also include SRAM or PROM devices used to perform boot programming, power-on configuration, or other functions to establish the initial configuration of the FPGA device. In some examples, the SRAM or PROM devices may be incorporated into the FPGA circuit or package.

[0071] The PCIe switches 560-564 communicate via associated PCIe links. In the example of FIG. 5, the PCIe switches 560-564 may be used to carry traffic between PCIe devices within each chassis and between each chassis. Each PCIe switch 560-564 includes a PCIe cross-connect switch for establishing switched connections between any PCIe interfaces handled by each PCIe switch. The PCIe switches described herein may logically interconnect various PCIe links of associated PCIe links based at least on the traffic carried by each PCIe link. In these examples, a domain-based PCIe signaling distribution may be included that may segregate PCIe ports of the PCIe switch according to user-defined groups. The user-defined groups may be managed by a management processor 511 that logically aggregates components into associated logical units and logically isolates the components and logical units from each other. In addition to or alternatively to domain-based separation, each PCIe switch port can be a non-transparent (NT) or transparent port. An NT port allows some logical isolation between endpoints, like a bridge, while a transparent port does not allow logical isolation and has the effect of connecting endpoints in a purely switched configuration. Access via one or more NT ports may involve an additional handshake between the PCIe switch and the initiating endpoint to select a particular NT port or to enable visibility via the NT port.

[0072] Advantageously, this NT port-based or domain-based separation allows physical components (i.e., CPU, GPU, SSD, NIC) to have visibility only to components that are included through the separation / separation. Thus, using logical separation between the PCIe fabric, groupings between multiple physical components can be achieved. This separation is inherently scalable and can be dynamically changed as needed by the management processor 511 or other control elements. The management processor 511 can control the PCIe switch circuitry that comprises the PCIe fabric to change the logical separation or separation between the PCIe ports and thus change the configuration of the groupings of the physical components. These groupings individually form computing units, and can be further grouped into clusters of many computing units / machines all attached to various host devices. Physical components can be added to or removed from a computing unit according to user commands received via a user interface, dynamically in response to load / idle conditions, dynamically in response to incoming or queued jobs to run, or preemptively due to anticipated needs, among other considerations described herein.

[0073] In a further example, memory-mapped direct memory access (DMA) conduits may be formed between individual CPU / PCIe device pairs. This memory mapping may occur on the PCIe fabric address space, among other configurations. To provide these DMA conduits on a shared PCIe fabric with many CPUs and GPUs, the logical partitioning described herein may be utilized. Specifically, NT port or domain-based partitioning on the PCIe switch may isolate individual DMA conduits between associated CPUs / GPUs. The PCIe fabric has a 64-bit address space, which allows for 264 bytes of addressable space, resulting in at least 16 exbibytes of byte-addressable memory. The 64-bit PCIe address space may be shared by all compute units or may be separated among the various compute units forming an arrangement for appropriate memory mapping to resources.

[0074] The PCIe interface supports multiple bus widths, such as x1, x2, x4, x8, x16, and x32, and each multiple of the bus width may have additional "lanes" for data transfer. PCIe also supports transfer of sideband signaling, such as associated clocks, power, and bootstrap, in addition to the System Management Bus (SMBus) interface and the Joint Test Action Group (JTAG) interface, among other signaling. PCIe may also have different implementations or versions as utilized herein. For example, PCIe version 3.0 or later (e.g., 4.0, 5.0, or later) may be utilized. Additionally, next-generation interfaces, such as Gen-Z, Cache Coherent CCIX, CXL, or OpenCAPI, may be utilized. Also, while PCIe is utilized in FIG. 5, it should be understood that a different communication link or bus may be utilized instead, such as NVMe, Ethernet, SAS, FibreChannel, Thunderbolt, SATA Express, among other interconnect, network, and link interfaces. NVMe is an interface standard for mass storage devices, such as hard disk drives and solid-state memory devices. NVMe can replace SATA interfaces for interfacing with mass storage devices in personal computer and server environments. However, these NVMe interfaces are limited to a one-to-one host-drive relationship, similar to SATA devices. In the examples described herein, a PCIe interface can be utilized to carry NVMe traffic and present a multi-drive system with many storage drives as one or more NVMe virtual logical unit numbers (VLUNs) via the PCIe interface.

[0075] Any of the links in Figure 5 may each use a variety of communication media, such as air, space, metal, optical fiber, or any other signal propagation path including combinations thereof. Any of the PCIe links in Figure 5 may comprise any number of composite PCIe links or single / multi-lane configurations. Any of the links in Figure 5 may each be direct links or may include a variety of equipment, intermediate components, systems, and networks. Any of the links in Figure 5 may each be common links, shared links, aggregated links, or may be composed of individual separate links.

[0076] A simple example of forming a compute unit and attaching it to a host device is now described. In FIG. 5, a configurable logical visibility may be provided to the host device to any / all CPUs 521-525, GPUs 531-535, SSDs 541-545, and NICs 551-555, or other physical components coupled to the PCIe fabric of computer system 500, as logically separated by the PCIe fabric. For example, any CPU 521-525 may transfer and retrieve storage data with any SSD 541-545 included in the same compute unit. Similarly, any CPU 521-525 may exchange data for processing by any GPU 531-535 included in the same compute unit. Thus, "m" SSDs or GPUs may be coupled with "n" CPUs to enable a large, scalable architecture with high levels of performance, redundancy, and density. In the graphics processing example, NT partitioning or domain-based partitioning in the PCIe fabric may be provided by one or more of the PCIe switches. This partitioning allows GPUs to interact with the desired CPU or CPUs, and two or more GPUs, such as eight GPUs, may be associated with a particular compute unit. Furthermore, dynamic GPU-compute unit relationships may be adjusted on the fly using partitioning across the PCIe fabric. Shared NIC resources may also be applied across compute units.

[0077] Figure 6 is a block diagram illustrating an implementation of a management processor 600. The management processor 600 illustrates an example of any of the control elements, control systems, interaction units, PoF units, fabric control elements, or management processors described herein, such as the PoF system 115 of Figure 1, the fabric control system 130 of Figure 1, the PoF unit 320 of Figure 3, the fabric control element 410 of Figure 4, or the management processor 511 of Figure 5. The management processor 600 includes a communication interface 601, a user interface 603, and a processing system 610. The processing system 610 includes processing circuitry 611 and a data storage system 612, which may include a random access memory (RAM) 613, but may include additional or different configurations of elements.

[0078] Processing system 610 is intended to generally represent a computer system on which at least software 620 is deployed and executed to render or otherwise perform the operations described herein. However, processing system 610 may also represent any computer system on which at least software 620 and data 630 are staged and from which software 620 and data 630 may be distributed, transmitted, downloaded, or provided to yet another computer system for deployment, execution, or further distribution. Processing circuitry 611 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of processing circuitry 611 include general purpose central processing units, microprocessors, application specific processors, and logic devices, as well as any other type of processing device. In some examples, processing circuitry 611 includes physically distributed processing devices, such as a cloud computer system.

[0079] The communication interface 601 includes one or more communication fabrics and / or network interfaces for communicating over a PCIe link, a packet network, a network such as the Internet, or the like. The communication interface may include a PCIe interface, an Ethernet interface, a serial interface, a serial peripheral interface (SPI) link, an Inter-Integrated Circuit (I2C) interface, a Universal Serial Bus (USB) interface, a UART interface, a wireless interface, or one or more local or wide area network communication interfaces that may communicate over an Ethernet or Internet Protocol (IP) link. The communication interface 601 may include a network interface configured to communicate using one or more network addresses that may be associated with different network links. Examples of the communication interface 601 include network interface controller equipment, transceivers, modems, and other communication circuitry. The communication interface 601 may communicate with a control element of a PCIe fabric or other communication fabric to establish logical partitioning within the fabric, such as via a management or control interface of one or more communication switches of the communication fabric. The communication interface 601 may communicate over the PCIe fabric to exchange traffic / communication with peripheral devices.

[0080] The user interface 603 may include a software-based interface or a hardware-based interface. Hardware-based interfaces include touch screens, keyboards, mice, voice input devices, audio input devices, or other touch input devices for receiving input from a user. Output devices such as displays, speakers, web interfaces, terminal interfaces, and other types of output devices may also be included in the user interface 603. The user interface 603 may provide output and receive input via a network interface, such as the communication interface 601. In a network example, the user interface 603 may packetize display or graphics data for remote display by a display system or computer system coupled via one or more network interfaces. The physical or logical elements of the user interface 603 may provide an alert or visual output to a user or other operator. The user interface 603 may also include associated user interface software executable by the processing system 610 that supports the various user input and output devices described above. The user interface software and user interface device, separately or in conjunction with each other and other hardware and software elements, may support a graphical user interface, a natural user interface, or any other type of user interface.

[0081] The user interface 603 may present a command line interface (CLI), an application programming interface (API), a graphical user interface (GUI), a REST interface, a Rest API, a WebSocket interface, or other interface to one or more users. The user interface may be utilized by an operator or administrator to assign assets (computing units / peripherals) to each host device. In some examples, the user interface provides an interface that allows an end user to determine one or more templates and dynamic tuning policy sets to use or customize for use in creating computing units. The user interface 603 may be utilized to manage, select, modify machine templates, change policies, etc. The user interface 603 may also provide telemetry information, such as in one or more status interfaces or status views. The status of various components or elements may be monitored via the user interface 603, such as processor status, network status, storage unit status, PCIe element status, among others. Various performance metrics, error status may be monitored using the user interface 603.

[0082] The storage system 612 and the RAM 613 may both comprise non-transitory data storage systems, although variations are possible. The storage system 612 and the RAM 613 may each comprise any storage medium that is readable by the processing circuitry 611 and that may store software and OS images. The RAM 613 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. The storage system 612 may include non-volatile storage media, such as solid-state storage media, flash memory, NAND flash or NOR flash, phase change memory, magnetic memory, or other non-transitory storage media, including combinations thereof. The storage system 612 and the RAM 613 may each be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems. The storage system 612 and the RAM 613 may each comprise additional elements, such as a controller that may communicate with the processing circuitry 611.

[0083] The software 620 or data 630 may be stored on or in the storage system 612 or RAM 613, which may comprise some other form of machine-readable processing instructions having computer program instructions, firmware, data structures, or processes that, when executed, direct the processor 600 to operate as described herein. The software 620 may reside in the RAM 613 during the execution and operation of the processor 600 and may reside in a non-volatile portion of the storage system 612 during a powered off state, among other places and states. The software 620 may be loaded into the RAM 613 during a start-up or boot procedure, as described for computer operating systems and applications. The software 620 may receive user input via the user interface 603. This user input may include user commands as well as other inputs, including combinations thereof.

[0084] The software 620 includes applications 621 and an operating system (OS) 622. The software 620 may drive the processor 600 to receive user commands to attach or detach peripheral devices to or from a host device. The software 620 may drive the processor 600 to receive, monitor, and provide telemetry to a user, telemetry data, statistics, operational data, and other data, and to modify operation according to telemetry data, policies, or other data and criteria. The software 620 may drive the processor 600 to manage peripheral and host device resources, establish domain or NT partitions between communication fabric elements, and interface with individual communication switches and control the operation of such communication switches, among other operations. The software may also include user software applications, application programming interfaces (APIs), or user interfaces. The software may be implemented as a single application or multiple applications. In general, the software, when loaded and executed on a processing system, may transform the processing system from a general-purpose device to a customized, dedicated device as described herein.

[0085] The software applications 621 may take different forms depending on the operation and device being performed by the management processor 600. For example, when the management processor 600 operates a fabric control system, an application set 640 may be deployed that includes a discovery application 641, an event application 642, a fabric interaction application 643, and a fabric user interface application 644. Alternatively, when the management processor operates a host device, an application set 650 may be deployed that includes a host interaction application 651 and a host user interface application 652. The software applications 623-629 each comprise executable instructions that may be executed by the processor 600 to operate a computer system or processing circuitry in accordance with the operations described herein.

[0086] The application set 640 includes a discovery application 641, an event application 642, a fabric interaction application 643, and a fabric user interface application 644. The discovery application 641 obtains an indication of peripheral devices available for association with a host device, the indication of the peripheral devices including a device identifier and a vendor identifier. The discovery application 641 obtains an indication of the host devices via network interfaces available for association with the peripheral devices. The discovery application 641 may store these indications in the data 630. Based on the indications, the event application 642 initiates instantiation and de-instantiation of the peripheral devices to the local peripheral interconnect interfaces of the host device. The fabric interaction application 643 intercepts host-originated traffic for peripheral devices received via the network interfaces, interprets the host-originated traffic to convert it from the network format to a fabric format suitable for transport over the communication fabric, and routes the host-originated traffic in the fabric format via the fabric interfaces for delivery to the peripheral devices. A fabric interaction application 643 intercepts peripheral originated traffic for the host device received via the fabric interface, interprets the peripheral originated traffic to convert it from the fabric format to a network format suitable for transport over the network link, and routes the peripheral originated traffic in the network format via the network interface for delivery to the host device. A fabric user interface application 644 receives operator commands to attach or detach peripheral devices to the host device and may present various information, status, telemetry, logs, etc., on the operation via various types of user interfaces.

[0087] The application set 650 includes a host interaction application 651 and a host user interface application 652. The host interaction application 561 emulates the behavior of a peripheral device coupled through a network interface as a local peripheral device coupled to a peripheral interconnect interface of a host system. The host interaction application 561 emulates a hardware plug-in process by at least providing an address space description to the local peripheral interconnect interface to access the peripheral device as a local peripheral device coupled to the host system. The host interaction application 561 removes the instantiation of the peripheral device from the local peripheral interconnect interface by at least emulating a hardware removal process in the local peripheral interconnect interface. The host interaction application 561 interfaces with the network stack of the host device to interact with the traffic of the peripheral device with the local peripheral interconnect interface. When instantiated on the local peripheral interconnect interface of the host device, a device driver of the host device may interface with the peripheral device through the local peripheral interconnect interface. The host interaction application 561 emulates the behavior of the peripheral device by at least intercepting host-originated traffic for the peripheral device from the local peripheral interconnect interface, interpreting the host-originated traffic to convert it from a native peripheral format (such as a PCIe frame or memory-mapped format) to a network format suitable for transfer over the network interface, and routing the host-originated traffic in packet format through the network interface for delivery to the peripheral device.The host interaction application 561 emulates the behavior of a peripheral device by at least receiving traffic from the peripheral device in packet format from the network interface and interpreting the traffic from the peripheral device in packet format to convert it to a native peripheral format suitable for the local peripheral interconnect interface. The host interaction application 561 initiates instantiation of the peripheral device to the local peripheral interconnect interface by at least triggering a start point process in the local peripheral interconnect interface to emulate a hardware plug-in process for a peripheral device having a local peripheral interconnect interface. The host user interface application 652 provides a host device operator with a local indication of the attachment and removal of peripheral devices and may receive operator commands to attach or remove peripheral devices, among other actions.

[0088] In addition to the software 620, other data 630 including various data structures may be stored by the storage system 612 and RAM 613. The data 630 may include templates, policies, telemetry data, event logs, or fabric status. The data 630 may include indications and identifications of peripheral devices and host devices. The data 630 may include current assignments of peripheral devices to host devices. The fabric status includes information and properties of various communication fabrics including pools of resources or pools of components, such as fabric type, protocol version, technology descriptors, header requirements, addressing information, and other data. The fabric data may include relationships between components and the particular fabrics to which they connect.

[0089] Various peripheral devices have been described herein that include data processing elements or other computer components coupled via one or more communication fabrics or networks. Various communication fabric or network types may be utilized herein. For example, a Peripheral Component Interconnect Express (PCIe) fabric may be utilized to couple to a CoPU, which may include various versions, such as 3.0, 4.0, or 5.0, among others. In lieu of a PCIe fabric, other point-to-point communication fabrics or buses with associated physical layers, electrical signaling, protocols, and layered communication stacks may be utilized. These may include, among others, Gen-Z, Ethernet, InfiniBand, NVMe, Internet Protocol (IP), Serial Attached SCSI (SAS), Fibre Channel, Thunderbolt, Serial Attached ATA Express (SATA Express), NVLink, Cache Coherent Interconnect for Accelerators (CCIX), Compute Express Link (CXL), Open Coherent Accelerator Processor Interface (OpenCAPI), Wi-Fi (802.11x), or cellular wireless technologies. The communications network coupled to the host system includes Ethernet or Wi-Fi (802.11x), which may refer to any of the various network communications protocol standards and bandwidths available, such as 10BASE-T, 100BASE-TX, 1000BASE-T, 10GBASE-T (10 GB Ethernet), 40GBASE-T (40 GB Ethernet), Gigabit (GbE), Terabit (TbE), 200GbE, 400GbE, 800GbE, or various other wired and wireless Ethernet formats and speeds.Cellular wireless technologies may include various wireless protocols and networks built around 3rd Generation Partnership Project (3GPP) standards, including, among others, 4G Long Term Evolution (LTE), 5G New Radio (NR), and related 5G standards.

[0090] Some of the aforementioned signaling or protocol types build on PCIe, thus adding additional functionality to the PCIe interface. Parallel, serial, or combined parallel / serial type interfaces may also be applied to the examples herein. Many of the examples herein utilize PCIe as an exemplary fabric type for coupling to peripheral devices, but it should be understood that others may be used instead. PCIe is a high-speed serial computer expansion bus standard, typically having multi-lane point-to-point connections between hosts and component devices or between peer devices. PCIe typically has multi-lane serial links connecting individual devices to a root complex. A PCIe communication fabric may be established using various switching circuits and control architectures described herein.

[0091] The various computer system components herein may be included in one or more physical enclosures, such as rack-mountable modules that may further be included in a shelf or rack unit. A number of components may be inserted or installed in a physical enclosure, such as a modular framework into which modules may be inserted and removed as required by a particular end user. An enclosed modular system may include physical support structures and enclosures that include circuits, printed circuit boards, semiconductor systems, and structural elements. Modules with components may be insertable into or removable from rack-mount style or rack unit (U) type enclosures. It should be understood that the components herein may be included in any physical mounting environment and need not include associated enclosures or rack-mount elements.

[0092] The functional block diagrams, operational scenarios and sequences, and flow diagrams provided in the figures represent exemplary systems, environments, and methodologies for implementing novel aspects of the present disclosure. For ease of explanation, it is understood and appreciated that the methods included herein may be in the form of functional diagrams, operational scenarios or sequences, or flow diagrams, and may be described as a series of operations, and that the methods are not limited by the order of operations, since some operations may occur in different orders and / or simultaneously with other operations as shown and described herein accordingly. For example, those skilled in the art will understand and appreciate that a method may alternatively be represented as a series of interrelated states or events, such as a state diagram. Moreover, not all operations illustrated in a methodology are required for a novel implementation.

[0093] The description and figures contained herein show specific implementations to teach those skilled in the art how to make and use the best options. Some conventional aspects have been simplified or omitted for the purpose of teaching the principles of the invention. Those skilled in the art will appreciate variations from these implementations that fall within the scope of the disclosure. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form multiple implementations. As a result, the present invention is not limited to the specific implementations described above, but only by the claims and their equivalents.

Claims

1. obtaining, via a network interface of a host device, an indication of peripheral devices available via the network interface for association with the host device; based on the instruction, initiating instantiation of the peripheral device onto a local peripheral interconnect interface of the host device by at least emulating a behavior of the peripheral device coupled through the network interface as a local peripheral device coupled to a peripheral interconnect interface of a host system; A method comprising:

2. 2. The method of claim 1, further comprising interfacing with a network stack of the host device to interact with traffic of the peripheral device with the local peripheral component interconnect interface.

3. The method of claim 1 , wherein the indication of the peripheral device includes a device identifier and a vendor identifier.

4. 2. The method of claim 1, wherein a device driver of the host device, when instantiated on a local peripheral interconnect interface of the host device, can interface with the peripheral device through the local peripheral interconnect interface.

5. 2. The method of claim 1, wherein emulating the behavior of the peripheral device comprises intercepting host-originated traffic for the peripheral device from the local Peripheral Component Interconnect interface; interpreting the host-originated traffic to convert it from a native peripheral format to a packet format suitable for transfer over the network interface; and routing the host-originated traffic in the packet format through the network interface for delivery to the peripheral device.

6. 2. The method of claim 1, wherein emulating the behavior of the peripheral device comprises receiving traffic from the peripheral device in a packet format from the network interface; and interpreting the traffic from the peripheral device in the packet format for conversion to a native peripheral format suitable for the local peripheral interconnect interface.

7. 2. The method of claim 1, wherein initiating instantiation of the peripheral device into the local peripheral interconnect interface comprises triggering a start point process within the local peripheral interconnect interface to emulate a hardware plug-in process for the peripheral device having the local peripheral interconnect interface.

8. 8. The method of claim 7, wherein emulating the hardware plug-in process includes providing an address space description to the local peripheral interconnect interface for accessing the peripheral device as the local peripheral device coupled to a host system.

9. 2. The method of claim 1, further comprising removing an instantiation of the peripheral device from the local peripheral interconnect interface by at least emulating a hardware removal process within the local peripheral interconnect interface.

10. 10. The method of claim 1, wherein the host device's local peripheral interconnect interface comprises a Peripheral Component Interconnect Express (PCIe) interface.

11. one or more computer readable storage media; Program instructions stored on the one or more computer readable storage media, the program instructions comprising: obtaining, via a network interface of a host device, an indication of peripheral devices available via the network interface for association with the host device; based on the instructions, initiating instantiation of the peripheral device onto a local peripheral interconnect interface of the host device by at least emulating a behavior of the peripheral device coupled through the network interface as a local peripheral device coupled to a host system; and program instructions executable by a processing system to direct the processing system to perform at least An apparatus comprising:

12. and program instructions stored on the one or more computer readable storage media, the program instructions comprising: interfacing with a network stack of the host device to interact with traffic of the peripheral device with the local peripheral interconnect interface; 12. The apparatus of claim 11, executable by the processing system to instruct the processing system to perform at least:

13. The apparatus of claim 11 , wherein the indication of the peripheral device includes a device identifier and a vendor identifier.

14. 12. The apparatus of claim 11, wherein when instantiated on a local peripheral interconnect interface of the host device, a device driver of the host device can interface with the peripheral device through the local peripheral interconnect interface.

15. and program instructions stored on the one or more computer readable storage media, the program instructions comprising: emulating a behavior of the peripheral device by at least intercepting host-originated traffic for the peripheral device from the local peripheral interconnect interface, interpreting the host-originated traffic to convert it from a native peripheral format to a packet format suitable for transfer over the network interface, and further routing the host-originated traffic in the packet format through the network interface for delivery to the peripheral device.

12. The apparatus of claim 11, executable by the processing system to instruct the processing system to perform at least:

16. and program instructions stored on the one or more computer readable storage media, the program instructions comprising: emulating a behavior of the peripheral device by at least receiving traffic from the peripheral device in a packet format from the network interface and interpreting the traffic from the peripheral device in the packet format for conversion to a native peripheral format suitable for the local peripheral interconnect interface; 12. The apparatus of claim 11, executable by the processing system to instruct the processing system to perform at least:

17. and program instructions stored on the one or more computer readable storage media, the program instructions comprising: initiating instantiation of the peripheral device into the local peripheral interconnect interface by at least triggering a start point process within the local peripheral interconnect interface to emulate a physical hardware plug-in process of the peripheral device into a host device; initiating de-instance of the peripheral device from the local peripheral interconnect interface by at least triggering an end-point process within the local peripheral interconnect interface to emulate a physical hardware unplug process of the peripheral device from the host device; 12. The apparatus of claim 11, executable by the processing system to instruct the processing system to perform at least:

18. 12. The apparatus of claim 11, wherein the host device's local peripheral interconnect interface comprises a Peripheral Component Interconnect Express (PCIe) interface.

19. a local interface configured to communicate with a local peripheral interconnect interface of a host device; a remote interface configured to communicate with a network stack of a network interface of the host device; An interaction unit, obtaining, via the remote interface, an indication of peripheral devices available via the network interface for association with the host device; based on the instructions, initiating instantiation of the peripheral device onto a local peripheral interconnect interface of the host device by at least emulating a behavior of the peripheral device coupled through the network interface as a local peripheral device coupled to a host system; an interaction unit configured to perform the A software element comprising:

20. configured to emulate a behavior of the peripheral device by at least intercepting host-originated traffic for the peripheral device from the local peripheral interconnect interface, interpreting the host-originated traffic to convert it from a native peripheral format to a packet format suitable for transfer over the network interface, and further routing the host-originated traffic in the packet format through the network interface for delivery to the peripheral device; 20. The software element of claim 19, further comprising the interworking unit configured to at least receive traffic originating from a peripheral device in a packet format from the network interface, and further configured to emulate a behavior of the peripheral device by interpreting traffic originating from the peripheral device in the packet format for conversion to the native peripheral format suitable for the local peripheral interconnect interface.

Images