Dynamic Server Rebalancing

Abstract

Improved apparatus, systems, and techniques are presented for coupling a remote peripheral device located on a host device as a local peripheral device of a client device. The method includes receiving an indication of a peripheral device located on a first computing device that is available for data processing and receiving a request from a second computing device to access the peripheral device. Based on the request, the method may include instructing the second computing device to emulate the peripheral device as a local device installed on the second computing device and routing data traffic from the second computing device to the first computing device for processing by the peripheral device.

Description

[Background technology]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of and priority to U.S. patent application Ser. No. 17 / 831,518, entitled "DYNAMIC SERVER REBALANCING," filed on June 3, 2022, and U.S. Provisional Patent Application Ser. No. 63 / 321,274, entitled "DYNAMIC SERVER REBALANCING," filed on March 18, 2022.

[0002] Clustered computer systems are becoming more prevalent due to the growing demand for data storage, data processing, and communication processing. Data centers typically include large rack-mounted and network-coupled data storage and data processing systems. These data centers may receive data for storage from external users over network links as well as data generated from applications running on processing elements within the data center. Often, data centers and associated computer equipment may be utilized to execute jobs for multiple simultaneous users or applications. Jobs include running jobs that may utilize the resources of the data center to process data using a central processing unit (CPU) or a graphics processing unit (GPU) as well as routing data associated with these resources between temporary and long-term storage or between various network locations. For example, GPU-based processing is becoming more popular for use in artificial intelligence (AI) and machine learning regimes. In these regimes, computer systems such as blade servers may include one or more GPUs along with associated CPUs for processing large data sets.

[0003] Specifically, a server typically includes a fixed arrangement between CPUs, GPUs, and storage elements housed in a common enclosure or chassis. When an incoming job is deployed in a data center, the granularity of computer resources is limited to individual servers. Thus, regardless of whether the entire resources of a server are actually required to execute the job, the deployed job typically occupies one or more servers along with all of the corresponding CPUs, GPUs, and storage elements of each server. To compensate, data center operators typically deploy a continuously increasing number of servers to accommodate the increasing traffic from the jobs. This strategy can face barriers due to large space and cost requirements in addition to the physical space required for rack-mounted servers. Summary of the Invention [Means for solving the problem]

[0004] Improved apparatus, systems, and techniques are presented for virtually coupling remotely located peripheral devices physically coupled to a host computer system to a client computer system. Although the peripheral devices are remotely located with respect to the host device and connected to the client device via a communication network link, the peripheral devices may be coupled to the client computer system as local devices. These improvements may provide peripheral devices such as GPUs, FPGAs, or ASICs (called co-processing units (CoPUs), or data processing accelerators) that are utilized on demand by the client computer system via the network link. These peripheral devices may be optionally associated and disassociated with various client devices (such as servers or other computer systems) as if they were local peripheral devices plugged into the client device. Thus, the client device may add a larger number of peripheral devices for use in processing workloads or user data than would normally be feasible compared to physically plugging the peripheral devices into the motherboard of the client device. The host device may share access to the physically connected peripheral devices with remote client devices when the peripheral devices are not being used by the host, thereby efficiently utilizing all the resources of the computer cluster without the costs associated with adding additional servers and space requirements.

[0005] In one example implementation, the method may include receiving an indication of a peripheral device available for data processing and located on a first computing device, receiving a request from a second computing device to access the peripheral device, and instructing the second computing device based on the request to emulate the peripheral device as a local device installed on the second computing device. The method may further include routing data traffic from the second computing device for processing by the peripheral device at the first computing device.

[0006] In another exemplary implementation, a system may include a first computing device including a network interface, the first computing device configured to obtain, via the network interface, an indication from a server rebalancing system of a peripheral device available via the network interface for processing, the peripheral device located on a second computing device. The first computing device may issue a request to access the peripheral device to the server rebalancing system via the network interface, and based on a response from the server rebalancing system, emulate a local installation of the peripheral device at the first computing system. The first computing device may issue data traffic to the server rebalancing system via the network interface for processing by the peripheral device.

[0007] In yet another example implementation, the method may include issuing an indication from the first computing device to a server rebalancing system via a network interface that a peripheral device located on the first computing device is available for processing, and receiving a second indication at the first computing device from the server rebalancing system via the network interface that the peripheral device is assigned to a second computing device. The method may further include receiving data traffic from the second computing device via the network interface from the server rebalancing system at the first computing device for processing by the peripheral device, and providing results of the processing of the data traffic by the peripheral device from the first computing device via the network interface.

[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the technical disclosure. It should be understood that this Summary is not intended to identify key features or essential features of the claimed subject matter, nor should it be used to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0009] Many aspects of the present disclosure can be best understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, like reference numbers indicate corresponding parts throughout the several views. Although several embodiments have been described in connection with these drawings, the present disclosure is not limited to the embodiments disclosed herein. On the contrary, the intention is to cover all alternatives, modifications, and equivalents.

[0010] [Figure 1] FIG. 1 illustrates a computer system in one implementation.

[0011] [Diagram 2] 1 is a flow diagram illustrating an example of the operation of a computer system in one implementation.

[0012] [Diagram 3] FIG. 2 illustrates a host device in one implementation.

[0013] [Figure 4] FIG. 1 illustrates a fabric control system in one implementation.

[0014] [Diagram 5] FIG. 1 illustrates a computer system in one implementation.

[0015] [Figure 6] FIG. 1 illustrates a control system in one implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In addition to being utilized to process running jobs and process data, data centers with associated computer equipment may transfer data associated with running jobs between temporary and long-term storage or between various network destinations. Data centers typically include a number of rack-mounted computer systems or servers that are packaged independently and house corresponding sets of processors, system memory, data storage devices, network interfaces, and other computer equipment coupled via internal data buses. Once installed, these servers are typically not modified or changed, except for minor upgrades or repairs of individual components to replace existing components. This relatively fixed arrangement of servers may be referred to as being in an integrated arrangement. Thus, each server represents a granular unit of computer equipment whose individual internal components are housed once by the server manufacturer and rarely changed once inserted into the rack by the system installer.

[0017] The limitations of separate server-based computer systems may be overcome using de-aggregated physical components and peripheral devices that may be dynamically attached to client computer systems while not being locally coupled to the local data bus of such client systems. Instead of having a fixed arrangement between computer devices and peripheral devices, with the entire computer system housed in a common enclosure or chassis, the examples herein may flexibly include any number of peripheral devices across any number of enclosures / chassis, dynamically formed into a logical arrangement over a communications fabric or network. Furthermore, in addition to de-aggregated components that do not have a traditional server motherboard relationship, the various exemplary integrated computer systems described herein may make unused locally attached data processing resources and peripheral devices available to other integrated computer devices. For example, peripheral devices of a host computer system may be emulated as locally mounted or locally installed on a client computer system for use, even when a client is accessing the devices remotely over a network. Thus, by not having idle or wasted portions of an integrated server that are not needed for a particular task, or for a particular portion of a task, and instead making those idle components available for use by other computer devices, the computer system may make better use of resources. Data center operators can achieve much higher utilization levels of their data centers than could be achieved using fixed-location servers and augment existing servers with additional capabilities over existing network connections. These actions and techniques may be referred to as dynamic server rebalancing.

[0018] The systems and operations described herein provide dynamic rebalancing and allocation of peripheral resources of individual computing devices, such as computer resources (CPUs), graphics processing resources (GPUs), network interface resources (NICs), communication fabric interface resources, data storage resources (SSDs), field programmable gate arrays (FPGAs), and system memory resources (RAMs), among multiple computing devices, even when the peripheral resources are not locally coupled to a client device utilizing the resources. The peripheral resources may also include co-processing units (CoPUs) or data processing accelerators, such as GPUs, tensor processing units (TPUs), FPGAs, or application specific integrated circuits (ASICs). Data processing of host device CPUs enhanced by CoPUs is gaining popularity for use in artificial intelligence (AI), machine learning systems, cryptocurrency mining and processing, advanced graphical visualization, biological systems modeling, autonomous vehicle systems, and a variety of other tasks.

[0019] In one example, peripheral resources may be de-aggregated and established as a pool of unused, unallocated, or free peripheral resources until allocated (configured) to a requesting client device using a communication fabric such as PCIe Express (Peripheral Component Interconnect Express) or Compute Express Link (CXL). A management processor, or dynamic server rebalancing system, may control the docking and dismantling between connected servers and computer systems and provide an interface to external users, job management software, or orchestration software. Peripheral resources and other elements (graphics processing, network, storage, FPGA, RAM, or other) may be made available by the host device and attached / detached on the fly to various client devices. In another example, peripheral resources may be located within the enclosures of individual servers. These peripheral resources may be established into a pool as described above for de-aggregated components, but instead are physically associated with the individual servers. Through the use of the improved techniques described herein, components located within a first server may be used for the activities of a second server as if those components were local devices to that second server. For example, graphics processing resources physically attached to a host device may be assigned to be virtually attached to a first client device via a dynamic server rebalancing system, and then removed from the first client device and assigned to a second client device. In another example, in the event of a resource failure, outage, or overload condition, additional peripheral resources from other host devices may be deployed to the client device to supplement the existing resources.

[0020] As a first exemplary system, FIG. 1 is presented. FIG. 1 is a system diagram illustrating a computer system 100 utilizing dynamic server rebalancing techniques that may include both disaggregated computer components and consolidated servers. The system 100 includes computer devices 110 and 140, a dynamic server rebalancing system 130, a network switch 131, and a disaggregated peripheral device 153. The computer devices 110 and 140 may communicate with the network switch 131 via network links 150-151. The peripheral device 153 communicates with the network switch 131 via a communication fabric link 152. Although only two computer devices are shown in FIG. 1, it should be understood that any number of computer devices may be included and coupled to the network switch 131 via associated network links.

[0021] The computing devices 110, 140 include network interfaces 111, 141, local peripheral device interconnect interfaces 112, 142, and peripheral over fabric (PoF) systems 115, 145. The network interfaces 111, 141 may be coupled to a network switch 131 via network links 150-151. The local interfaces 112, 142 may be coupled to local peripheral devices 113, 143 via local links. The computing device 110 and its associated components and connections are described below, but for ease of description, the same functionality may apply to the computing device 140 unless otherwise noted. The PoF system 115 may be coupled to both the network interface 111 and the local interface 112 via software and hardware connections, such as via software interfaces to associated protocol stacks or programming interfaces of the network interface 111 and the local interface 112.

[0022] In operation, computing device 110 may use an on-board central processing unit (CPU) along with peripheral devices including graphics processing devices (e.g., GPUs), data storage devices (e.g., SSDs), memory devices (e.g., DRAMs), network interfaces (e.g., NICs), and user interface devices to execute system software and user applications for various tasks. An operator of computing device 110, or an OS or other component of computing device 110, may desire to add additional peripheral devices for use by computing device 110. Conversely, computing device 110 may indicate that it has local resources or peripheral devices that are idle or otherwise available for use by other computing devices in system 100. Various improved techniques and systems are presented herein to facilitate making local peripheral devices available remotely to other computing devices and adding remote peripheral devices to a computing device without physically connecting such peripheral devices to the computing device. Peripheral devices, including data processing elements (such as CoPUs) or other peripheral devices (such as data storage or memory devices), may be configured to be associated with computing device 110 even if such devices or elements are not physically local to computing device 110. Computing device 110, as well as elements included in dynamic server rebalancing system 130, may enable remote sharing or addition of peripheral devices for use by computing device 110 as if the remote peripheral devices were local devices coupled via a local interface, such as a PCIe interface. Although PCI / PCIe connections are referenced herein as an example of a common peripheral device communication protocol, it should be understood that other peripheral device protocols, such as CXL, may be used without departing from the scope of this disclosure.By remotely sharing peripheral devices and other resources between computing devices, any associations between any of the peripheral devices 113, 143, and 140 and any of the host devices 110, 140 may be created and modified on the fly. These associations are made through a network interface of the computing device 110 and an optional communication interface coupled to the peripheral device 140, as described in more detail below.

[0023] Referring now to the description of the elements of FIG. 1, the host device 110 comprises a computer system having a processing element, a data storage and memory element, and a user interface element. Examples of the host device 110 include a server, a blade server, a desktop computer, a laptop computer, a tablet computer, a smart phone, a gaming system, an element of a distributed computer system, a customer equipment, an access terminal, an Internet appliance, a media player, or other computer system. Typically, the host device 110 will include a motherboard or other system circuit board having a central processing unit (CPU) coupled together with a memory device, such as random access memory (RAM) or dynamic RAM (DRAM). The CPU may be a component within a processing system formed from one or more microprocessor elements, including an Intel® microprocessor, an Apple® microprocessor, an AMD® microprocessor, an ARM® microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a tensor processor, an application specific processor, or other microprocessor or processing element. Various peripheral devices 113 may be physically coupled locally to the peripheral device interconnect interface 112 of the host device 110 through corresponding slots, connectors, or through cabling within the enclosure or chassis of the host device 110. These peripheral devices 113 may include graphics cards housing a graphics processing unit (GPU), data storage drives using various computer-readable media, network interface controllers (NICs) having physical layer elements for coupling to a network link (e.g., Ethernet), or other devices including user interface devices. For example, a PCIe device may be included in the host device 110, which is coupled to a PCIe slot on the motherboard of the host device 110. Such devices 113 are referred to as local peripheral devices.The host device 110 also includes various software that is executed by the processing system of the host device 110. This software typically includes an operating system, user applications, device drivers, user data, hypervisor software, telemetry software, and various other software elements.

[0024] The dynamic server rebalancing system 130, sometimes referred to as a control system, control module, management system, management processor, or fabric control system, comprises the control and interaction elements for the network switch 131. The dynamic server rebalancing system 130 may be included in a top-of-rack (ToR) switch chassis, or other network switching or routing elements. The dynamic server rebalancing system 130 comprises a management operating system (OS), an operator control interface, and various other elements as shown in Figures 4-6. The dynamic server rebalancing system 130 may comprise one or more microprocessors and other processing circuitry that retrieve and execute software from an associated storage system (not shown). The dynamic server rebalancing system 130 may be implemented within a single processing device, but may be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of the dynamic server rebalancing system 130 include general purpose central processing units, application specific processors, and logic devices, as well as any other types of processing devices, combinations thereof, or variations thereof. In some examples, the dynamic server rebalancing system 130 comprises an Intel® microprocessor, an Apple® microprocessor, an AMD® microprocessor, an ARM® microprocessor, an FPGA, an ASIC, an application specific processor, or other microprocessor or processing element. The dynamic server rebalancing system 130 includes at least one network interface, in some examples at least one fabric interface. The network interface comprises a network stack and associated physical layer interface used to communicate with the network switch 131 and to control the elements of the network switch 131 as well as to communicate with devices coupled to ports 133 of the network switch 131. The fabric interface may comprise a communication link subsystem used to communicate with the network switch 131 and to communicate between peripheral devices coupled to ports 134.The fabric interface may include one or more PCIe interfaces or other suitable fabric interfaces.

[0025] Network switch 131 includes network ports 133 that provide switched network connections to computing devices, as shown for network links 150-151. Network switch 131 includes various network switching circuitry for communicatively linking individual ports to other ports based on traffic patterns, addressing, or other traffic properties. In one example, network switch 131 comprises an Ethernet or Wi-Fi (802.11xx) switch that supports wired or wireless connections, which may refer to any of the various network communication protocol standards and bandwidths available, such as 10BASE-T, 100BASE-TX, 1000BASE-T, 10GBASE-T (10 GB Ethernet), 40GBASE-T (40 GB Ethernet), Gigabit (GbE), Terabit (TbE), 200GbE, 400GbE, 800GbE, or various other wired and wireless formats and speeds.

[0026] Network switch 131 optionally includes fabric ports 134, which may be part of separate fabric switch elements that communicate with dynamic server rebalancing system 130 via the fabric ports. Fabric ports 134 may couple to peripheral devices 153 via associated fabric links 152, which typically comprise point-to-point multi-lane serial links. Types of fabric ports and links include PCIe, Gen-Z, InfiniBand, NVMe, FibreChannel, NVLink, Cache Coherent Interconnect for Accelerators (CCIX), Compute Express Link (CXL), and Open Coherent Accelerator Processor Interface (OpenCAPI), among others. In FIG. 1, fabric ports 134 are shown coupled to a set or group of peripheral devices 153, which includes individual peripheral devices coupled to a backplane with a shared fabric connection to fabric ports 134 via links 152. Although a single group of uniform peripheral devices 153 is shown (e.g., just a box of GPUs, a JBOG), it should be understood that any number of groups in any arrangement of peripheral devices could be utilized instead, and individual peripheral devices could be coupled to fabric ports 134 without the use of a shared backplane.

[0027] The peripheral devices 113, 143, and 153 may include various co-processing units (CoPUs) or data processing accelerators, such as graphics processing units (GPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), or application specific integrated circuits (ASICs), as described above. The peripheral devices may also include data storage devices, such as solid-state storage devices (SSDs) including flash memory or other media types, hard drives (HDDs) including rotating magnetic media, magnetoresistive random access memory (MRAM), or other data storage devices with various media types and interface types. The peripheral devices may include fabric-bound memory devices, such as dynamic random access memory (DRAM), static random access memory (SRAM), 3D XPoint memory, solid-state memory devices, magnetic random access memory (MRAM) devices, or various other memory devices. A peripheral device may include a network interface controller (NIC) that includes various network interface elements, such as physical layer elements (PHYs), transport layer elements, TCP / IP traffic processing elements, routers, switches, or bridges, along with associated cable connectors.

[0028] Now, with reference to an exemplary set of operations of the elements of Figure 1, operations 200 are illustrated in Figure 2. Operations 200 may be subdivided into those performed by dynamic server rebalancing system 130 (labeled switch (SW) operations) and those performed by computing device 110 (labeled server operations), although various operations may be performed by different devices than those listed in Figure 2.

[0029] In operation 201, the dynamic server rebalancing system 130 discovers a network-coupled server device, such as the computing device 110. The dynamic server rebalancing system may receive initialization messaging provided by the computing device 110, such as a device identifier (ID), vendor ID, addressing, and other information. This information may be provided automatically by the computing device 110 upon power-up or reset (e.g., initiated as an operation of the server 110, not as an operation of the dynamic server rebalancing system 130), or may be queried by the dynamic server rebalancing system 130 using various protocol-specific command or control traffic. A communication protocol or handshake operation may be utilized between the server device 110 and the dynamic server rebalancing system 130, indicating the host IP address, port number, and login / password of the server device 110, so that the dynamic server rebalancing system 130 may issue an attach / detach operation to the server device 110.

[0030] Regardless of the technique used to discover individual host devices via link 150, dynamic server rebalancing system 130 may establish and maintain one or more data structures that list these properties and identifiers of the host devices along with the corresponding IP addresses, port numbers, and login / password parameters. One exemplary data structure is shown in FIG. 1 as “Server Devices 135.” Dynamic server rebalancing system 130 may update this data structure over time to add or remove server devices in response to operational status, changes in presence, or changes in availability of the dynamic server rebalancing system. A similar discovery process may be implemented to discover peripheral devices 153 via link 152.

[0031] In some examples, the dynamic server rebalancing system 130 may discover many network devices, but only discover catalog computing devices 110 that support the network-coupled peripheral device mode through the use of specialized software, firmware, or hardware deployed on such computing devices 110, such as, for example, the PoF module 115. This specialized module, such as a driver module or kernel level module, may report a status to the dynamic server rebalancing system 130 that includes a mode support indication to the dynamic server rebalancing system 130 along with a server-related network address, network port, or network socket.

[0032] In response to the discovery process between the dynamic server rebalancing system 130 and the server device 110, the server device 110 (or one or more modules therein) may generate and present to the dynamic server rebalancing system (202) a list of available peripheral components physically located at the server device 110. The list may include all peripheral components at the server device 110 or may be limited to peripheral components that are idle or available for use by other computer systems. In some examples, the server device 110 may report resources including peripheral devices as well as storage capacity or processing resources that do not involve making the entire component or peripheral device available. For example, the server device 110 may make available some of its CPU processing power while retaining some of its own processing power, such as by using virtualization. The list of peripheral devices may include device identifiers (IDs), vendor IDs, addressing, performance specifications or requirements, and other information by which peripheral devices may be identified for access. For example, the server device 110 may include multiple network ports and may dedicate different ports to one or more peripheral devices. In embodiments in which multiple peripheral devices may be accessed through a single port, the access request may use a device identifier or other information to inform server device 110 which peripheral device is being accessed.

[0033] At 203, based on receiving a list of available peripheral devices from the server device 110 and optionally from a discovery operation for additional peripheral devices 153, the dynamic server rebalancing system 130 may compile a data structure of available peripheral devices. An exemplary data structure is shown in FIG. 1 as “peripheral devices 136”. The peripheral device data structure 136 may identify the type of resource or peripheral device, its specifications, the associated server device 110, addressing information and identifiers, ports 133 or network connections 150, status (e.g., whether the resource is currently in use or assigned to the client server device 110 and any one), or other details. The peripheral device data structure 136 may be incorporated into the server device data structure 135 or maintained separately. The peripheral data structure may be updated by the dynamic server rebalancing system 130 as resources come online or become unavailable or as peripheral devices are attached / detached from the server system 110. The dynamic server rebalancing system may provide a list of available peripheral devices to the server device 110 connected to the network switch 131 via the network link 150. The list may be provided to the server device 110 automatically, for example, at regular intervals or whenever the list is updated. In some examples, the list may be provided to the server device 110 in response to a request to access a resource from the server device. A subset of the total number of peripheral devices in the "peripheral devices 136" may be provided based on the server device 110 requesting only certain types of devices. The PoF system 115 of the computing device 110 may then make a selection or request (204) of peripheral devices to be presented to the dynamic server rebalancing system 130 via the network interface 111. This selection may indicate an identification or identifier of the peripheral device that the server device 110 would like to be coupled to via the network link 150, such as a device ID or vendor ID used to identify the peripheral device on the local peripheral device interconnect interface 112.By requesting use of a peripheral device, server device 110 may be considered a "client" server or computing device of that peripheral device, while the computing device where the requested peripheral device is physically located may be referred to as a "host" server or computing device. In some examples, a single server device 110 may simultaneously be a "host" of several peripheral devices and a "client" of other peripheral devices. For purposes of the exemplary illustration of FIG. 2, computing device 110 may be a "client" server, while computing device 140 may be a "host" server.

[0034] At 205, in response to selection of the peripheral device by the client server 110, the dynamic server rebalancing system 130 may provide one or more commands instructing the PoF system 115 to remove the peripheral device from the available device pool (e.g., of "peripheral devices 136") and attach the peripheral device to the client system 110. The dynamic server rebalancing system 130 may also send an indication to the host system 140 indicating to the host system 140 that the peripheral device is occupied or unavailable while the peripheral device is attached to the client system 110.

[0035] Considering peripheral device 147A of host server 140 as an exemplary selected peripheral device, in response to the command issued by dynamic server rebalancing system 130, PoF system 115 performs a hot plug or attach process to instantiate a virtual or emulated version of peripheral device 147A, shown as 147B in FIG. 1, to local peripheral device interconnect interface 112 of client device 110 via virtual connection 116 (206). This hot plug or attach process may include emulating to local peripheral device interconnect interface 112 that a new hardware device is physically present in a slot or connector of client device 110 and triggering a start point process when the hardware device is not actually physically present in the slot or connector of client device 110. PoF system 115 may invoke the start point process in local peripheral device interconnect interface 112 without a corresponding hardware interrupt or hardware indicator that would normally result from a physical plug-in or power-on of a peripheral device to client device 110. This may also include modifying the local peripheral device interconnect interface 112 to accept an interrupt or attach command from the PoF system 115, rather than only from hardware modifications. The PoF system 115 also emulates a hardware plugging process by providing an address space description to the local peripheral device interconnect interface 112 to access the peripheral device 147B as a local peripheral device coupled to the client system 110. In an example where the local peripheral device interconnect interface 112 comprises a PCIe subsystem of the client device 110, an "attach" command may be issued by the PoF system 115 indicating a vendor ID and a device ID to the PCIe subsystem. The PCIe subsystem may also request or provide a memory addressing location for memory-mapped access to the peripheral device 147B.The PoF system 115 may emulate these addressing locations and the structural properties of such locations (eg, buffer sizing and formatting) to further emulate the behavior of the physical peripheral device 147A as a virtual peripheral device 147B.

[0036] When instantiated on the local peripheral device interconnect interface 112 of the client device 110, the PoF system 115 may emulate the peripheral device 147B at the client server 110 (207). The device driver of the client device 110 may interface with the peripheral device 147B through the local peripheral device interconnect interface 112 to provide access to operating system processes, user applications, kernel resources, and various other interfaces. Any tools and libraries associated with the peripheral device driver function similarly for a local peripheral device physically coupled to the client device 110 or for a peripheral device mapped over a network link through the PoF system 115. Thus, the device driver for the peripheral device 147B is typically unaware that an actual peripheral device 147A is not locally connected to the client device 110. User applications, operating systems, kernel resources, hypervisors, etc. may all interface with the peripheral device 147B as they would normally do when locally connected, emulating the behavior of the local hardware interface of the peripheral device 147B to the local peripheral device interconnect interface 112 through the PoF system 115. This emulation may include the above behaviors for instantiation and subsequent handling of communications between the local peripheral device interconnect interface 112 and the network interface 111. These communications may include configuration traffic, command and control handshakes, input / output traffic, and read / write traffic or data transfers between the client device 110 and the peripheral device 147A located at the host device 140. Thus, the PoF system 115 acts as an interworking unit for traffic of the peripheral devices 147A / B between the local peripheral device interconnect interface 112 and the network interface 111.The PoF system 115 interfaces with the network stack of the network interface 111 to send and receive this traffic to and from the actual peripheral device 147A over the network link 150. This may include intercepting the client-originated traffic for the peripheral device 147B from the local peripheral device interconnect interface 112 and interpreting the client-originated traffic for conversion or re-bundling from a native format (e.g., PCIe frames or the like of the local peripheral device interconnect interface 112) to an IP packet format or an Ethernet frame format suitable for transport over the network interface 111. The PoF system 115 then routes the host-originated traffic in packet format over the network interface 111 for delivery over the link 150 to the host server 140 and the physical peripheral device 147A.

[0037] The dynamic server rebalancing system 130 may receive traffic originating from a client device via link 150. Because network link 150 couples to client device 110 and network link 151 couples to host device 140, an interworking operation between the two port connections 133 is established. If the peripheral device being accessed is between peripheral device 153 and not host 140, the dynamic server rebalancing system 130 provides interworking (208) at least between network link 150 and network link 151, or in some examples, between network link 150 and fabric link 152. The connection between network link 150 and fabric link 152 may comprise protocol translation, which strips network frames or IP packets from various network protocol specific headers and then repackages or encapsulates the payload data into frames or packets (or other fabric native data crumbs) suitable for transfer over fabric link 152. Interworking traffic between network links of the same type, such as links 150-151, may not require format translation. Various read, write, input / output, control, command, or other traffic may be processed in this manner to forward transactions originating at the client device 110 to the peripheral device 147A at the host device 140. Similarly, the reverse operation may occur for transactions and responses originating from the host device 140 and the peripheral device 147A for forwarding to the client device 110.

[0038] Commands and other traffic sent from the client 110 to the host 140 may be received and processed at the network interface 141. The PoF system 145 communicates with the network stack of the network interface 141 to interpret these received network frames or packets for conversion to a native format suitable for the local peripheral device interconnect interface 142. The local peripheral device interconnect interface 142 provides the native format communication to the peripheral device 147A for execution or processing. The processing results may be intercepted or passed from the peripheral device interconnect interface 142 to the PoF system 145 and then passed to the network interface 141 for transmission to the client device 110 via the network link 151 and the dynamic server rebalancing system 130, similar to the transmission processing described for the host device 110 above.

[0039] Upon receipt by the client system 110 via the link 150 and the network interface 111, these network frames or packets are processed by the network stack of the network interface 111. The PoF system 115 communicates with the network stack of the network interface 111 and interprets these received network frames or packets to convert them into a native format suitable for the local peripheral device interconnect interface 112 as if they were sent from the emulated peripheral device 147B. The local peripheral device interconnect interface 112 provides communications in native format to various software elements of the client device 110, such as device drivers that interface with user-level applications. Thus, the client device 110 may use the peripheral device 147A via the emulated peripheral device 148B provided by the PoF 115 as if it were locally coupled to a local bus or connector of the client device 110.

[0040] At 209, when peripheral device 147A is no longer needed by client device 110, PoF system 115 may operate to remove peripheral device 147B from client device 110 and send a disconnection indication to dynamic server rebalancing system 130 via link 150. In some examples, host device 140 may send an indication to dynamic server rebalancing system 130 indicating that peripheral device 147A is no longer available, for example, based on host device 140 needing the peripheral device or its unavailability due to a shutdown, forced stop, or other operation. In such a case, dynamic server rebalancing system 130 may send an indication to client device 110 to disconnect peripheral device 147B. PoF system 115 may effect removal of instantiation of peripheral device 147B from local peripheral device interconnect interface 112 by at least emulating a hardware device removal process or a “hot unplug” operation to local peripheral device interconnect interface 112. This hot unplug or removal process involves emulating to the local peripheral device interconnect interface 112 that the hardware device is no longer physically present in the slot or connector of the client device 110 and triggering the end point process. The PoF system 115 invokes the end point process in the local peripheral device interconnect interface 112 without the corresponding hardware interrupt or hardware indicator that would normally result from a physical unplug or power off of the peripheral device to the client device 110. This may also include modifying the local peripheral device interconnect interface 112 to accept interrupts or detach commands from the PoF system 115 rather than only from hardware changes. The established address space description for the peripheral device 147B in the host device 110 may be torn down or removed in the local peripheral device interconnect interface 112.In another example, a “detach” command may be issued by the PoF system 115, which indicates the PCIe vendor ID and PCIe device ID of the peripheral device 147B to the local peripheral device interconnect interface 112.

[0041] At 210, the dynamic server rebalancing system 130 may update the data structure 136 to indicate that peripheral device 147A is no longer assigned or attached to the client system 110 and may return peripheral device 147A to the available peripheral device pool (optionally moving it to an unassigned, unavailable pool, or reassigning it to the host system 140). Peripheral device 147A may be returned to the available pool based on an indication from the client system 110 that peripheral device 147B has been disconnected, or the dynamic server rebalancing system 130 may detach peripheral device 147A from the client 110 without requiring confirmation from the client 110.

[0042] As discussed above, when peripheral device 147A is removed, it may be returned to the pool of peripheral devices that remain in an inactive, removed state until needed by a host device or other device. The attachment process may then proceed as described above. Further description of the pool of components or peripheral devices is provided in FIG. 5. Additionally, instead of a GPU or device similar to peripheral device 147A, other fabric-coupled devices may be attached / detached as described in the operation of FIG. 2, such as FPGAs, SSDs, NICs, memory devices, user interface devices, or other peripheral devices.

[0043] As noted above, with respect to peripheral device 147A, computing device 140 may be the host and computing device 110 may be the client, but for other peripheral devices, the relationships may be reversed. For example, while computing device 110 is remotely accessing peripheral device 147A, computing device 140 may simultaneously be remotely accessing peripheral device 148A from computing device 110 via emulated peripheral device 148B of FIG.

[0044] In some exemplary embodiments, a computing device 110 may access its own peripheral devices 113 locally via the peripheral device interconnect interface 112 and access only remote peripheral devices 143, 153 via the link 150 and the dynamic server rebalancing system 130. In these cases, only available or idle peripheral devices or resources may be reported to the dynamic server rebalancing system 130, while peripheral devices 113 being used by the computing system itself may not be reported as available. However, in some examples, a server 110 capable of dynamic server rebalancing (e.g., via the PoF system 115) may report all local peripheral devices 113 to the dynamic server rebalancing system 130. In such embodiments, the computing system may not directly access its local peripheral devices, but instead may request the use of a peripheral device from an available pool of peripheral devices via the dynamic server rebalancing system 130 for all peripheral device needs. In this manner, the computing system 110 may function as both a host and a client to itself, accessing its own local peripheral devices "remotely." Such an implementation may simplify resource rebalancing between computing devices 110 without contention for local resources between the host device and remote computing systems. For example, if a local peripheral device 148A of the host 110 is idle and attached to a client system 140, and the host 110 develops a need for the peripheral device 148A, it may cause a contention or interruption of operation at the client system 140. Such contention methods may be avoided by all systems utilizing the same shared pool of resources.

[0045] FIG. 3 is presented to illustrate the detailed structure and operation of the computing device 110. The elements of FIG. 3 may be applied to the computing device 110 of FIG. 1, but it should be understood that the computing device 110 may utilize other structures and operations. FIG. 3 includes a computing device 300 as an example of a computing device, server, computing system, blade server, etc. The computing device 300 may include one or more network links, each having an associated network address. For example, as described herein, a first network link 380 may couple the computing device 300 to a dynamic server rebalancing system or a network switch for coupling remote peripheral devices. Another network link 381 may communicate with other computing devices, further networks, or the Internet, among various other networks and endpoints, for transmissions unrelated to remote peripheral data traffic. The example described with respect to FIG. 3 also utilizes TCP / IP style networking for communication with the computing device 300 and a PCIe fabric for communication with local and emulated or networked peripheral devices.

[0046] The computing device 300 includes a user space 302 and a kernel space 301. The kernel space 301 may be a software system that includes core operating system (OS) elements such as an OS kernel, device drivers, hardware interface subsystems, network stacks, memory management subsystems, machine clock / time modules, and other low-level elements utilized to manage the computing device 300's resources between user-level and kernel-level software in addition to acting as an interface between hardware components and user applications. The user space 302 may include user applications, tools, games, graphical or command-line user interface elements, and other similar elements. Typically, the user space elements interface with the device driver elements of the kernel space 301 through application programming interfaces (APIs) or other software-defined interfaces to share access to low-level hardware elements among all user software, such as network controllers, graphics cards, audio devices, video devices, user interface hardware, and various communication interfaces. These device driver elements receive user-level traffic and interact with the hardware elements that ultimately drive link layer communication, data transfer, data processing, logic, or other low-level functions.

[0047] Within kernel space 301, computing device 300 may include network stack 330, peripheral over fabric (PoF) unit 320, PCI / PCIe module 340, and device drivers 350, such as for connected peripheral devices. Other kernel space elements are omitted for clarity and to focus on kernel level elements relevant to the operation of this specification. User space 302 includes user commands 360 and user applications 361. Network stack 330 comprises a TCP / IP stack and includes various layers or modules typical of a network stack, although some elements are omitted for clarity. Ethernet driver 334 includes functions of link layer, media access controller (MAC) addressing, Ethernet frame processing, and interfacing with a network interface controller (not shown) that handles physical layer operations and structures. IP module 333 performs packet processing, IP addressing, and internetwork operations. The TCP / UDP module 332 packetizes user data and handles error correction and retransmission, acknowledgment of transfers, etc., in addition to interfacing between user application data structures and the IP module 333. The socket layer 331 interfaces with user applications and other components of the computing device 300 and acts as an endpoint for packetized communication. Individual sockets can be established, each of which handles a specific communication purpose, type, protocol, or other communication separation. Several sockets can be established by the network stack 330, each of which can act as an endpoint for a different communication type. For TCP / UDP, a socket is typically identified by an IP address and a port number, and a host device may have many such port numbers for multiple IP addresses, each with its own set of port numbers, in addition to a single IP address. Thus, many sockets may be established, each with a specific purpose.User-level applications, user processes, or even kernel-level processes, modules, and elements may interface with the network stack 330 through specific sockets.

[0048] In operation, in response to attach / detach commands forwarded by the server rebalancing / control entity (and directed to the sockets), the PoF unit 320 may invoke the hot plug / unplug functions of the PCI / PCIe module 340 and emulate the hardware behavior for these functions to establish the functionality of the remote peripheral device as if it were a local peripheral device of the computing device 300. The PoF unit 320 interfaces with the socket layer 331 to forward and receive packets carrying traffic associated with peripheral devices that may be remotely located from the computing device 300. Instead of directly interfacing with the socket layer 331, the PoF unit 320 may instead use a TCP Offload Engine (TOE) stack and Remote Direct Memory Access (RDMA) for a particular network interface controller vendor type. The socket layer 331, or equivalent as described above, is identified by an IP address and port number and is typically dedicated to traffic associated with a particular peripheral device or all remote peripheral devices for the computing device 300. A username / password or other security credentials may be passed along with packets received by the socket layer 331. The PoF unit 320 has "hooks" or software interface functions to communicate with the socket layer 331. Packets arrive from peripheral devices through the network stack 330 and are interpreted by the PoF unit 320, which then converts the communications into a format suitable for the PCI / PCIe module 340. The packets received by the interworking unit 320 may include PCIe device status information of the peripheral devices. The PCI / PCIe module 340 receives these communications from the PoF unit 320 as if they originated from local peripheral devices of the computing device 300. Thus, the PoF unit 320 emulates the behavior of local peripheral devices to the PCI / PCIe module 340, and such peripheral devices appear local to the PCI / PCIe module 340.The device drivers, device tools or toolsets, and device-centric libraries function similarly for locally connected or remote PCIe devices mapped through the PoF unit 320. To achieve this emulation of local devices, the PoF unit 320 may establish several functions or libraries that present targets for the PCI / PCIe module 340 to communicate with for I / O transactions, configuration transactions, reads / writes, and various other communications.

[0049] Advantageously, user applications may interact with peripheral devices located remotely from computing device 300 using standard device drivers 350 that interface with PCI / PCIe module 340. Communications issued by PCI / PCIe module 340 that are normally intended for local hardware devices are intercepted by PoF unit 320 and interpreted for forwarding via network stack 330 and network link 380. When GPU peripheral devices are utilized, in one example, a graphics driver may be utilized without modification by a user application, such as a machine learning, deep learning, artificial intelligence, or gaming application, even though the GPU is located remotely with respect to computing device 300.

[0050] FIG. 4 is presented to discuss the detailed structure and operation of the dynamic server rebalancing system. The elements of FIG. 4 may be applied to the dynamic server rebalancing system 130 of FIG. 1, but it should be understood that the dynamic server rebalancing system 130 may utilize other structures and operations. FIG. 4 includes a dynamic server rebalancing system 400 as an example of a management node, a computing device, a server, a computing system, a blade server, and the like. The dynamic server rebalancing system 400 may establish and manage a communication fabric for sharing resources, such as peripheral devices, between connected computing nodes, such as the computing device 110 and the peripheral device pool 153 of FIG. 1. The dynamic server rebalancing system 400 may include one or more network links, each having an associated network address. For example, as described herein, a first network link 480 may couple to a network switch, which in turn couples to a computing device or a server device for coupling peripheral devices. The example described with respect to FIG. 4 also utilizes TCP / IP style networking for communicating with the dynamic server rebalancing system 400, and in some examples, a PCIe interface for communicating with the pool of peripheral devices.

[0051] The dynamic server rebalancing system 400 may include a user space 402 and a kernel space 401. The kernel space 401 may be a software system that includes core operating system (OS) elements such as an OS kernel, device drivers, hardware interface subsystems, network stacks, memory management subsystems, machine clock / time modules, and other low-level elements utilized to manage the resources of the dynamic server rebalancing system 400 between user-level and kernel-level software in addition to acting as an interface between the hardware components and user applications. The user space 402 may include user applications, tools, telemetry, event handlers, user interface elements, and other similar elements. Typically, the elements of the user space 402 interface with the device driver elements of the kernel space 401 through application programming interfaces (APIs) or other software-defined interfaces to share access to the low-level hardware elements among all user software, such as network controllers, fabric interfaces, sideband communication / control interfaces, maintenance interfaces, user interface hardware, and various communication interfaces. These device driver elements receive user level traffic and interact with hardware elements that ultimately drive link layer communication, data transfer, data processing, logic, or other low level functions.

[0052] Within kernel space 401, dynamic server rebalancing system 400 may include network stack 430, fabric module 440, and PCI / PCIe interface 460. Other kernel space elements are omitted for clarity and to focus on kernel level elements relevant to the operation of this specification. Network stack 430 comprises a TCP / IP stack and includes various layers or modules typical of a network stack, although some elements are omitted for clarity. Ethernet driver 434 includes the functions of link layer, media access controller (MAC) addressing, Ethernet frame processing, and interfacing with a network interface controller (not shown) that handles physical layer operations and structures. IP module 433 performs packet processing, IP addressing, and internetwork operations. TCP / UDP module 432 packetizes user data and handles error correction and retransmission, acknowledgement of transfers, etc., in addition to interfacing between user application data structures and IP module 433. The socket layer 431 interfaces with user applications and other components of the dynamic server rebalancing system 400 and acts as an endpoint for packetized communications. Individual sockets can be established, each of which handles a specific communication purpose, type, protocol, connected device, or other communication separation. Several sockets can be established by the network stack, each of which can act as an endpoint for a different communication type or communication link. In the case of TCP / UDP, a socket is typically identified by an IP address and a port number, and a device may have many such port numbers for a single IP address, as well as multiple IP addresses, each with its own set of port numbers. Thus, many sockets may be established, each with a specific purpose. User-level applications, user processes, or even kernel-level processes, modules, and elements may interface with the network stack through specific sockets.

[0053] The fabric module 440 may include drivers and other elements for managing the sharing or rebalancing of resources between computing devices connected to the dynamic server rebalancing system 400. The fabric module 440 provides a path for command and control of the fabric itself, such as for logical partitioning / isolation or attachment / removal of peripheral devices. Traffic related to reads, writes, configuration, and I / O of peripheral devices may also be handled by the fabric module 440. In some examples, the fabric module 440 may include a PCI / PCIe subsystem that includes the equivalent of a protocol stack for a PCI / PCIe link. The fabric module 440 may interface with physical layer elements, such as a PCI / PCIe interface 460, and also presents a software / programming interface for the configuration handler 412. Additionally, the fabric module 440 may interface user space elements (e.g., command processor 414) with the PCI / PCIe interface 460. PCI / PCIe interface 460 may comprise fabric chips or fabric switch circuits that may provide one or more physical fabric links that couple to fabric module 440, to local devices of dynamic server rebalancing system 400, and to a pool of peripheral devices coupled via associated PCIe links (e.g., peripheral device 153 connected via link 152 in FIG. 1), or to additional fabric chips or fabric switch circuits that provide portions of the fabric and additional fabric links.

[0054] The user space 402 includes a server rebalance control element 410, which may further comprise a monitor 411, a configuration handler 412, an event handler 413, a command processor 414, and a user interface 415. The command processor 414 may communicate with the fabric module 440 to control the communication fabric utilized to establish logical partitioning or allocation among peripheral devices coupled to the fabric and provide routing of communication / traffic to and from selected peripheral devices. When a selected peripheral device is removed, the peripheral device may be placed in a pool of unused devices. The user interface 415 may receive operator commands to manage the fabric or to control the addition or removal of peripheral devices to and from the computing device. The user interface 415 may display or indicate a list of computing devices and peripheral devices along with their associated status or telemetry. The user interface 415 may display or indicate which peripheral devices are associated with which computing devices. The user interface 415 may display or indicate traffic histograms, logs, faults, alerts, and various other telemetry and status. The user interface 415 may comprise a terminal interface, an application programming interface (API), a representational state transfer (REST) ​​interface or RestAPI, a web interface, a WebSocket interface, among other types of user interfaces, including software, hardware, virtualization, or those transmitted over various intermediate links. An event handler 413 may initiate attach / detach and device discovery operations for computing devices. A configuration handler 412 interacts traffic between network-coupled computing devices and, optionally, a collection of PCI / PCIe connected peripheral devices from a peripheral device pool. The configuration handler 412 interfaces with a fabric module 440 for fabric communication and a network stack 430 for network communication.The configuration handler 412 may interact with the frame, or packet, format, size, and type for transmitting communications over the network link and the PCI / PCIe link. The configuration handler 412 interfaces with the network stack 430 through a socket layer 431 through a specific socket indicated by at least an IP address and a port number. The monitor 411 may monitor various telemetry, operations, logs, and status for the dynamic server rebalancing system 400. The monitor 411 may maintain data structures indicating indicators, addresses, or identities of peripheral devices in addition to indicating computer devices and associated sockets (IP addresses and ports). The monitor 414 may maintain logs and data structures in computer readable media, such as data structures 435 and 436 in a memory device 465 locally connected to the PCI / PCIe interface 460 of the dynamic server rebalancing system 400.

[0055] During operation, the event handler 413 may initiate operations to discover compatible computing devices coupled through the network interface and to discover peripheral devices coupled to the communications fabric. The event handler 413 may instruct the configuration handler 412 to discover computing devices through the network stack 430. Socket information of the compatible computing devices may be determined and stored, for example, in data structure 435, for later use. In some examples, the event handler 413 may instruct the command processor 414 to discover peripheral devices through the fabric module 440 and the PCI / PCIe interface 460. The PCI / PCIe interface 460 scans the communications fabric to determine which peripheral devices and resources are available. The command processor 414 may form a pool of free peripheral devices and an indication of the assigned peripheral devices, store the device / vendor IDs of the peripheral devices, and store an indication of the PCIe addressing and buffer structure or characteristics of each peripheral device in the data structure 436, for example.

[0056] A request to add a peripheral device to a computing device is received, such as via a user interface 415 or via a network stack 430, and the command processor 414 may detach or attach the selected peripheral device to a logical partition of the communications fabric via the fabric module 440. This may trigger the event handler 413 to initiate a notification to the host device where the selected peripheral device is physically located that the selected peripheral device has been assigned to the remote client computing device. The event handler 413 may also issue an attach command with the peripheral device information (e.g., vendor ID and device ID) to the client device, which will then attach the peripheral device as described herein. From here, communications between the client device and the peripheral device at the host device are mediated using the configuration handler 412, which interprets and exchanges traffic between the socket layer 431 via the fabric module 440. In some examples, the event handler 413 may connect the requesting client device and the peripheral device not associated with the host computing device via the fabric module 440 and the PCI / PCIe interface 460. At some point, it may be desirable for an attached peripheral device to be removed or detached from a particular client device. Event handler 413 may detect these detach events as received by user interface 415 or from the computing device via network stack 430. Event handler 413 then issues a detach command through configuration handler 412 to detach the affected peripheral device, as described herein. Command processor 414 may remove the logical partitioning or allocation of the detached peripheral device and return the detached peripheral device to an inactive state or to a free pool of peripheral devices for later use.

[0057] FIG. 5 is a system diagram illustrating a computer system 500. The computer system 500 may include elements of the computer system 100 of FIG. 1, the computer device 300 of FIG. 3, or the dynamic server rebalancing system 400 of FIG. 4, although variations are possible. The computer system 500 includes a rack-mounted arrangement of multiple modular chassis. One or more physical enclosures, such as modular chassis, may further be included in a shelf or rack unit. The chassis 510, 520, 530, 540, and 550 are included in the computer system 500 and may be mounted in a common rack-mounted arrangement in one or more data centers or across multiple rack-mounted arrangements. Within each chassis, modules or peripheral devices may be mounted to shared circuit boards and switch elements along with various power systems, structural support, and connector elements. The enclosed modular system may include physical support structures and enclosures that include circuits, printed circuit boards, semiconductor systems, and structural elements. Modules comprising components of computer system 500 are insertable into and removable from rack-mount style enclosures. In some examples, the components of Figure 5 are included in a "U" style chassis for mounting within a larger rack-mount environment. It should be understood that the components of Figure 5 can be included in any physical mounting environment and need not include associated enclosures or rack-mount elements.

[0058] The chassis 510 may comprise a management module or a top-of-rack (ToR) switch chassis, such as the dynamic server rebalancing systems of 130, 300 in FIG. 1 and 400 in FIG. 4, and may comprise a management processor 511, an Ethernet switch 516, and a PCIe switch 560. The management processor 511 may comprise a management operating system (OS) 512, a user interface 513, and an interaction unit 514. The management processor 511 may be coupled to the Ethernet switch 516 via one or more network links via a network interface controller. The management processor 511 may be coupled to the PCIe switch 560 via one or more PCIe links comprising one or more PCIe lanes.

[0059] Ethernet switch 516 may include network ports that provide switched network connections to attached devices, as shown for network link 566. In an exemplary embodiment, network link 566 may connect dynamic server rebalancing system chassis 510 to blade server motherboards 561-563 of chassis 520, 530, and 540. Ethernet switch 516 includes various network switching circuits for communicatively linking individual ports to other ports based on traffic patterns, addressing, or other traffic properties. In one example, the Ethernet switch 516 comprises an Ethernet or Wi-Fi (802.11xx) switch hosting wired or wireless connections, which may refer to any of the various network communication protocol standards and bandwidths available, such as 10BASE-T, 100BASE-TX, 1000BASE-T, 10GBASE-T (10 GB Ethernet), 40GBASE-T (40 GB Ethernet), Gigabit (GbE), Terabit (TbE), 200GbE, 400GbE, 800GbE, or various other wired and wireless formats and speeds. The PCIe switch 560 may be coupled to a PCIe switch 564 in the chassis 550 via one or more PCIe links. These one or more PCIe links are represented by PCIe inter-module connections 565.

[0060] The network links 566, PCIe links 565, and components of the dynamic server rebalancing system chassis 510 form a fabric that communicatively couples all of the various physical computing elements of FIG. 5. In some examples, a management processor 511 may communicate with elements of the fabric via a special management PCIe link or sideband signaling (not shown), such as an inter-integrated circuit (I2C) interface, to control the operation, partitioning, and attachment / detachment of components of the fabric. These control operations may include configuring and tearing down peripheral devices, changing logical partitioning within the fabric, monitoring telemetry of the fabric, controlling the power up / down operation of modules on the fabric, updating firmware of the various circuits that comprise the fabric, and other operations.

[0061] Chassis 520, 530, and 540 may comprise blade server computer systems (such as computer system 110 of FIG. 1). Chassis 520 may comprise a blade server motherboard and Ethernet switch 561 having various components and resources including CPU 521, GPU 522, TPU 523, NIC 524, and SSD 525, although different numbers and arrangements of system components may be included. Similarly, chassis 530 may include a blade server motherboard 562 and locally connected components 531-535, and chassis 540 may include a blade server motherboard 563 and locally connected components 541-545. Power systems, monitoring elements, internal / external ports, mounting / removal hardware, and other associated functionality may be included in each chassis. A component or module of example blade server 520 (e.g., PoF unit 115 of FIG. 1 or PoF unit 320 of FIG. 3) may generate a list of components 521-525 in chassis 520 that are available for remote sharing with other computing devices in system 500 and may provide a list of some or all of those components to a management device in chassis 510 via link 566. For example, the list may include all components 521-525 along with an indication of which are available for sharing and which are unavailable (e.g., because they are in use by blade server 520), or the list may include only those devices available for sharing.

[0062] Chassis 550 may comprise a disaggregated collection of peripheral devices (such as peripheral device 153 of FIG. 1). Chassis 550 may comprise multiple GPUs 551-555, each coupled to a PCIe fabric via a backplane and PCIe switch 564 and associated PCIe links (not shown). In an exemplary embodiment, chassis 550 may comprise a JBOG (just a box of GPUs) system, although other arrangements of disaggregated peripheral devices may also be provided. For example, chassis for CPUs, NICs, or SSDs may also be presented, or combinations of various types of disaggregated peripheral devices may be provided within a single chassis. For example, chassis 550 may include modular bays for mounting modules with corresponding elements for each CPU, GPU, SSD, NIC, or other module type. In addition to the component types described above, other component types may be included as well, such as FPGAs, CoPUs, RAM, memory devices, or other components. Power systems, monitoring elements, internal / external ports, mounting / removal hardware, and other associated functionality may be included in each chassis 550. Further description of the individual elements of chassis 520, 530, 540, and 550 is included below.

[0063] Once the various CPU, GPU, TPU, SSD, or NIC components of the computer system 500 are installed in the associated chassis or enclosures and reported or discovered by the management device 510, the components may be logically assigned or organized into any number of separate, arbitrarily defined arrangements and attached to the computer device. These arrangements may consist of a selected number of CPUs, GPUs, SSDs, and NICs, including zero of any type of module. For the exemplary computer device 540 shown in FIG. 5, a remote peripheral GPU 532 (which may be physically located on the host computer device 530) is attached. The GPU 532 may be attached to the client computer device 540 via network link 566 using a logical division or allocation within the network fabric depicted by logical domain 570. A logical arrangement of computer components, including locally connected physical components and remote or logically connected peripheral components, may be referred to as a "computing unit." The network fabric may be configured by management processor 511 to selectively route traffic between selected client devices and host devices and peripheral devices locally attached to the host devices while maintaining logical separation between components not physically or virtually contained on the selected client devices. In this manner, a de-aggregated and flexible "bare metal" configuration may be established between components of computer system 500. Individual peripheral devices may be deployed according to specific user identities, computing device identities, running jobs, or usage policies.

[0064] In some examples, the management processor 511 may provide for the installation or removal of peripheral devices and host devices through one or more user interfaces or job interfaces. For example, the management processor 511 may provide a user interface 513 that may present an indication of available peripheral components to be installed, an indication of available computer devices, and software and configuration information. In some examples, the creation user interface 513 may provide a template for installing a predefined arrangement of peripheral devices to a computer device based on a use case or usage category. For example, the user interface 513 may provide suggested templates or configurations for a game server unit, an artificial intelligence learning computation unit, a data analysis unit, and a storage server unit. For example, a game server unit or an artificial intelligence processing template may specify additional graphics processing resources when compared to a storage server unit template. Furthermore, the user interface 513 may provide customization of the template or arrangement configuration and options for a user to create an arrangement template from a component type arbitrarily selected from a list or category of components.

[0065] In additional examples, the management processor 511 may provide policy-based dynamic adjustments to the placement during operation. In some examples, the user interface 513 allows a user to define policies for adjustments to the configuration information of the computing device during operation, as well as adjustments to peripheral devices assigned to the computing device. In one example, during operation, the management processor 511 may analyze telemetry data to determine current resource utilization by the computing device. Based on the current utilization, the dynamic adjustment policy may specify allocation or removal of general processing resources, graphics processing resources, storage resources, networking resources, memory resources, and the like, to or from the host device. For example, the telemetry data may indicate that the current utilization level of the computing device's allocated storage resources is approaching a threshold level, and an additional storage device may be allocated to the host device.

[0066] Management processor 511 may provide control and management of a multi-protocol communication fabric, including combining different communication protocols such as PCIe and Ethernet. For example, management processor 511 and devices connected via links 566 and 565 may provide communication coupling of physical components using multiple different implementations or versions of Ethernet, PCIe, and similar protocols. Additionally, next generation interfaces, such as Gen-Z, CCIX, CXL, OpenCAPI, or wireless interfaces, including Wi-Fi interfaces or cellular wireless interfaces, may be utilized. Also, while Ethernet and PCIe are used in FIG. 5, it should be understood that different or additional communication links or buses may instead be utilized, such as NVMe, SAS, FibreChannel, Thunderbolt, SATA Express, among other interconnect, network, and link interfaces.

[0067] Referring now to the description of the components of the computer system 500, the management processor 511 may comprise one or more microprocessors and other processing circuitry that retrieves and executes software for management of the operating system 512, the user interface 513, and the interaction unit 514, or components or modules, or any combination thereof, from an associated storage system. The management processor 511 may be implemented in a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of the management processor 511 include general-purpose central processing units, application-specific processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof. In some examples, the management processor 511 comprises an Intel®, AMD® microprocessor, Apple® microprocessor, ARM® microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), application-specific processor, or other microprocessor or processing element.

[0068] A management operating system (OS) 512 is executed by the management processor 511 and provides management of the resources of the computer system 500 as well as execution of the user interface 513 and the interaction unit 514. The management OS 512 provides the functionality and operations described herein for the management processor 511, and in particular for the user interface 513 and the interaction unit 514.

[0069] The user interface 513 may present a graphical user interface (GUI), an application programming interface (API), a REST interface, a Rest API, a command line interface (CLI), a WebSocket interface, or other interface to one or more users. The user interface 513 may be utilized by end users or administrators to configure computing devices and peripheral devices, establish configurations of computing devices and peripheral devices, receive status, telemetry, and other notifications regarding the operation of computing devices and peripheral devices, and perform other operations. The user interface 513 may be utilized to manage, select, and modify templates. The user interface 513 may be utilized to manage, select, and modify policies. The user interface 513 may also provide telemetry information for the operation of the computer system 500 to the user, such as in one or more status interfaces or status views. The status of various components or elements of the computer system 500 may be monitored via the user interface 513, such as CPU status, GPU status, NIC status, SSD status, Ethernet status, PCIe switch / fabric status, among others. Various performance metrics, error status can be monitored using the user interface 513.

[0070] The interaction unit 514 provides various fabric interaction functions and operations described herein along with discovery, attachment, and removal of peripheral devices with respect to a computing device. Specifically, the interaction unit 514 may discover computing devices and peripheral devices in the computing system 500 coupled via communication links (e.g., 565-566), receive indications of available peripheral devices located on the computing devices, and catalog these devices in one or more data structures 590. The data structures 590 are illustrated by an exemplary server device data structure 591 and a peripheral device data structure 592. The cataloging may include noting device identifiers, vendor identifiers, addresses, ports, sockets, assignments or attachments between devices, or other properties. The interaction unit 514 may receive operator instructions via a user interface 513 from the computing device to establish an arrangement between the computing device and one or more peripheral devices (e.g., attaching a peripheral device from a host device to be emulated in a client device). The instructions may identify which computing device and which peripheral device or types of peripheral devices are to be coupled. In response to the instruction, the interaction unit 514 initiates the attachment of one or more peripheral devices from the available device pool to the client computing device, such as by issuing one or more attach commands to the PoF unit of the client device. The attach instructions alert the PoF unit of the client device to initiate instantiation of the peripheral devices to the local peripheral device interconnect interface of the client device by at least emulating the behavior of the peripheral devices coupled through the network interface as local peripheral devices coupled to the peripheral device interconnect interface of the client system. The interaction unit 514 may also notify the host device of the linked peripheral device that the peripheral device has been assigned or attached to the client device.The interworking unit 514 may then interwork traffic between the client system and the host system along the network link 566, and in some examples, between the network link 566 and the PCIe link 565. The interworking unit 514 may receive instructions to detach the peripheral device from the client device and issue a detach command to the PoF unit of the client device. Once detached, the peripheral device may be returned to a pool of free devices for later use by other computing devices.

[0071] An example of an attachment operation is shown in Fig. 5. An instruction to attach GPU 532 (physically located in computing device 530) to computing device 540 may be received, and interaction unit 514 may provide an attach command and an identifier of GPU 532 to computing device 540 via corresponding network link 566. Once attached, computing device 540 may become a client device for GPU 532, and computing device 530 may be a host device. Table 591 shows a target computing device identifier of 001236 (corresponding to client device 540) with an IP address of 10.10.10.3, and a corresponding socket (not shown) corresponding to a port number. Table 592 shows GPU 532 with vendor ID and device ID provided to target client device 540 for attachment. To allow proper exchange of communications between the GPU 532 and the target client device 540 over the communications fabric without interference with other peripheral and computing devices, logical isolation of the GPU 532 may be established between the GPU 532 and the management processor 511 in the communications fabric. This arrangement may be referred to as a logical domain 570, which includes elements of the communications links (e.g., Ethernet link 566) between the host 530, the client 540, the fabric switch circuitry 510, the fabric switch circuitry itself, and the GPU 532. Communications may be directed from the client device 540 to the GPU 532 in the host device 530 over the network link 566 using the device ID and vendor ID of the GPU 532 intercepted and interpreted by the interworking unit 514 using a socket corresponding to the host device 530, i.e., the IP address of 10.10.10.2 and the associated port number. Communications of GPU 532 to client device 540 may be directed over network link 566 using the IP address of client device 540, ie, 10.10.10.3, to be intercepted and interpreted by interaction unit 514.

[0072] Multiple instances of elements 511-514 (e.g., two or more dynamic server rebalancing systems) may be included in computer system 500. A user command, such as one received via a GUI, may be received by any of the management instances and forwarded by the receiving management instance to a handling management instance. Each management instance may have a unique or pre-assigned identifier that may assist in the delivery of the user command to the appropriate management instance. Additionally, the management processors of each management instance may communicate with each other, such as using a mailbox process or other data exchange technique. This communication may be via a dedicated sideband interface, such as an I2C interface, or via a PCIe or Ethernet interface coupling each management processor.

[0073] Multiple CPUs 521, 531, 541 are included in the system 500. Each CPU may comprise a CPU module including one or more CPUs or microprocessors and other processing circuitry that retrieves and executes software, such as operating systems, device drivers, and applications from an associated storage system. Each CPU may be implemented in a single processing device, but may be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of each CPU include general-purpose central processing units, application-specific processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof. In some examples, each CPU comprises an Intel® microprocessor, an Apple® microprocessor, an AMD® microprocessor, an ARM® microprocessor, a graphics processor, a computational core, a graphics core, an ASIC, an FPGA, or other microprocessor or processing element. Each CPU may also communicate with other CPUs, such as CPUs in the same storage assembly / enclosure or another storage assembly / enclosure, via one or more PCIe interfaces and a PCIe fabric.

[0074] Multiple GPUs 522, 532, 542, 551-555 and TPUs 523, 533, 543 are included in the system 500, which may represent any type of CoCPU. Each GPU may comprise a GPU module including one or more GPUs. Each GPU includes graphics processing resources that may be allocated to one or more host devices. A GPU may include graphics processors, shaders, pixel rendering elements, frame buffers, texture mappers, graphics cores, graphics pipelines, graphics memory, or other graphics processing and processing elements. In some examples, each GPU comprises a graphics "card" that includes circuitry supporting the GPU chip. Exemplary GPU cards include nVIDIA® or AMD® graphics cards that include graphics processing elements along with various support circuits, connectors, and other elements. Similarly, each tensor processing unit (TPU) may comprise a TPU module including one or more TPUs. Each TPU may include circuitry and resources for AI acceleration and processing configured for neural network machine learning applications. In further examples, other styles of graphics processing units, graphics processing assemblies, or co-processing elements may be utilized, such as machine learning processing units, AI accelerators, FPGAs, ASICs, or other specialized processors that may include specialized processing elements to focus processing and memory resources for processing specialized data sets.

[0075] A number of NICs 524, 534, 544 are included within the system 500, each having an associated MAC address or Ethernet address. Each NIC may comprise a NIC module including one or more NICs. Each NIC may comprise a network interface controller card for communicating over a TCP / IP (Transmission Control Protocol (TCP) / Internet Protocol) network or for carrying user traffic, such as iSCSI (Internet Small Computer System Interface) or NVMe (NVM Express) traffic for components of an associated host device. The NICs comprise Ethernet interface equipment and may communicate over wired, optical, or wireless links. External access to components of the computer system 500 may be provided over a packet network link provided by the NIC. The NICs may communicate with other components of an associated host device over associated PCIe links of a PCIe fabric. In some examples, the NICs are provided to communicate with a management processor 511 over an Ethernet link. In additional examples, the NICs are provided to communicate with one or more other chassis, rack mount systems, data centers, computer platforms, communication fabrics, or other components over an Ethernet link.

[0076] A number of SSDs 525, 535, 545 are included in the system 500. Each SSD may comprise an SSD module including one or more SSDs. Each SSD includes one or more storage drives, such as a solid-state storage drive with a PCIe interface. Each SSD also includes a PCIe interface, a control processor, and power system elements. Each SSD may include a processor or control system for traffic statistics and status monitoring, among other operations. In yet another example, each SSD instead comprises a different data storage medium, such as a magnetic hard disk drive (HDD), a cross-point memory (e.g., an Optane® device), a static random access memory (SRAM) device, a programmable read-only memory (PROM) device, or other magnetic, optical, or semiconductor-based storage medium, along with associated enclosures, control systems, power systems, and interface circuits.

[0077] In addition to the CPU, GPU, TPU, SSD, and NIC, other dedicated devices may be utilized in the computing platform. These other dedicated devices may include co-processing modules with various memory, storage, and interface components, as well as dedicated co-processing circuitry, fabric-coupled RAM devices, ASIC circuitry, or FPGA circuitry, among other circuitry. The other dedicated devices may each include a PCIe interface or an Ethernet interface, thereby incorporating the network fabric of system 500, either directly or by attachment to a computing device, for remote access. These other dedicated devices may include PCIe endpoint devices or other computing devices that may or may not have a root complex.

[0078] An FPGA device may be used as an example of another dedicated device. An FPGA device may receive processing tasks from another peripheral device, such as a CPU or GPU, and offload those processing tasks to an FPGA programmable logic circuit. An FPGA is typically initialized to a program state using configuration data, which includes various logic arrangements, memory circuits, registers, processing cores, specialized circuits, and other functions that provide specialized or application-specific circuits. In addition to changing the circuits implemented therein, an FPGA device may be reprogrammed to perform different sets of processing tasks at different times. An FPGA device may be used to perform machine learning tasks, implement artificial neural network circuits, implement custom interfaces or glue logic, perform encryption / decryption tasks, perform blockchain computation and processing tasks, or other tasks. In some examples, the CPU will provide the FPGA with data to be processed by the FPGA locally or remotely via a PCIe interface. The FPGA may process the data to generate results and provide the results to the CPU via a PCIe interface. Two or more CPUs and / or FPGAs may be involved to parallelize tasks across two or more devices or to serially process data through two or more devices. In some examples, the FPGA arrangement may include locally stored configuration data that may be supplemented, replaced, or overwritten using configuration data stored in a configuration data storage. This configuration data may comprise firmware, programmable logic programs, bitstreams, or objects, PCIe device initial configuration data, among other configuration data described herein. The FPGA arrangement may also include SRAM or PROM devices used to perform boot programming, power-on configuration, or other functions to establish the initial configuration of the FPGA device. In some examples, the SRAM or PROM devices may be incorporated into the FPGA circuit or package.

[0079] The blade server motherboards 561-563 may include printed circuit boards or backplanes to which computer components may be attached or connected. For example, peripheral devices 521-525, 531-535, and 541-545 may be connected to PCIe ports or other slots of the blade server motherboards 561-563. The blade server motherboards 561-563 may each include one or more network switches or ports (not shown) for connecting to a network link 566, such as an Ethernet switch. The blade server motherboards 561-563 communicate with other components of the system 500 via the network link 566, thereby allowing access to remote peripheral devices and enabling external devices to access and utilize resources of local peripheral devices, such as devices 521-525 for chassis 520, devices 531-535 for chassis 530, and devices 541-545 for chassis 540. The blade server motherboards 561-563 may logically interconnect the devices of the system 500 for management by the management processor 511. Attach or detach commands for remote peripheral devices may be sent or received through the blade server motherboards 561-563 via network link 566, and the blade server motherboards 561-563 may receive a list of available resources from the management processor 511 or issue requests to the management processor 511 to access remote resources.

[0080] The PCIe switch 564 may communicate with other components of the system 500 via associated PCIe links 565. In the example of FIG. 5, the PCIe switch 564 may be used to carry traffic between PCIe devices in the chassis 564 and a switching unit in the chassis 510, from which the traffic may be directed to other chassis via network links 566. The PCIe switch 564 may comprise a PCIe cross-connect switch for establishing switched connections between any PCIe interfaces handled by the PCIe switch in the system 500. The PCIe switches described herein may logically interconnect various ones of the associated PCIe links based at least on the traffic carried by each PCIe link. In these examples, a domain-based PCIe signaling distribution may be included that may segregate the PCIe ports of the PCIe switch according to user-defined groups. The user-defined groups may be managed by a management processor 511 that logically groups components into associated logical units and logically isolates the components and logical units from each other. In addition to or alternatively to domain-based isolation, each PCIe switch port can be a non-transparent (NT) or transparent port. An NT port allows some logical isolation between endpoints, like a bridge, while a transparent port does not allow logical isolation and has the effect of connecting endpoints in a purely switched configuration. Access through one or more NT ports may involve an additional handshake between the PCIe switch and the initiating endpoint to select a particular NT port or to enable visibility through the NT port.

[0081] In a further example, memory-mapped direct memory access (DMA) conduits may be formed between individual CPU / PCIe device pairs. This memory mapping may occur on the PCIe fabric address space, among other configurations. To provide these DMA conduits on a shared PCIe fabric with many CPUs and GPUs, the logical partitioning described herein may be utilized. Specifically, NT port or domain-based partitioning on the PCIe switch may isolate individual DMA conduits between associated CPUs / GPUs. The PCIe fabric has a 64-bit address space, which allows for 264 bytes of addressable space, resulting in at least 16 exbibytes of byte-addressable memory. The 64-bit PCIe address space may be shared by all compute units or may be separated among the various compute units forming an arrangement for appropriate memory mapping to resources.

[0082] The PCIe interface supports multiple bus widths, such as x1, x2, x4, x8, x16, and x32, and each multiple of the bus width may have additional "lanes" for data transfer. PCIe also supports transfer of sideband signaling, such as associated clocks, power, and bootstrap, in addition to the System Management Bus (SMBus) interface and the Joint Test Action Group (JTAG) interface, among other signaling. PCIe may also have different implementations or versions as used herein. For example, PCIe version 3.0 or later (e.g., 4.0, 5.0, or later) may be used. Furthermore, next generation interfaces, such as Gen-Z, Cache Coherent CCIX, CXL, or OpenCAPI, may be used. Also, while PCIe is used in FIG. 5, it should be understood that a different communication link or bus may be used instead, such as NVMe, Ethernet, SAS, FibreChannel, Thunderbolt, SATA Express, among other interconnect, network, and link interfaces. NVMe is an interface standard for mass storage devices, such as hard disk drives and solid-state memory devices. NVMe can replace SATA interfaces for interfacing with mass storage devices in personal computer and server environments. However, these NVMe interfaces are limited to a one-to-one host-drive relationship, similar to SATA devices. In the examples described herein, a PCIe interface can be utilized to carry NVMe traffic and present a multi-drive system with many storage drives as one or more NVMe virtual logical unit numbers (VLUNs) via the PCIe interface.

[0083] Any of the links in Figure 5 may each use a variety of communication media, such as air, space, metal, optical fiber, or any other signal propagation path including combinations thereof. Any of the PCIe links in Figure 5 may include any number of composite PCIe links or single / multi-lane configurations. Any of the links in Figure 5 may each be direct links or may include a variety of equipment, intermediate components, systems, and networks. Any of the links in Figure 5 may each be common links, shared links, aggregated links, or may be composed of individual separate links.

[0084] A simple example of a compute unit formation and attachment of peripheral components from a host device to a remote client device is now described. In FIG. 5, a configurable logical visibility may be provided to the computing device of any / all available peripheral devices or resources coupled to the network fabric of computer system 500, as listed in data structure 592. The computing device may request access to available resources and may logically connect remote peripheral devices as if they were local resources. The logically linked components may form a compute unit. For example, CPU 521, GPU 522, and SSD 525 of chassis 520 may request and be granted access to remote GPU 532 from chassis 530, NIC 544 from chassis 540, and GPU 551 from chassis 550, forming one compute unit. Thus, in some of the following examples, "m" SSDs or GPUs may be combined with "n" CPUs to enable a large, scalable architecture with high levels of performance, redundancy, and density. This division allows the GPUs to be interacted with one or more CPUs as desired, and more than one GPU may be associated with a particular computing unit, such as eight GPUs. The management processor 511 may be configured to update one or more data structures maintained by the management processor 511 as resources or peripheral devices are attached or detached from various computing devices or units.

[0085] 6 is a block diagram illustrating an implementation of a management processor 600. The management processor 600 illustrates an example of any of the control elements, control systems, interaction units, dynamic server rebalancing systems, PoF units, fabric control elements, or management processors described herein, such as the PoF system 115 of FIG. 1, the dynamic server rebalancing system 130 of FIG. 1, the PoF unit 320 of FIG. 3, the server rebalancing control element 410 of FIG. 4, or the management processor 511 of FIG. 5. The management processor 600 includes a communication interface 601, a user interface 603, and a processing system 610. The processing system 610 includes processing circuitry 611 and a data storage system 612, which may include a random access memory (RAM) 613, but may include additional or different configurations of elements.

[0086] Processing system 610 is intended to generally represent a computer system on which at least software 620 is deployed and executed to render or otherwise perform the operations described herein. However, processing system 610 may also represent any computer system on which at least software 620 and data 630 are staged and from which software 620 and data 630 may be distributed, transmitted, downloaded, or provided to another computer system for deployment, execution, or further distribution. Processing circuitry 611 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of processing circuitry 611 include general-purpose central processing units, microprocessors, application-specific processors, and logic devices, as well as any other type of processing device. In some examples, processing circuitry 611 includes physically distributed processing devices, such as a cloud computer system.

[0087] The communication interface 601 includes one or more communication fabrics and / or network interfaces for communicating over an Ethernet link, a PCIe link, a packet network, a network such as the Internet, or the like. The communication interface 601 may include one or more local or wide area network communication interfaces that may communicate over an Ethernet interface, a PCIe interface, a serial interface, a serial peripheral interface (SPI) link, an inter-integrated circuit (I2C) interface, a universal serial bus (USB) interface, a UART interface, a wireless interface, or an Ethernet or Internet Protocol (IP) link. The communication interface 601 may include a network interface configured to communicate using one or more network addresses that may be associated with different network links. Examples of the communication interface 601 include network interface controller devices, transceivers, modems, and other communication circuitry. The communication interface 601 may communicate with a control element of a network or other communication fabric to establish logical partitioning or remote resource allocation within the fabric, such as via a management or control interface of one or more communication switches of the communication fabric. The communication interface 601 may communicate over a PCIe fabric to exchange traffic / communications with peripheral devices.

[0088] The user interface 603 may include a software-based interface or a hardware-based interface. Hardware-based interfaces include touch screens, keyboards, mice, voice input devices, audio input devices, or other touch input devices for receiving input from a user. Output devices such as displays, speakers, web interfaces, terminal interfaces, and other types of output devices may also be included in the user interface 603. The user interface 603 may provide output and receive input via a network interface, such as the communication interface 601. In a network example, the user interface 603 may packetize display or graphics data for remote display by a display system or computer system coupled via one or more network interfaces. The physical or logical elements of the user interface 603 may provide an alert or visual output to a user or other operator. The user interface 603 may also include associated user interface software executable by the processing system 610 that supports the various user input and output devices described above. The user interface software and user interface device, separately or in conjunction with each other and other hardware and software elements, may support a graphical user interface, a natural user interface, or any other type of user interface.

[0089] The user interface 603 may present a command line interface (CLI), an application programming interface (API), a graphical user interface (GUI), a REST interface, a Rest API, a WebSocket interface, or other interface to one or more users. The user interface may be utilized by an operator or administrator to assign assets (computing units / resources / peripheral devices) to each host device. In some examples, the user interface provides an interface that allows an end user to determine one or more templates and dynamic tuning policy sets to use or customize for use in creating computing units. The user interface 603 may be utilized to manage, select, modify machine templates, change policies, etc. The user interface 603 may also provide telemetry information, such as in one or more status interfaces or status views. The status of various components or elements may be monitored via the user interface 603, such as processor status, network status, storage unit status, PCIe element status, among others. Various performance metrics, error status may be monitored using the user interface 603.

[0090] The storage system 612 and the RAM 613 may both comprise memory devices or non-transitory data storage systems, although variations are possible. The storage system 612 and the RAM 613 may each comprise any storage medium that is readable by the processing circuit 611 and that may store software and OS images. The RAM 613 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. The storage system 612 may include non-volatile storage media, such as solid-state storage media, flash memory, NAND flash or NOR flash, phase change memory, magnetic memory, or other non-transitory storage media, including combinations thereof. The storage system 612 and the RAM 613 may each be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems. The storage system 612 and the RAM 613 may each comprise additional elements, such as a controller that may communicate with the processing circuit 611.

[0091] The software 620 or data 630 may be stored on or in the storage system 612 or RAM 613, which may comprise some other form of machine-readable processing instructions having computer program instructions, firmware, data structures, or processes that, when executed, direct the processor 600 to operate as described herein. The software 620 may reside in the RAM 613 during the execution and operation of the processor 600 and may reside in a non-volatile portion of the storage system 612 during a powered off state, among other places and states. The software 620 may be loaded into the RAM 613 during a start-up or boot procedure, as described for computer operating systems and applications. The software 620 may receive user input via the user interface 603. This user input may include user commands as well as other inputs, including combinations thereof.

[0092] The software 620 includes applications 621 and an operating system (OS) 622. The software 620 may receive user or computing device commands and drive the processor 600 for attachment or removal of peripheral devices to and from the computing device. The software 620 may drive the processor 600 to receive and monitor telemetry data, statistics, operational data, and other data, provide telemetry to the user, and modify operation according to telemetry data, policies, or other data and criteria. The software 620 may drive the processor 600 to manage peripheral device and computing device resources, establish domain or NT partitioning between communication fabric elements, and interface with individual communication switches and control operation of such communication switches, among other operations. The software 620 may also include user software applications, application programming interfaces (APIs), or user interfaces. The software 620 may be implemented as a single application or multiple applications. In general, the software 620, when loaded and executed on the processing system 610, may transform the processing system 610 from a general-purpose device into a customized, special-purpose device as described herein.

[0093] The software applications 621 may take different forms depending on the operation and device implemented by the management processor 600 and may include a set of applications 640 or 650. For example, when the management processor 600 operates a dynamic server rebalancing system, a set of applications 640 may be deployed that includes a discovery application 641, an event application 642, a fabric interaction application 643, and a fabric user interface application 644. Alternatively, when the management processor 600 operates a computing device such as a blade server, a set of applications 650 may be deployed that includes a server interaction application 651 and a server user interface application 652. The software applications 641-644 and 651-652 each comprise executable instructions that may be executed by the processor 600 to operate a computing system or processing circuitry in accordance with the operations described herein.

[0094] The application set 640 includes a discovery application 641, an event application 642, a fabric interaction application 643, and a fabric user interface application 644. The discovery application 641 may obtain an indication of a computing device and associated local peripheral devices or disaggregated peripheral devices available for remote association with a client computing device. The indication of the computing device or peripheral device may include addressing information, device identifiers, vendor identifiers, device specifications or requirements, associations between devices, or other information. The discovery application 641 may obtain the indication of the computing device and peripheral devices through a network interface, through a PCIe interface, or through other connection links between the devices. The discovery application 641 may store these indications in the data 630. Based on the indication, the event application 642 initiates instantiation and de-instantiation of peripheral devices from the host device or disaggregated peripheral device to the local peripheral device interconnect interface of the client device. The fabric interworking application 643 intercepts client-originated traffic for remote peripheral devices received via the network interface, interprets the client-originated traffic, performs any necessary format conversion to deliver the traffic to the target peripheral device, and routes the client-originated traffic in the appropriate format via the network or PCIe interface for delivery to the peripheral device. Similarly, the fabric interworking application 643 intercepts peripheral device-originated traffic from either the host device or a disaggregated peripheral device that is directed to a client device, interprets the peripheral device-originated traffic, performs any necessary format conversion to forward the traffic to the client device, and routes the peripheral device-originated traffic in the appropriate format via the network interface for delivery to the client device.Fabric user interface application 644 may receive operator instructions to attach or remove peripheral devices to a computing device and present various information, status, telemetry, logs, etc. to the operation via various types of user interfaces. Commands or requests to attach or remove peripheral devices received from a networked computing system may be received via event application 642, fabric interaction application 643, or fabric user interface application 644, depending on the implementation.

[0095] The application set 650 includes a server interaction application 651 and a server host user interface application 652. The server interaction application 561 may interface with the network stack of the computing device to interact with the traffic of the peripheral device with the local peripheral device interconnect interface. As a local peripheral device coupled to the peripheral device interconnect interface of the client system, the server interaction application 561 may emulate the behavior of the peripheral device coupled through the network interface. As a local peripheral device coupled to the client system, the server interaction application 561 emulates a hardware plugging process by at least providing an address space description to the local peripheral device interconnect interface to access the peripheral device. The server interaction application 561 removes the instantiation of the peripheral device from the local peripheral device interconnect interface by at least emulating a hardware removal process in the local peripheral device interconnect interface.

[0096] When instantiated on the local peripheral device interconnect interface of the client device, the device driver of the client device may interface with the peripheral device through the local peripheral device interconnect interface. The server interaction application 561 emulates the behavior of the peripheral device by at least intercepting client-originated traffic for the peripheral device from the local peripheral device interconnect interface, interpreting the client-originated traffic for conversion from a native peripheral device format (such as a PCIe frame or memory-mapped format) to a network format (e.g., frames or packets with associated encapsulation and addressing / headers / footers) suitable for transfer over the network interface, and routing the client-originated traffic in packet format through the network interface for delivery to the peripheral device. The server interaction application 561 emulates the behavior of the peripheral device by at least receiving peripheral device-originated traffic in packet format from the network interface, and interpreting the peripheral device-originated traffic in packet format for conversion to a native peripheral device format suitable for the local peripheral device interconnect interface. The server interaction application 561 initiates the instantiation of a peripheral device to a local peripheral device interconnect interface by at least triggering a start point process in the local peripheral device interconnect interface to emulate a hardware plugging process for a peripheral device having a local peripheral device interconnect interface. In the case of a host server device, the server interaction application 651 may route traffic from remote client devices to local peripheral devices connected to the local peripheral device interconnect interface.For example, client-originated traffic may be received by the host's server interaction application 651 over a network connection and converted to a format for use by a host peripheral device interconnect interface, such as a PCIe frame, such that client-originated traffic may be routed from the server interaction application 651 to a peripheral device interconnect interface and from there to a local peripheral device for processing. In some examples, the server interaction application 651, in addition to executing commands received from a remote client device, may notify a host server system that a local peripheral device assigned to a remote client is unavailable for use by that host and the OS or software local to that host, may send an indication, command, or request over the network indicating the availability status of the local peripheral device, may request to remove the local peripheral device from the remote client, or may request access to a remote peripheral device from an available peripheral device pool.

[0097] The server user interface application 652 provides local instructions for installing and removing peripheral devices to an operator of the computing device, and may receive operator instructions for installing or removing peripheral devices, among other operations.

[0098] In addition to the software 620, other data 630 including various data structures may be stored by the storage system 612 and RAM 613. The data 630 may include templates, policies, telemetry data, event logs, or fabric status. The data 630 may include indications and identifications of peripheral and computing devices. The data 630 may include current assignments of peripheral devices to client devices. The fabric status includes information and properties of various communication fabrics including pools of resources or pools of components, such as fabric type, protocol version, technology descriptors, header requirements, addressing information, and other data. The fabric data may include relationships between components and the particular fabrics to which they connect.

[0099] Various peripheral devices have been described herein that include data processing elements or other computer components coupled via one or more communication fabrics or networks. Various communication fabric or network types may be utilized herein. For example, a Peripheral Component Interconnect Express (PCIe) fabric may be utilized to couple to a CoPU, which may include various versions, such as 3.0, 4.0, or 5.0, among others. In lieu of a PCIe fabric, other point-to-point communication fabrics or buses with associated physical layers, electrical signaling, protocols, and layered communication stacks may be utilized. These may include, among others, Gen-Z, Ethernet, InfiniBand, NVMe, Internet Protocol (IP), Serial Attached SCSI (SAS), Fibre Channel, Thunderbolt, Serial Attached ATA Express (SATA Express), NVLink, Cache Coherent Interconnect for Accelerators (CCIX), Compute Express Link (CXL), Open Coherent Accelerator Processor Interface (OpenCAPI), Wi-Fi (802.11x), or cellular wireless technologies. The communications network couples to the host system and includes Ethernet or Wi-Fi (802.11x), which may refer to any of the various network communications protocol standards and bandwidths available, such as 10BASE-T, 100BASE-TX, 1000BASE-T, 10GBASE-T (10 GB Ethernet), 40GBASE-T (40 GB Ethernet), Gigabit (GbE), Terabit (TbE), 200GbE, 400GbE, 800GbE, or various other wired and wireless Ethernet formats and speeds. Cellular wireless technologies may include various wireless protocols and networks built around 3rd Generation Partnership Project (3GPP) standards, including 4G Long Term Evolution (LTE), 5G New Radio (NR), and related 5G standards, among others.

[0100] Some of the aforementioned signaling or protocol types build on PCIe, thus adding additional functionality to the PCIe interface. Parallel, serial, or combined parallel / serial type interfaces may also be applied to the examples herein. Many of the examples herein utilize PCIe as an exemplary fabric type for coupling to peripheral devices, but it should be understood that others may be used instead. PCIe is a high-speed serial computer expansion bus standard, typically having multi-lane point-to-point connections between hosts and component devices or between peer devices. PCIe typically has multi-lane serial links connecting individual devices to a root complex. A PCIe communication fabric may be established using various switching circuits and control architectures described herein.

[0101] The various computer system components herein may be included in one or more physical enclosures, such as rack-mountable modules that may further be included in a shelf or rack unit. A number of components may be inserted or installed in a physical enclosure, such as a modular framework into which modules may be inserted and removed as required by a particular end user. An enclosed modular system may include physical support structures and enclosures that include circuits, printed circuit boards, semiconductor systems, and structural elements. Modules with components may be insertable into or removable from rack-mount style or rack unit (U) type enclosures. It should be understood that the components herein may be included in any physical mounting environment and need not include associated enclosures or rack-mount elements.

[0102] The functional block diagrams, operational scenarios and sequences, and flow diagrams provided in the figures represent exemplary systems, environments, and methodologies for implementing novel aspects of the present disclosure. For ease of explanation, it is understood and appreciated that the methods included herein may be in the form of functional diagrams, operational scenarios or sequences, or flow diagrams, and may be described as a series of operations, and that the methods are not limited by the order of operations, since some operations may occur in different orders and / or simultaneously with other operations as shown and described herein accordingly. For example, those skilled in the art will understand and appreciate that a method may alternatively be represented as a series of interrelated states or events, such as a state diagram. Moreover, not all operations illustrated in a methodology are required for a novel implementation.

[0103] The description and figures contained herein show specific implementations to teach those skilled in the art how to make and use the best options. Some conventional aspects have been simplified or omitted for the purpose of teaching the principles of the invention. Those skilled in the art will appreciate variations from these implementations that fall within the scope of the disclosure. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form multiple implementations. As a result, the present invention is not limited to the specific implementations described above, but only by the claims and their equivalents.

Claims

1. receiving an indication of a peripheral device available for data processing and located on a first computing device; receiving a request from a second computing device to access the peripheral device; instructing the second computing device based on the request to emulate the peripheral device as a local device installed on the second computing device; routing data traffic from the second computing device for processing by the peripheral device at the first computing device; A method comprising:

2. receiving a second indication of a second peripheral device available for processing and located on the second computing device; receiving a second request from the first computing device to access the second peripheral device; instructing the first computing device to emulate the second peripheral device as a second local device installed on the first computing device based on the second request; routing data traffic from the first computing device to the second computing device for processing by the second peripheral device; The method of claim 1 further comprising:

3. receiving a second indication of a second peripheral device available for processing and located on the first computing device; receiving a second request from the first computing device to access the second peripheral device; allocating the second peripheral device to the first computing device based on the second request; at the first computing device, routing data traffic from the first computing device to the second peripheral device; The method of claim 1 further comprising:

4. maintaining, in a server rebalancing system, a data structure identifying peripheral devices available for processing, including said peripheral device; providing a list from the server rebalancing system to a networked computer device identifying the peripheral devices available for processing; updating the data structure to indicate that the peripheral device is assigned to the second computing device based on the step of instructing the second computing device to emulate the second peripheral device as a local device; routing the data traffic based on the data structure; The method of claim 1 further comprising:

5. receiving a notification at the server rebalancing system regarding removal of the peripheral device from the second computing device; based on the notification, dissociating the peripheral device from the second computing device and updating the data structure to indicate that the peripheral device is available for processing; The method of claim 4 further comprising:

6. discovering, via the peripheral interface, a second peripheral device available for processing; receiving a second request from the second computing device to access the second peripheral device; instructing the second computing system to emulate a local installation of the second peripheral device on the second computing device based on the second request; converting messaging received from the second computing device from a first data format to a second data format; routing messaging in the second data format to the second peripheral device via the peripheral interface; The method of claim 1 further comprising:

7. The method of claim 1 , wherein the instructions and the requests are received via a network interface.

8. obtaining an indication of the peripheral devices available for processing at the second computing device via a network interface; issuing the request from the second computing device to access the peripheral device; emulating the peripheral device as the local device installed on the second computing device based on a response to the request received at the second computing device; issuing the data traffic from the second computer via the network interface for processing by the peripheral device; The method of claim 1 further comprising:

9. issuing, from the first computing device via a network interface, the indication that the peripheral device located at the first computing device is available for processing; receiving at the first computing device via the network interface a second indication that the peripheral device has been assigned to the second computing device; receiving at the first computing device via the network interface the data traffic from the second computing device for processing by the peripheral device; providing results of the processing of the data traffic by the peripheral device from the first computing device via the network interface; The method of claim 1 further comprising:

10. 1. A system comprising: a first computing device including a network interface; the first computing device, obtaining, from a server rebalancing system via the network interface, an indication of peripheral devices available for processing via the network interface, the peripheral devices being located on a second computing device; issuing a request to access the peripheral device to the server rebalancing system via the network interface; emulating the peripheral device as a local device installed on the first computer system based on a response from the server rebalancing system; issuing data traffic to the server rebalancing system via the network interface for processing by the peripheral device; A system configured to implement the above.

11. The system comprises: The first computing device, a network stack configured to exchange data via the network interface; a local peripheral device interconnect interface configured to exchange signals with a local peripheral device; and further comprising the first computing device, the first computing device is configured to emulate the peripheral device as a local device; The first computing device comprises: The system of claim 10 , further comprising interaction traffic for the peripheral device between the network stack and the local peripheral device interconnect interface.

12. the system further comprising the first computing device configured to emulate the peripheral device as a local device; The first computing device comprises: intercepting traffic from the first computing device for the peripheral device from the local peripheral device interconnect interface; interpreting the traffic from the first computing device to convert it from a native peripheral device format to a packet format suitable for transfer over the network interface; and routing the traffic from the first computing device in the packet format through the network interface for delivery to the peripheral device. The system of claim 11.

13. receiving traffic from the peripheral device in packet format from the server rebalancing system via the network interface; emulating the peripheral device as a local device, interpreting traffic from the peripheral device in the packet format for conversion to a native peripheral device format suitable for the local peripheral device interconnect interface; The system of claim 11 , further comprising the first computing device further configured to:

14. the first computing device configured to emulate the peripheral device as a local device; The first computing device comprises: emulating a hardware plugging process for the peripheral device using the local peripheral device interconnect interface; providing an address space description to the local peripheral device interconnect interface for accessing the peripheral device as a local peripheral device coupled to the first computing device; The system of claim 11 further comprising:

15. The server rebalancing system further includes a second network interface and a processor, The processor, receiving, via the second network interface, an indication of the peripheral devices available for processing located on the second computing device; receiving the request to access the peripheral device from the first computing device via the second network interface; instructing the first computing device via the second network interface based on the request to emulate the peripheral device as a local device installed on the first computing device; routing data traffic from the first computing device to the second computing device for processing by the peripheral device; The system of claim 10 configured to implement:

16. the second computing device, a third network interface; and and the second computing device including the peripheral device; The second computing device is issuing the indication to the server rebalancing system via the third network interface that the peripheral device located on the second computing device is available for processing; receiving a second indication from the server rebalancing system via the third network interface that the peripheral device is to be assigned to the first computing device; receiving the data traffic from the first computing device via the third network interface from the server rebalancing system for processing by the peripheral device; providing results of the processing of the data traffic by the peripheral device from the second computing device to the server rebalancing system via the third network interface; The system of claim 10 configured to implement:

17. issuing an indication from a first computing device via a network interface to a server rebalancing system that a peripheral device located at the first computing device is available for processing; receiving at the first computing device from the server rebalancing system via the network interface a second indication that the peripheral device is to be assigned to a second computing device; receiving data traffic from the server rebalancing system at the first computing device via the network interface from the second computing device for processing by the peripheral device; providing results of the processing of the data traffic by the peripheral device from the first computing device via the network interface; A method comprising:

18. receiving, at the first computing device, the data traffic in a packet format suitable for transfer over the network interface; interpreting said packet-formatted data traffic for conversion to a native peripheral device format suitable for said peripheral device; processing the data traffic in the peripheral device; interpreting a result of said processing from said native peripheral format to said packet format; providing the results in the packet format to the server rebalancing system via the network interface; 20. The method of claim 17, further comprising:

19. receiving at the server rebalancing system via the network interface an indication of the peripheral devices available for processing on a first computing device; receiving, at the server rebalancing system, a request to access the peripheral device from the second computing device via the network interface; instructing the second computing device, via the network interface, by the server rebalancing system based on the request, to emulate the peripheral device as a local device installed on the second computing device; routing data traffic from the second computing device through the server rebalancing system to the first computing device for processing by the peripheral device; 20. The method of claim 17, further comprising:

20. obtaining at the second computing device from the server rebalancing system an indication of the peripheral devices available for processing; issuing a request to access the peripheral device from the second computing device via the network interface to the server rebalancing system; emulating the peripheral device as a local device installed on the second computer system based on a response from the server rebalancing system at the second computer system; issuing the data traffic from the second computer system via the network interface to the server rebalancing system for processing by the peripheral device; 20. The method of claim 17, further comprising:

Images