Thread-level boot management control of a compute node for context switching using a boot controller

Abstract

A method for performing a system boot includes receiving boot configuration instructions at a board management controller (BMC) for booting a compute node with an operating system, the compute node being placed on a sled that includes a plurality of compute nodes, the BMC configured to manage a plurality of communication interfaces that provide communication to the plurality of compute nodes; transmitting the boot instructions from the BMC via the communication interface to a boot controller of the compute node to execute basic input / output system (BIOS) firmware stored external to the compute node; and executing the BIOS firmware on the compute node to initiate loading of an operating system to be executed by the compute node.

Description

[Technical field]

[0001] The present disclosure relates to remote activation of computing resources, including remote boot control of compute nodes, such as compute nodes of a streaming array of compute threads in a rack assembly, and network reconfiguration of a rack assembly in a data center. [Background technology]

[0002] In recent years, online services that enable online or cloud gaming in a streaming format between cloud gaming servers and clients connected over a network have been continuously promoted. The streaming format has become increasingly popular due to the availability of game titles on demand, the ability to execute more complex games, the ability to network players together in the case of multiplayer games, the ability to share assets between players, the ability to share an instantaneous experience between players and / or spectators, the ability of friends to watch friends play video games, and the ability of friends to join in the ongoing gameplay of friends.

[0003] A data center may be configured with multiple computing resources to support online or cloud gaming. For example, each computing resource may be configured to execute a gaming application for gameplay of the gaming application, which may then be streamed to a user. Demand for computing resources may vary depending on one or more parameters, including duration of demand, geographic area of ​​demand, type of game being sought, etc. Limited demand for computing resources for online gaming may result in periods of idle computing resources.

[0004] A data center dedicated to online gaming may be limited to making short-term changes to computing resources to handle various other computing services that are different from gaming. That is, because the computing resources are configured for gaming, they are not configured to provide other types of services that require applications that require a different computing resource platform. When the computing resources are statically configured, it may not be possible to change the configuration of these computing resources to support these other services. Changing the configuration of these computing resources may also prove difficult if configuration parameters must be changed locally at the computing resources before any configuration changes are implemented.

[0005] It is against this background that embodiments of the present disclosure have been made. Summary of the Invention

[0006] Embodiments of the present disclosure relate to providing remote booting of computing resources, including remote boot control of compute nodes, such as compute nodes of a streaming array of compute threads in a rack assembly, and network reconfiguration of a rack assembly in a data center.

[0007] An embodiment of the present disclosure discloses a method for performing a system boot. The method includes receiving a boot configuration instruction at a board management controller (BMC) for booting a compute node using an operating system, the compute node being arranged on a sled including a plurality of compute nodes, the BMC being configured to manage a plurality of communication interfaces providing communication to the plurality of compute nodes. The method includes sending the boot instruction from the BMC via the communication interface to a boot controller of the compute node to execute a basic input / output system (BIOS) firmware stored external to the compute node. The method includes effecting execution of the BIOS firmware on the compute node to initiate loading of an operating system to be executed by the compute node.

[0008] Another embodiment of the present disclosure discloses a method. The method includes detecting at a cloud managed controller a decrease in demand for a first priority service supported by a data center including a plurality of rack assemblies, each of the plurality of rack assemblies configured in a first configuration that facilitates the first priority service, the cloud managed controller managing configurations of the plurality of rack assemblies, the first priority service being implemented by a first plurality of applications. The method includes sending a reconfiguration message from the cloud managed controller to a rack controller of the rack assembly to reconfigure the rack assembly from the first configuration to a second configuration, the second configuration facilitating the second priority service, the second priority service having a lower priority than the first priority service, the second plurality of services being implemented by a second plurality of applications. The method includes configuring the rack assembly in the second configuration. Each rack assembly of the plurality of rack assemblies includes one or more network storage and one or more streaming arrays, each streaming array including one or more compute threads, each compute thread including one or more compute nodes.

[0009] Another embodiment of the present disclosure discloses a non-transitory computer readable medium storing a computer program for performing a system boot. The non-transitory computer readable medium includes program instructions for receiving, at a board management controller (BMC), boot configuration instructions for booting a computing node using an operating system, the computing nodes being arranged on a sled including a plurality of computing nodes, the BMC being configured to manage a plurality of communication interfaces providing communication to the plurality of computing nodes. The non-transitory computer readable medium includes program instructions for program instructions for sending boot instructions from the BMC via the communication interface to a boot controller of the computing node to execute a basic input / output system (BIOS) firmware stored external to the computing node. The non-transitory computer readable medium includes program instructions for program instructions for executing the BIOS firmware on the computing node and initiating the loading of an operating system to be executed by the computing node.

[0010] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrated by way of example of the principles of the disclosure.

[0011] The present disclosure is best understood by reference to the following detailed description taken in conjunction with the accompanying drawing figures. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a diagram of a game cloud system for providing games over a network among one or more computing nodes located in one or more data centers, according to one embodiment of the disclosure. [Figure 2A] FIG. 1 illustrates a diagram of multiple rack assemblies including multiple computing nodes in a representative data center of a game cloud system, according to one embodiment of the present disclosure. [Figure 2B]FIG. 1 illustrates a diagram of multiple rack assemblies including multiple computing nodes in a representative data center of a game cloud system, each networked storage accessible by a corresponding array of computing nodes, according to one embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a diagram of a compute sled including multiple compute nodes arranged in a rack assembly configured for remote boot control of the compute nodes using a board management controller, according to one embodiment of the disclosure. [Figure 4] FIG. 2 is a flow diagram illustrating steps in a method for performing a remote boot of a compute node using a board management controller, according to one embodiment of the present disclosure. [Diagram 5] FIG. 2 is a flow diagram illustrating steps in a method for reconfiguring a rack assembly of a data center according to one embodiment of the present disclosure. [Figure 6] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Although the following detailed description contains many specific details for purposes of explanation, one of ordinary skill in the art will recognize that many variations and modifications to the following details are within the scope of the disclosure. Accordingly, the aspects of the disclosure described below are set forth without any loss of generality to, and without imposing limitations on, the claims that follow this description.

[0014] Generally speaking, embodiments of the present disclosure provide remote boot control of compute nodes, such as compute nodes of a streaming array of compute threads of a rack assembly. Notably, no external hardware is required for the boot process of the compute nodes, allowing flexibility as to which software is used for the external boot. Also, embodiments of the present disclosure provide network reconfiguration of a rack assembly of a data center. Notably, dark time utilization of computing resources of a data center is achieved through reconfiguration of the networking of the rack assembly, such as reconfiguring the internal networking to provide communication between compute nodes within the rack assembly via network interfaces.

[0015] Based on the foregoing general understanding of the various embodiments, illustrative details of the embodiments will now be described with reference to the various figures.

[0016] Throughout the specification, references to an "application" or a "game" or a "video game" or a "game application" or a "game title" are meant to represent any type of interactive application that is directed through the execution of input commands. By way of example only, interactive applications include applications for games, word processing, video processing, video game processing, and the like. Additionally, the terms introduced above are interchangeable.

[0017] 1 is a diagram of a system 100 for providing games over a network 150 between one or more computing nodes located in one or more data centers, according to an embodiment of the disclosure. The system is configured for providing games over a network between one or more cloud gaming servers, and more specifically, for remote boot control of computing nodes, such as computing nodes of a streaming array of computing threads in a rack assembly, and for network reconfiguration of the rack assembly of the data center to provide better utilization of computing resources (e.g., multiple purposes) in the data center. Cloud gaming involves running a video game at a server to generate game-rendered video frames, which are then sent to a client for display.

[0018] It is also understood that cloud gaming and / or other services can be executed using physical machines (e.g., central processing units (CPUs) and graphics processing units (GPUs)), or virtual machines, or a combination of both, in various embodiments (e.g., within a cloud gaming environment or a standalone system). For example, virtual machines (e.g., instances) can be created using a hypervisor on host hardware (e.g., located in a data center) that utilizes one or more components of a hardware layer, such as multiple CPUs, memory modules, GPUs, network interfaces, communication components, and the like. These physical resources can be arranged in racks, such as a rack of CPUs, a rack of GPUs, a rack of memory, and the physical resources of the racks can be accessed using a top-of-rack switch that facilitates a fabric for assembling and accessing the components used for the instances (e.g., when building the virtualized components of the instances). In general, the hypervisor can present multiple guest operating systems of multiple instances that are configured with virtual resources. That is, each operating system may be configured with a corresponding set of virtualized resources supported by one or more hardware resources (e.g., located in a corresponding data center). For example, each operating system may be supported by a virtual CPU, multiple virtual GPUs, virtual memory, virtualized communication components, etc. Additionally, the configuration of an instance may be transferred from one data center to another to reduce latency. Instant-use defined for a user or game may be used when saving a game session of the user. Instant-use may include any number of configurations described herein to optimize fast rendering of video frames for a game session. In one embodiment, instant-use defined for a game or user may be transferred between data centers as configurable settings. The ability to transfer instant-use settings allows for efficient migration of game play from data center to data center when users connect to play games from different geographic locations.

[0019] System 100 includes a game cloud system 190 implemented across one or more data centers (e.g., data centers 1 through N). As shown, an instance of game cloud system 190 may be located in data center N providing management functions, and the management functions of game cloud system 190 may be distributed across multiple instances of game cloud system 190 at each data center. In some implementations, game cloud system management functions may be located outside of any of the data centers.

[0020] The game cloud system 190 includes an assigner 191 configured to assign each of the client devices (e.g., 1-N) to corresponding resources in a corresponding data center. In particular, when the client device 110 logs into the game cloud system 190, the client device 110 may be connected with an instance of the game cloud system 109 at data center N, which may be geographically closest to the client device 110. The assigner 191 may perform diagnostic tests to determine available transmit and receive bandwidth to the client device 110. The diagnostic tests may also include determining latency and / or round trip time between the corresponding data center and the client device 110. Based on the tests, the assigner 191 may assign resources very specifically to the client device 110. For example, the assigner 191 may assign a particular data center to the client device 110. Additionally, the assigner 191 may assign a particular compute node of a particular compute thread, of a particular streaming array, of a particular rack assembly, to the client device 110. The allocation may be performed based on knowledge of assets (e.g., games) available at the compute nodes. Previously, client devices were generally assigned to data centers without further assignment to rack assemblies. In this manner, the assigner 191 may assign client devices requesting execution of a particular computationally intensive gaming application to compute nodes that may not be running the computationally intensive application. Additionally, load management of the assignment of compute-intensive gaming applications requested by clients may be performed by the assigner 191. For example, the same compute-intensive gaming application requested for a short period of time may be distributed to different compute nodes in different compute threads in one rack assembly or different rack assemblies to reduce the load on a particular compute node, compute thread and / or rack assembly.

[0021] In some embodiments, the allocation may be performed based on machine learning. In particular, resource demand may be predicted for a particular data center and its corresponding resources. For example, if a data center can predict that it will soon be handling many clients running computationally intensive gaming applications, the assigner 191 can assign client devices 110 based on that information, allocating resources that may not currently be utilizing all of their resource capabilities. In another case, assigner 191 may switch client device 110 from game cloud system 190 in data center N to resources available in data center 3 in anticipation of increased load at data center N. Additionally, future clients may be assigned resources in a distributed manner such that resource load and demand may be distributed across the entire game cloud system, across multiple data centers, across multiple rack assemblies, across multiple computational threads, and / or across multiple computational nodes. For example, client device 110 may be assigned resources from game cloud systems in both data center N (e.g., via path 1) and data center 3 (e.g., via path 2).

[0022] Once a client device 110 has been assigned to a particular computational node of a corresponding computational thread of a corresponding streaming array, the client device 110 connects to the corresponding data center via a network, i.e., the client device 110 may communicate with a data center different from the data center that performs the assignment, such as data center 3.

[0023] System 100 provides games via game cloud system 190, which, according to one embodiment of the disclosure, are executed remotely from client devices (e.g., thin clients) of corresponding users who are playing the games. System 100 may provide game control to one or more users who play one or more games through cloud gaming network or game cloud system 190 via network 150 in either single player or multiplayer mode. In some embodiments, the cloud gaming network or game cloud system 190 may include multiple virtual machines (VMs) running on a host machine's hypervisor, where one or more virtual machines are configured to run a game processor module that utilizes hardware resources available to the host's hypervisor. The network 150 may include one or more communication technologies. In some embodiments, the network 150 may include fifth generation (5G) network technologies having advanced wireless communication systems.

[0024] In some embodiments, communication may be facilitated using wireless technology. Such technologies may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks in which a provider's coverage area is divided into small geographical areas called cells. Analog signals representing sound and images are digitized in the phone, converted by an analog-to-digital converter, and transmitted as a bit stream. All 5G wireless devices within a cell communicate with a local antenna array and low-power automatic transceiver (transmitter and receiver) within the cell by electromagnetic waves over frequency channels assigned by the transceiver from a frequency pool reused by other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cellular networks, mobile devices moving from one cell to another are automatically transferred to the new cell. It should be understood that 5G networks are just one example type of communication network, and embodiments of the present disclosure may utilize previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies following 5G.

[0025] As shown, the system 100, including the game cloud system 190, can provide access to multiple game applications. In particular, each client device may be requesting access to a different game application from the cloud gaming network. For example, the game cloud system 190 can provide one or more game servers, which can be configured as one or more virtual machines running on one or more hosts to execute corresponding game applications. For example, the game server can manage virtual machines supporting game processors that instantiate instances of users' game applications. Thus, multiple game processors of the one or more game servers associated with multiple virtual machines are configured to execute multiple instances of one or more game applications associated with game play of multiple users. In this manner, the backend server support provides streaming of gameplay media (e.g., video, audio, etc.) for multiple game applications to multiple corresponding users. That is, the game servers of the game cloud system 190 are configured to stream data (e.g., rendered images and / or frames of corresponding gameplay) back to corresponding client devices over the network 150. In that way, computationally complex game applications can continue to run on the backend servers in response to controller inputs received and forwarded by the client devices. Each server can render images and / or frames, which are then encoded (e.g., compressed) and streamed to corresponding client devices for display.

[0026] In an embodiment, each virtual machine defines a resource environment capable of supporting an operating system on which gaming applications can be executed. In one embodiment, the virtual machine can be configured to emulate the hardware resource environment of a gaming console, and an operating system associated with the gaming console is executed on the virtual machine to support the execution of gaming titles developed for that gaming console. In another embodiment, the operating system can be configured to emulate the native operating system environment of the gaming console, but the underlying virtual machine may or may not be configured to emulate the hardware of the gaming console. In another embodiment, an emulator application is executed on the operating system of the virtual machine, and the emulator is configured to emulate the native operating system environment of the gaming console to support gaming applications and / or video games designed for that gaming console. It should be appreciated that a variety of current and traditional gaming consoles can be emulated in the cloud-based gaming system. In this manner, users can access gaming titles from a variety of gaming consoles via the cloud gaming system.

[0027] In one embodiment, the cloud gaming network or game cloud system 190 is a distributed game server system and / or architecture. In particular, a distributed game engine that executes game logic is configured as a corresponding instance of a corresponding game application. In general, a distributed game engine takes each function of the game engine and distributes them to be executed by multiple processing entities. Individual functions may be further distributed across one or more processing entities. The processing entities are configured in various configurations, such as physical hardware and / or virtual components or virtual machines and / or virtual containers, where a container is different from a virtual machine because it virtualizes an instance of a game application running on a virtualized operating system. The processing entities may utilize and / or rely on servers and their underlying hardware on one or more servers (computing nodes) of the cloud gaming network or game cloud system 190, which may be located on one or more racks. Coordination, allocation, and management of the execution of their functions for the various processing entities is performed by a distributed synchronization layer. In such a way, the execution of their functions can be controlled by the distributed synchronization layer to generate media (e.g., video frames, audio, etc.) for the game application in response to controller inputs by the player. The distributed synchronization layer can efficiently execute their functions across the distributed processing entities (e.g., via load balancing) such that critical game engine components / functions are distributed and restructured for more efficient processing.

[0028] 2A is a diagram of rack assemblies 210 including computing nodes in a representative data center 200A of a gaming cloud system, according to one embodiment of the disclosure. For example, the data centers may be distributed around the world, such as in North America, Europe, and Japan.

[0029] The data center 200 includes a plurality of rack assemblies 220 (e.g., rack assemblies 220A-220N). Each rack assembly includes a corresponding network storage and a plurality of computing threads. For example, a representative rack assembly 220N includes a network storage 211 and a plurality of computing threads 230 (e.g., threads 230A-230N), as well as a rack controller 250 configured for internal and external network configuration of the components of the rack assembly 220N. Other rack assemblies may be similarly configured with or without modification. In particular, each computing thread includes one or more computing nodes that provide hardware resources (e.g., processors, CPUs, GPUs, etc.). For example, a computing thread 230N in the plurality of computing threads 230 of the rack assembly 220N is shown to include four computing nodes, although it is understood that a rack assembly may include one or more computing nodes. Each rack assembly is coupled to a cluster switch configured to provide communication with a management server configured for management of the corresponding data center. For example, the rack assembly 220N is coupled to a cluster switch 240N. The cluster switch also provides communication to external communication networks (eg, the Internet, etc.).

[0030] In particular, the cluster fabric (e.g., cluster switches, etc.) provides communication between one or more cluster rack assemblies, distributed storage 270, and a communication network. In addition, the cluster fabric / switch also provides data center support services such as management, logging, monitoring, event generation, boot management information tracking, etc. The cluster fabric / switch may provide communication to an external communication network via a router system and a communication network (e.g., the Internet). The cluster fabric / switch also provides communication to storage 270. A cluster of rack assemblies in a representative data center of a gaming cloud system may include one or more rack assemblies as a design choice. In one embodiment, the cluster includes 50 rack assemblies. In other embodiments, the cluster may include more or less than 50 rack assemblies.

[0031] In one network configuration, each rack assembly provides high-speed access to a corresponding network storage, such as within the rack assembly. In one embodiment, this high-speed access is provided via a PCIe fabric that provides direct access between the compute nodes and the corresponding network storage. In other embodiments, the high-speed access is provided via other network fabric topologies and / or networking protocols, including Ethernet, Infiniband, Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE), and the like. For example, in rack assembly 220N, the high-speed access is configured to provide a data path 201 between a particular compute node of a corresponding compute thread to the corresponding network storage (e.g., storage 211). In particular, the network fabric (e.g., PCIe fabric) can provide network storage bandwidth (e.g., access, etc.) of over 4 gigabytes per second (GB / s) per compute node (e.g., rack assembly) at non-volatile memory express (NVMe) latency. Additionally, control path 202 is configured to communicate control and / or management information between network storage 210 and each compute node.

[0032] In another network configuration (not shown), each rack assembly provides high-speed access between the compute nodes. For example, reconfiguration of the PCIe fabric within the rack assembly is performed to enable high-speed communication between the compute nodes within the rack assembly, such as when forming a supercomputer or when forming an artificial intelligence (AI) computer.

[0033] As illustrated, the cloud management controller 210 and / or corresponding rack controller (e.g., rack controller 220B) of the data center 200 communicates with the assigner 191 (shown in FIG. 1 ) to allocate resources to the client devices 110. In particular, the cloud management controller 210 and / or corresponding rack controller (e.g., rack controller 220B) of the data center 200 can cooperate with an instance of the gaming cloud system 190′ and, together with the first instance of the gaming cloud system 190 (e.g., in FIG. 1 ), allocate resources to the client devices 110. That is, the assigner 191 can cooperate with the cloud management controller 210 and / or corresponding rack controller to allocate users to resources. In one embodiment, the corresponding rack controller is primarily responsible for operationalizing the rack assembly, including configuring internal and external networks, powering on the computing threads and / or computing nodes located within the corresponding rack controller, etc. The corresponding rack controller communicates status information, such as the number of work threads available for allocation, the number of compute nodes available for allocation, etc., to the cloud management controller 210. This information may be communicated by the cloud management controller 210 and / or the corresponding rack controller to the assigner 191 so that the assigner can use the information to assign users to resources (e.g., compute nodes, etc.). In some embodiments, the corresponding rack controller and network storage pair are configured on the same server (e.g., network storage), while in other embodiments, the corresponding rack controller and network storage are configured on separate servers. In embodiments, the allocation is performed based on the required resources and bandwidth, and asset awareness, such as knowing that it exists in the data center. Thus, for illustrative purposes, an embodiment of the present disclosure is configured to assign a client device 110 to a particular compute node 232 of a corresponding compute thread 231 of a corresponding rack assembly 220B.

[0034] In one embodiment, the cloud management controller 210 is configured to manage remote boot control of computing resources, such as compute nodes of a corresponding computational thread of a corresponding streaking array in a corresponding rack assembly, in coordination with a corresponding rack controller. In another embodiment, the cloud management controller 210 is configured to directly manage remote boot control of computing resources, such as compute nodes of a thread server arranged in a streaming array of a rack assembly. That is, the cloud management controller 210 and / or the corresponding rack controller are configured to manage remote boot control of computing resources. For example, the cloud management controller 210 and / or corresponding rack controllers may manage boot files (e.g., BIOS firmware) used for start-up and / or boot operations of the computing nodes, which may be located at storage addresses 265A-N of the remote storage 260. Additionally, the cloud management controller 210 and / or corresponding rack controllers may control boot management information 270 for each computing resource in the data center, including boot images (e.g., operating system images) for the computing nodes in the data center. In that way, the computing resources (e.g., rack assemblies, computing threads, computing nodes, etc.) of the data center may be dynamically reconfigured depending on the intended use of these resources (e.g., gaming during peak user demand, secondary use during off hours (e.g., performing system maintenance, performing AI modeling, running applications by supercomputers, etc.).

[0035] The streaming rack assembly is configured around compute nodes to execute gaming applications, video games, and / or stream audio / video of game sessions to one or more clients. Additionally, within each rack assembly, game content may be stored on storage servers that provide network storage. The network storage is equipped with large amounts of storage and high-speed networking to serve many compute nodes. In particular, a storage protocol (e.g., a network file system, etc.) is implemented over a network fabric topology that is used to access the network storage. Data may be stored in the network storage using file storage, block storage, or object storage techniques, based in part on the underlying storage protocol being implemented. For example, a PCIe fabric storage protocol may access block storage data from the network storage.

[0036] 2B is a diagram of multiple rack assemblies 221 including multiple computing nodes in an exemplary data center 200B of a gaming cloud system, where each network storage is accessible by a corresponding array of computing nodes, according to one embodiment of the disclosure. Data center 200B is similar to data center 200A, where like-numbered components have similar functions. However, data center 200B has a different configuration of rack assemblies than the rack assemblies of data center 200A, where the network storage is accessed by a single streaming array of computing nodes, as described below.

[0037] Data center 200B includes multiple rack assemblies 221 (e.g., rack assemblies 221A through 221N). Each rack assembly includes one or more streaming arrays, each including a corresponding network storage and multiple computational threads. For example, representative rack assembly 221N includes streaming arrays 225A through 225N. In one embodiment, rack assembly 221N includes two streaming arrays, each including a network storage, a corresponding rack controller, and multiple computational threads. For example, the streaming array 225N includes multiple computation threads 235 that access the network storage 211N. In particular, each streaming array includes a corresponding rack controller and network storage. For example, in a representative rack assembly 221N, the streaming array 225A includes a rack controller 250A and a network storage 211A, and the streaming array 225N includes a rack controller 250N and a network storage 211N. In some embodiments, the corresponding rack controller and network storage pairs of the corresponding streaming arrays are configured on the same server (e.g., network storage), while in other embodiments, the corresponding rack controller and network storage are configured on separate servers. In yet other embodiments, a rack assembly includes multiple streaming arrays, each with a corresponding network storage, and a single rack controller of the rack assembly manages boot control (e.g., remote boot control, control boot management information, etc.) of computing resources across the streaming arrays. In yet other embodiments, a rack assembly includes multiple streaming arrays that access a single network storage, and a single rack controller of the rack assembly manages boot control (e.g., remote boot control, control boot management information, etc.) of computing resources across the streaming arrays. As shown, each computational thread includes one or more computational nodes that provide hardware resources (e.g., processors, CPUs, GPUs, etc.). For example, while computational thread 235X of streaming array 225N is shown to include four computational nodes, it is understood that a rack assembly can include one or more computational nodes.

[0038] Each rack assembly is coupled to a cluster switch configured to provide communication through a corresponding rack controller with a cloud management controller 210 configured for management of a corresponding data center, as described above. The corresponding rack controller provides internal management of the resources of the corresponding rack assembly, including handling the internal details of the resources in the corresponding rack assembly, including ensuring that those resources start up and remain operational by performing management of the compute nodes, compute threads, streaming arrays, etc. For example, rack assembly 221N is coupled to cluster switch 240N. The cluster switch also provides communication to other rack assemblies (e.g., via the corresponding cluster switch) and to an external communication network (e.g., the Internet, etc.).

[0039] In one network configuration, each streaming array of a corresponding rack assembly provides high-speed access to a corresponding network storage, as described above. In one embodiment, the high-speed access is provided via a PCIe fabric that provides direct access between the compute nodes and the corresponding network storage. In other embodiments, the high-speed access is provided via other network fabric topologies and / or networking protocols, including Ethernet, InfiniBand, remote RDMA over RoCE, etc. The compute nodes are capable of running gaming applications and streaming audio / video of the gaming sessions to one or more clients, and the corresponding network storage (e.g., storage server) holds the gaming applications, game data, and user data. For example, in the streaming array 225A of the rack assembly 221N, the high-speed access is configured to provide a data and control path 201A between a particular compute node of a corresponding compute thread and the corresponding network storage (e.g., storage 211A). The path 201N is also configured to communicate control and / or management information between the network storage 211N and each compute node of the streaming array 225N.

[0040] In another network configuration (not shown), each rack assembly provides high-speed access between the compute nodes. For example, reconfiguration of the PCIe fabric within the rack assembly is performed to enable high-speed communication between the compute nodes within the rack assembly, such as when forming a supercomputer or when forming an artificial intelligence (AI) computer.

[0041] As discussed above, the cloud management controller 210 of the data center 200B, in coordination with the rack controllers of the corresponding rack assemblies, communicates with the assigner 191 to allocate resources to the client devices 110 to support the gaming cloud systems 190′ and / or 190. In an embodiment, the allocation is performed based on knowledge of assets, such as knowing what resources and bandwidth are needed and that are present in the data center. Thus, for illustrative purposes, an embodiment of the present disclosure is configured to assign the client device 110 to a particular compute node 232B of a corresponding compute thread 231B of a corresponding streaming array of the rack assembly 221B.

[0042] FIG. 3 is a diagram of a compute sled 300 including multiple compute nodes arranged in a rack assembly configured for remote boot control of the compute nodes using a board management controller (e.g., via management control from a corresponding rack controller) and / or for network reconfiguration of the corresponding rack assembly at the compute node level, in accordance with one embodiment of the present disclosure.

[0043] Each computational thread 300 includes one or more computational nodes (e.g., nodes 1-4) disposed within a corresponding rack assembly. Although FIG. 3 illustrates a computational thread including four computational nodes, it is understood that any number of computational nodes may be provided for a computational thread including one or more computational nodes. Computational thread 300 may provide a hardware platform (e.g., a circuit board) that provides computational resources (e.g., via the computational nodes). For example, computational thread 300 illustrates multiple computational nodes (e.g., nodes 1-4) and supporting hardware that supports the operation of the computational nodes. Computational thread 300 may be implemented within any of the streaming arrays and / or rack assemblies previously described in FIGS. 2A-2B, or any other rack assembly, streaming array, and / or data center configuration.

[0044] Thread switch / management panel 315A integrates switching functions provided by PCIe switchboard 320A and thread management functions provided in part through board management controller (BMC) 350. For example, PCIe switchboard 320A provides communication with compute nodes via NT ports or the like. BMC 350 may also be directly connected to PCIe switchboard 320A (e.g., via a 1x channel or the like). As shown, panel 315A integrates multiple functions in one embodiment, although in other embodiments the functions may be implemented separately in independent controllers.

[0045] In one network configuration, the compute sled 300 provides the compute nodes with high-speed access to network storage using PCIe (e.g., Gen4) communication, according to one embodiment of the present disclosure. In another network configuration, the compute sled 300 is configured to provide high-speed communication between compute nodes of the compute sled within a rack assembly and with compute nodes between different rack assemblies. In one embodiment, the compute node includes multiple I / O interfaces. For example, the compute node may include an M.2 port and multiple lanes for PCIe Gen4 (bidirectional) via channels and / or connections 335. In particular, the compute sled 300 includes a PCIe switch board 320A that provides eight PCIe lanes to an array-level PCIe fabric, such as via a PCIe cable 305 (e.g., quad form factor, double density (QSFP-DD), etc.). The system can be expanded with additional devices using the PCIe (e.g., Gen4) interface (e.g., four lanes). In particular, the PCIe interface is used to connect to a PCIe fabric that includes a PCI Express switch 320A for high-speed storage.

[0046] Additionally, each compute node is configured for Ethernet connectivity 311 (e.g., Gigabit Ethernet) via Ethernet patch panel 310A, which is configured to connect Ethernet cables between the compute nodes (e.g., nodes 1-4) and a rack-level network switch (not shown).

[0047] Board management controller (BMC) 350 is also configured, in part, to manage one or more communication interfaces (e.g., PCIe, USB, UART, GPIO, I2C, I3C, etc.), each of which may be used to communicate with a computational node on computational thread 300 via a corresponding communication channel (e.g., 330). That is, BMC 350 is configured to manage multiple communication interfaces that provide communication to multiple computational nodes (e.g., computational nodes 1-4) on computational thread 300. For example, the corresponding computational node includes one or more universal asynchronous receiver-transmitter (UART) connections configured for transmitting and / or receiving serial data. In particular, there may be one or more UART ports, which are used for management purposes (e.g., connecting computational nodes to BMC 350). The ports may also be used for remote control operations such as "power on," "power off," and diagnostics. Additionally, another UART port may provide a serial console function.

[0048] Thread management of resources within computational thread 300 may be facilitated via one or more communication channels implementing a management interface utilized by a corresponding rack controller of a corresponding rack assembly. The management interface enables transmission of data packets to various different devices within computational thread 300, such as from a corresponding rack controller. The management interface may implement rack management (e.g., through the execution of software) as provided by the rack controller, and the rack management software may run on the rack controller or may be implemented via network storage. The rack management software executes to manage the operation of the rack via the management interface, including powering on / off system and resources of the compute sleds 300 (e.g., BMC 350, PCIe switch 320A, fans 390, power interposer board 340A, compute nodes 1-4, boot controllers on the compute nodes, etc.), performing firmware updates, obtaining thread status (e.g., temperature, voltage, fan speed, etc.), and providing access to the UART interfaces of the individual compute nodes and / or BMC 350 (each compute node and / or BMC may have a UART or other interfaces / ports including I2C, I3C, etc.).

[0049] In particular, management control may be provided from a corresponding rack controller via a management interface via an Ethernet connection 335 and / or via a PCIe cable 305 as a primary embodiment. For example, the management interface 327 may be implemented as separate wires (e.g., two separate wires for transmit and receive) within the PCIe cable 305 and may be used or further transmitted via an I2C, I3C, UART, or some other interface. In particular, in addition to an Ethernet connection, the management interface may be provided via a low-speed interface (e.g., implemented by the BMC 350) realized using either an I2C, I3, UART, or other communication interface to transmit control packets to devices within the computational sled 300. In that way, management control over the management interface may be provided via Ethernet (i.e., first option) and / or via PCIe (i.e., second option) via the management interface 327 and / or an I2C, I3, UART, or other communication interface.

[0050] In one embodiment, the management interface may be provided directly by or via the BMC 350, such as when receiving board management control signals from the PCIe cable 305 via the management interface 327, and delivered to the compute node through the communications interface 330 (e.g., I2C, I3C, UART, etc.). For example, one PCIe lane may be reserved for communication with the BMC 350 for board management control. As previously mentioned, the BMC 350 is configured to manage the communications interface for the purpose of providing remote booting of the compute node. In particular, the BMC 350 is configured to communicate with the corresponding computing node on the computing thread 300 for the purpose of performing remote boot control of the computing node. For example, no storage is required on the computing node to store a boot configuration file, since external hardware is used in the computing node's boot process, which is handled by communication between the BMC 350 and the corresponding computing node. This allows flexibility as to which software is used for the external boot of the computing node. That is, a computing node may be configured with one operating system that provides services corresponding to a first priority service (e.g., a primary service) and then reconfigured with a second operating system that provides services corresponding to a second priority service (e.g., a secondary service).

[0051] In another embodiment, the management interface may be provided by or through another helper chip, such as a complex programmable logic device (CPLD) 325 that manages communication (i.e., manages the transmission of data packets) over the management interface. Transmission through the CPLD 325 provides a robust backup mechanism for delivering board management control signals across the computational threads 300. For example, a backup mechanism may be implemented if Ethernet communication fails for any reason, or if the PCIe switch 320A fails, or if the BMC 350 crashes (e.g., its firmware becomes corrupted). Under such circumstances, the management interface 327 is configured to implement a backup mechanism for delivering board management control signals. In particular, the management interface 327 connects the first CPLD 325, which is also connected to the BMC 350 or directly to an I2C, I3, UART, or other communication interface (e.g., via channel 330). That is, the backup mechanism allows the BMC 350 to be bypassed so that board management control signals from the rack controller are routed through the CPLD 325 and delivered via channel 330. While the backup mechanism is highly robust and reliable (i.e., the CPLD 325 is not susceptible to failures because it is implemented in dedicated hardware), the associated performance may be limited in scope and / or speed. However, the backup mechanism provides for implementation of at least low-level operations (i.e., operations programmed via the CPLD 325), including power control, reset operations, and collection of some limited diagnostics.

[0052] BMC 350 may provide board or thread management functions. Additionally, in an embodiment, board management provided by BMC 350 includes control, monitoring, and management of the compute nodes, which is performed using Universal Asynchronous Receiver Transmitter (UART), I2C, I3C, or other signals that deliver serial data (e.g., power on / off, diagnostics, and logging information) for each compute node over connection 330. In other embodiments, board management may be provided over other interfaces, such as Ethernet (e.g., communicating directly with a rack controller). Additionally, BMC 350 may be configured to provide electromagnetic compatibility (EMC) control for controlling electromagnetic energy and debugging delays using UART signals. BMC 350 may be configured to provide control status information to management panel 330A, such as via control status light emitting diodes (LEDs), which are further configured to display status using LEDs and buttons. BMC 350 is configured to monitor temperature and voltage. BMC 350 is also configured to manage fans configured for cooling. BMC 350 is also configured to manage Ethernet connections for board management.

[0053] Compute sleds 300 include power interposer boards 340A configured to provide power to corresponding compute sleds via one or more bus bar connections. That is, BMC 350 may be configured to enable power control / delivery to compute nodes via rack management bus 360 using general purpose input / output (GPIO), and / or I2C, and / or some other communication interface to power interposer 340A (e.g., from a 12 volt or higher bus bar to each compute node). In particular, power management functions are used to manage and / or monitor power delivery to each compute node via power interposer board 340A via connections. For example, rack management bus 360 may be configured to provide thread management control signals. Power monitoring (e.g., sensors connected via I2C channels to measure current, voltage, and other conditions through the power connections to the compute nodes) may be performed and communicated via rack management bus 360 to BMC 350, etc. The BMC 350, in coordination with a corresponding rack controller, may decide to power off individual components (e.g., compute nodes, etc.) or entire compute threads under certain conditions, including, for example, when it detects that the voltage is too high, when it detects that the temperature is too high, when it detects that the fan 390 on the compute thread 300 is inoperable, and when it detects other adverse conditions. In particular, the BMC 350 provides management information (e.g., power conditions, etc.) to the corresponding rack controller and provides thread management instructions. The BMC 350 can operate independently of the rack controller under certain extreme conditions, but generally works in coordination with the rack controller for thread management. For example, thread management is performed externally (e.g., via the rack controller) and communicated internally via the rack management bus 360. This includes disabling threads under various conditions, such as during scheduled maintenance of a rack assembly, when a power outage occurs in the corresponding data center due to a fire in the data center, or when a backup power source is implemented. Each compute node also includes a power input connector (eg, 12 volts for designed power consumption) connected to the power interposer 340A via a corresponding bus bar 370 connection.

[0054] 4 is a flow diagram 400 illustrating steps in a method for performing a remote boot and / or remote start-up of a computing resource, according to one embodiment of the disclosure. For example, flow diagram 400 may be implemented to perform a remote boot of a computing node of a computing thread that includes one or more computing nodes using a board management controller (BMC) of the computing thread and a corresponding boot controller of the computing node (e.g., boot controller 320-1 of computing node 1, boot controller 320-2 of computing node 2, boot controller 320-3 of computing node 3, or boot controller 320-4 of computing node 4 in computing thread 300 of FIG. 3). Notably, in one embodiment, no storage is required to boot the OS since external hardware is used in the compute node boot process, allowing flexibility as to which software is used for the external boot. In other embodiments, there may be storage on the compute node partially configured to boot the OS. Although flow diagram 400 is described in the context of remote booting compute nodes of compute threads configured in a streaming array of rack assemblies located in a cluster of rack assemblies in a data center, such as the data centers of FIGS. 2A-2B, it will be appreciated that the operations described in flow diagram 400 may be generally applied to remote booting of any computing resource using external hardware for storing boot files (e.g., BIOS, operating system configuration files, etc.).

[0055] Traditionally, a system startup may execute one or more processes to initialize hardware and load an operating system used by the computer system to execute applications. In general, a computer system typically loads and executes basic input / output system (BIOS) firmware from a read-only non-volatile memory (e.g., Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, etc.) located on the computer system to initialize the computer system's hardware upon system startup. Once the hardware has been initialized, the BIOS firmware is used to boot an operating system from a local storage device, such as a hard drive or solid-state drive (SSD), such as by executing one or more boot loader programs as directed by the BIOS firmware. Because the BIOS firmware used to load the operating system is located in read-only memory, modifying the operating system requires modifying the BIOS firmware on the local system and / or changing settings that may be stored in local volatile memory (which may be battery powered), which may be difficult to accomplish in a time sufficient to meet the real-time needs of a client and / or user.

[0056] The embodiments of the present disclosure provide a remote boot of a computing node of a computing thread (e.g., a game console) that has no internal storage and no BIOS used to boot on the platform (computing node). In some embodiments, there may be optional minimal storage for the boot controller (located inside the boot controller or outside the boot controller), while in other embodiments, there is no storage for the boot controller. That is, the remote boot includes swapping the BIOS implementation and correspondingly swapping the OS implementation. For example, the computing node may be configured to play a game using a backend streaming server of a cloud gaming system, and the computing node is disposed on a computing thread of a streaming array that includes multiple computing threads. The streaming array is disposed in a rack assembly of a data center that includes multiple clustered rack assemblies, and each rack assembly includes one or more network storages and one or more streaming arrays. The use of external hardware in the boot process eliminates the need for storage such as non-volatile memory to store BIOS firmware and / or other boot configuration files (e.g., less hardware reduces the cost of each compute node). The compute node may be configured with volatile memory such as random access memory (RAM) that is used during operation of the compute node (e.g., to run an operating system, applications, etc.). In this manner, remote booting of a compute node provides flexibility as to which software (e.g., selection among multiple BIOS firmwares) to boot externally, thereby providing flexibility in loading a desired operating system to be used to run an application on the compute node. For example, if a compute node is configured for gaming, an operating system appropriate for the game is loaded to run the gaming application on the compute node (e.g., may not use a BIOS). On the other hand, a computing node may be configured for non-gaming services that require a different operating system. In that case, the computing node may be configured to load a new operating system using a remote boot process to run other applications that provide those non-gaming services on the computing node. The required BIOS firmware required for gaming and non-gaming use cases is significantly different. Non-gaming use cases transform the computing node and / or system into something closer to a personal computer (PC) running a typical or standard OS. In this case, the BIOS may be stored locally in non-volatile memory (e.g., SSD, etc.). On the other hand, gaming use cases are more specialized and do not necessarily require traditional BIOS firmware. For example, some functions associated with BIOS firmware (e.g., functions typically performed by a BIOS to implement an OS on a PC) are not necessary for gaming use cases, and BIOS functions for implementing a gaming console may be performed by the gaming OS itself during loading without the use of a dedicated BIOS firmware.

[0057] In particular, at 410, the method includes receiving, at a board management controller of the compute thread (e.g., BMC 350 of compute thread 300 of FIG. 3 ), a startup configuration instruction (from the cloud management controller) for booting the compute node with an operating system. The instruction may be delivered to the BMC by the cloud management controller via a corresponding rack controller. The cloud management controller may be configured to manage remote booting of the compute nodes in the data center in coordination with the corresponding rack controller, and to collect information related to the booting of the compute nodes, such as storing a boot image for each compute node. More specifically, the BMC of the corresponding compute thread is configured to manage multiple communication interfaces providing communication with the multiple compute nodes. For example, the communication interfaces may include an Inter-Integrated Circuit (I2C) bus interface (e.g., providing communication using PCI, PCIe, etc.), I3C, or PCI, or PCIe, or UART, or USB, etc.

[0058] In particular, the BMC communicates with a boot controller in the compute node using one or more of these communication interfaces. In one embodiment, the BMC may be configured on a system-on-chip (SoC), which includes a central processing unit (CPU) and other components. The BMC executes a corresponding OS and / or firmware that may be loaded from a non-volatile memory (e.g., flash, ROM, etc.) located on the integrated circuit of the SoC. The OS of the BMC requires management and / or updating. While the BMC may provide management and update functionality, management of the BMC may be provided through a side channel, such as using the management interface described above in connection with FIG. 3. Thus, the management interface may be used to manage and update the OS and / or firmware, and may also perform other operations on the BMC described above, including performing power on / off, power reset, etc. Also, the boot controller on a compute node may require firmware to function, which may be loaded via various methods depending on the hardware design. For example, in one embodiment, the boot controller may have its firmware integrated into a small ROM located on the motherboard of the corresponding compute node. In another embodiment, the boot controller firmware may be loaded remotely via the BMC on the corresponding compute thread, which pushes the boot controller firmware through a low-level interface via channel 330 (e.g., I2C, UART, serial peripheral interface (SPI), etc.).

[0059] The boot controller may be configured to load the BIOS firmware onto the main CPU of the compute node (e.g., the RAM or system memory used by the CPU) according to the boot instructions. More specifically, in an embodiment of the present disclosure, the BMC is configured to provide instructions to the boot controller to load the BIOS and operating system from one or more remote locations, where the instructions may be provided via a PCIe, USB, UART, or another interface, and the BIOS firmware may be accessed via a PCIe fabric. In addition to triggering the boot process, the BMC may perform other operations that allow it to return information about the motherboard and / or boot controller of the compute node. For example, the board information may include board type, board version, serial number. The board information may also include the CPU information of the compute node, including model number, serial number, hardware encryption key, and other information. This board information may be used to verify compatibility between the BIOS or operating systems booting on the corresponding compute node. In another embodiment, the BIOS or operating system may be locked (e.g., by encryption) to a particular compute node system.

[0060] In particular, at 420, the method includes transmitting a boot instruction from the BMC to a boot controller of the compute node via the selected communication interface to execute a BIOS firmware stored remotely from (or external to) the compute node. That is, the BIOS firmware is stored on external hardware (i.e., external to the compute node). For example, the BIOS firmware may be stored in a memory of the BMC (memory on a BMC chip), in remote storage of a corresponding rack assembly, or in storage external to the corresponding rack assembly (e.g., remote storage accessible by one or more rack assemblies of a data center, such as storage addresses 260A-N in FIGS. 2A-2B).

[0061] In one embodiment, a pull method is implemented to access a BIOS firmware located outside the compute node. In particular, the pull method includes receiving, at the BMC, a request from the compute node to access a storage address that stores the BIOS firmware. That is, the storage address of the BIOS firmware is known to the corresponding rack controller, and further, the storage address is included in the startup configuration instructions provided by the rack controller to the BMC, and the storage address is included in the boot instructions delivered from the BMC to the compute node. The pull method further includes facilitating the compute node to access the storage address to obtain the BIOS firmware. That is, the BMC forwards the request over an appropriate network to provide access and delivery of the BIOS firmware to the compute node. While the corresponding rack controller knows the internal details required for remote boot control of the corresponding compute node, the upper cloud management controller may provide high-level instructions, such as moving threads and / or rack assemblies from one operating mode to another. In this way, certain compute node management may be performed at the rack assembly level.

[0062] In another embodiment, a push method is implemented to access BIOS firmware located outside the compute node. In particular, the push method includes the BMC accessing the BIOS firmware at a storage address outside the compute node. That is, the storage address of the BIOS firmware is known to a corresponding rack controller, and the storage address is included in a boot configuration instruction provided to the BMC by the rack controller. In that way, the BIOS firmware is accessible by the BMC from the storage address. The push method further includes transmitting the BIOS firmware from the BMC to the compute node in association with a boot instruction for executing the BIOS firmware by the compute node. For example, the BIOS firmware may be included in the boot instruction. In that way, the compute node does not need to take an additional step to access the BIOS firmware.

[0063] In one embodiment, the communication interface is selected based on the operating systems initialized and loaded through execution of the BIOS firmware. For example, each operating system may be associated with a corresponding communication interface. Control signals of each operating system are delivered to the compute node via the corresponding communication interface. As an example, control signals of a first operating system are delivered via a first communication interface (I2C), control signals of a second operating system are delivered via a second communication interface (e.g., UART), control signals of a third operating system are delivered via a third communication interface (e.g., USB), etc. In this way, boot instructions delivered from the BMC to the boot controller of the compute node based on the operating system are delivered via a corresponding communication interface managed by the BMC for communicating control signals for that operating system. Thus, the BMC may be configured as a switch that delivers control signals for each of a plurality of operating systems loadable by the compute node via a corresponding communication interface selected from a plurality of communication interfaces managed by the BMC. That is, the BMC sled switch board provides a communication interface to the compute node, such as PCI, PCIe, I2C, UART, USB, etc.

[0064] At 430, the method includes executing the BIOS firmware on the compute node (CPU) to initiate loading of an operating system to be executed by the compute node. As previously discussed, the execution of the BIOS firmware (e.g., a first stage boot loader) may include execution of other associated firmware and / or software or programs (e.g., second stage boot loader(s)) to complete loading of the operating system for use by the compute node. That is, in some cases the BIOS firmware may be complete and may not require any additional firmware and / or software or programs to complete loading of the operating system, and in other cases the BIOS firmware may be used to call and / or execute other firmware and / or software or programs.

[0065] For purposes of explanation only, execution of the BIOS firmware may include initializing system hardware and loading an operating system. In particular, the BIOS firmware may perform a power-on self-test (POST) process to initialize system hardware (e.g., video cards and other hardware devices), may perform memory tests to configure memory and drive parameters, may configure plug-and-play devices, and may identify any boot devices for loading a subsequent boot loader (e.g., second stage, etc.) that is executed to load an operating system (e.g., an operating system configuration file).

[0066] The method further includes loading at least a portion of the operating system from a storage address into a system memory (e.g., RAM) of the compute node, the storage address including one or more operating system configuration files. In one embodiment, the entire operating system may be loaded into the system memory. However, some operating systems may be very large and it may be more efficient to load a portion of the operating system into the system memory so that the remaining portion can be used and / or loaded as needed. For example, a first portion of the operating system may be loaded into the system memory of the compute node and executed, and a second portion of the operating system may be stored in corresponding network storage such that it can be accessed by the compute node (e.g., via a PCIe fabric) during execution of the operating system by the compute node. For example, data and / or files of the second portion may be accessed and loaded into the system memory as needed, or may be accessed remotely during execution of the operating system.

[0067] In another embodiment, the cloud management controller, in coordination with a corresponding rack controller, is configured to manage one or more images of the operating system running on the computing nodes. That is, the cloud management controller and / or the corresponding rack controller of the data center are configured to manage the software images (e.g., operating system images) of each computing node. In that way, the cloud management controller and / or the corresponding rack controller manages the operating systems running on the various computing nodes and rack assemblies available in the data center. For example, the management information may be stored in storage 260 of FIG. 2A or FIG. 2B.

[0068] In one embodiment, during a switch of an operating system of a compute node, a job or jobs executing on the compute node prior to the switch may be paused (or paused) and / or transferred for continued execution. For example, a job may be executed by a compute node executing an application using a first operating system that is different from the new operating system loaded via remote boot. To illustrate, the job may relate to a gaming application, and the gaming application may be paused to be started at a later time or transferred to another compute node for seamless transfer and execution. For example, the state (e.g., configuration) of a compute node during execution of a job is captured and stored. The state is transferred to another compute node, such as on the same compute thread, or on the same streaming array, or on the same rack assembly, or on a compute node in a different rack assembly in the same data center, or on a compute node in a different data center. The state is initialized on the other compute node, and then the application is executed on the new compute node using the transferred state and the job is resumed. In one implementation, a paused job is paused at a selected pause or suspend point that is not a predefined pause point, such as the end of a level in a gaming application.

[0069] Alternative jobs (e.g., those enabled via remote boot) running on the compute node during the dark time utilization of the compute node may be paused and / or transferred to other datacenters to continue running, such as to a less utilized datacenter or to a less utilized portion of the same datacenter. For example, these alternative jobs may include machine learning workloads, or some other type of CPU / GPU workload, or video encoding workload, etc. In various implementations, the decision to pause and / or transfer these alternative jobs may depend on what business model is implemented or selected for these alternative workloads. Some cost-effective jobs purchased by a customer may be performed by purchasing a certain amount of computation time (i.e., not including any transfer) and may be executed during different dark time periods. Other customers may pay for a certain number of jobs to be completed during a period (e.g., dark time period, period beyond dark time period) and may require pausing and transferring from one resource to another. In another scenario, jobs may be paused without transferring if a higher-priced job requires processing, and so on. In this manner, embodiments of the present disclosure support pausing and transferring jobs under various scenarios.

[0070] A major difference between gaming use cases and dark time usage cases (i.e., running replacement jobs) is scale. In particular, gaming workloads typically involve a single user occupying a single compute node, with multiple compute nodes supporting multiple users in a one-to-one relationship. In a dark time usage use case, the single user may be, for example, an organization, potentially requiring the use of many server racks or rack assemblies, if not an entire data center. For security reasons, resources for organization users in a dark time usage use case are allocated on a rack assembly basis (e.g., rack controllers and network storage) so that resources allocated for dark time usage for that organization are isolated from other customers. This may include placing those resources on a different network with different firewall rules, etc., requiring reconfiguration of those resources. This may also include providing resources with additional network access, e.g., access to a different computer network of the customer or requesting organization, or another organization, to access data.

[0071] For purely illustrative purposes, the paused job may be associated with a game, or a video game or game application. In particular, an ongoing game may be paused at any point in the game, and the game resumed at the same point in the game at some future point (e.g., immediately after transfer or later). The game state of the game and configuration parameters of the computational nodes during which the game is running are captured and stored. That is, the paused game state is saved with enough data to reconstruct the game state when the game is resumed. During such pause of the game, the game state is collected and saved in storage, which avoids the need for the cloud gaming system to store the state in active memory or registers of the hardware. This frees up the system for other gameplay or allows the compute nodes to provide other services (e.g., dark time utilization of compute nodes), and allows gameplay to be resumed at any time, forming any remote compute node (e.g., client). When the game is resumed, the game state is loaded onto a new compute node (e.g., the same or a different compute node) that is tasked with resuming the game. Loading the game state involves generating the game state from multiple saved files and data structures, and the reconstructed game state can leave the compute node in the same, or nearly the same, state as when the game was paused.

[0072] For example, a cloud gaming system may include a cloud management controller, a rack controller, storage, and a number of computing nodes managed by the cloud gaming system and coupled via a network. Each computing node configured as a game console may include a hardware layer, an operating system layer, and an application layer. The operating system layer is configured to interact with the hardware layer, the operating system layer including a state manager. The application layer is configured to instruct at least a portion of the operating system and a portion of the hardware layer. The application layer includes a game, and the state manager is configured to capture game state data of the computing node and store the captured game state data when the game is paused. The state manager is also configured to apply the game state data to the same or a different computing node to resume the game at the point where the game was paused.

[0073] 5 is a flow diagram 500 illustrating steps in a method for reconfiguring a rack assembly of a data center, according to one embodiment of the disclosure. The flow diagram 500 may be implemented within the data storage system illustrated in FIGS. 2A-2B. In particular, the flow diagram may be implemented to provide for reconfiguration of a rack assembly at any suitable time.

[0074] For example, reconfiguration of rack assemblies may be performed to enable utilization of computing resources during dark times of a cloud gaming system configured primarily for gaming. That is, in a particular geographic area served by a regional data center, there are periods of low or inactivity (i.e., dark times) during which computing resources are underutilized, such as during work or school hours, large social gatherings (e.g., local football games), or late nights or early mornings (e.g., sleep hours) when users are not playing games on the cloud gaming system. Embodiments of the present disclosure provide for utilization of computing resources during those dark times or periods. As such, monetization of computing resources during those dark times is important for cost reasons (i.e., to improve the net profit of each computing resource within a 24-hour period).

[0075] Thus, one or more rack assemblies of a data center may be repurposed to process different workloads other than gaming during dark times. In general, it is more efficient to convert all computing nodes of the corresponding rack assembly to perform similar tasks, e.g., tasks performed during dark times, and a network reconfiguration of the rack assembly may be required to accomplish those tasks. Since all computing nodes may be newly configured to process new tasks performed during dark times, the reconfiguration of the rack assembly does not adversely affect any computing nodes that may be performing tasks (e.g., gaming) under the previous network configuration of that rack assembly. That is, the computing nodes processing games may be transferred to other rack assemblies in the same data center or a different data center, as described above.

[0076] At 510, the method includes sending a reconfiguration message from the cloud management controller to a rack controller of the rack assembly to reconfigure the rack assembly from the first configuration to the second configuration. Other resources in the data center may also be reconfigured, including resources outside the cloud gaming rack assembly. In particular, some use cases may require network / internet connectivity to other networks of third-party cloud hosting companies (e.g., AWS Cloud Computing), or other networks. This connectivity may be established as part of the network configuration. In other cases, reconfiguration of resources in the data center may be performed for security purposes to isolate harmful workloads (e.g., from hackers, or external companies requesting services, or untrusted customers) and / or prevent them from adversely affecting in any way the overall functionality and integrity of the data center (e.g., primarily providing cloud gaming services). As an example, the reconfiguration may be utilized to combat a distributed denial of service (DDOS) or hack of a portion of one or more data centers by an external party or an untrusted customer. In a first configuration, the rack assembly is configured to facilitate a first priority service implemented by a first plurality of applications (e.g., gaming). In a second configuration, the rack assembly is configured to facilitate a second priority service implemented by a second plurality of applications (e.g., performing artificial intelligence or AI services). The second priority service (e.g., AI) may have a lower priority than the first priority service (gaming), such that the rack assembly primarily supports gaming, but when demand for gaming decreases (i.e., dark times), the rack assembly may be configured to have more available computing resources throughout the entire 24-hour period by handling the second priority service.

[0077] In one embodiment, the rack assembly includes one or more network storages and one or more streaming arrays, as described above in Figures 2A-2B, each of which includes one or more computing threads, each of which includes one or more computing nodes.

[0078] A rack assembly may be sent a reconfiguration message when it is deemed appropriate to switch to a different configuration. For example, reconfiguration of one or more rack assemblies may occur at a particular defined time (e.g., after 2:00 a.m., indicating the start of dark time) or may occur in anticipation of when dark time will occur based on projected usage of computing resources in the data center. In one embodiment, dark time is measured through metrics and the cloud management controller is configured to detect a decrease in demand for a first priority service supported by the data center including the plurality of rack assemblies. For example, each of the plurality of rack assemblies is configured in a first configuration that facilitates a first priority service (e.g., gaming).

[0079] Generally, the switch to a different configuration of the rack assembly may occur during dark times. However, there may be periods outside of dark times when it may be beneficial to perform a reconfiguration of the rack assembly. For example, if a second priority service (e.g., non-gaming workload) generates more profit than a first priority service (e.g., gaming), a priority inversion may occur, in which case the second priority service is prioritized over the first priority service (e.g., gaming). That is, the data center may lower the priority of the first priority service (e.g., gaming).

[0080] In one embodiment, the cloud management controller manages the network configuration of the multiple rack assemblies. For example, the cloud management controller may maintain an inventory of the number of rack assemblies and the number of compute nodes configured or available for gaming (e.g., first priority service), such as within a particular geographic region. As such, when gaming demand decreases over a particular period of time, indicating the onset of dark time, the cloud management controller may reconfigure one or more rack assemblies to support second priority service over gaming, while maintaining a sufficient number of rack assemblies to support current and projected gaming demand by users of the data center.

[0081] At 520, the method includes configuring the rack assembly in a second configuration such that the rack assembly can run applications supporting a second priority service (e.g., AI), thereby utilizing compute nodes that would otherwise be idle.

[0082] In one embodiment, reconfiguring the rack assembly may include rebooting the compute nodes of the rack assembly with a different operating system. In general, it may be beneficial for the compute nodes of the rack assembly to run the same operating system and even the same version of the same operating system. For example, the different operating system may be configured to support second priority services (e.g., running applications during dark times). Rebooting the compute nodes may include the operations previously described in FIG. 4 and may include receiving a startup configuration instruction at the BMC to boot the compute nodes with the new operating system. The compute nodes are arranged on a sled that includes one or more compute nodes, and the BMC is configured to manage multiple communication interfaces that provide communication to the compute nodes. The rebooting process may include sending a boot instruction from the BMC via the communication interface to a boot controller of the compute node to execute a basic input / output system (BIOS) firmware. The rebooting process may include executing the BIOS firmware on the compute node to load an operating system to be executed by the compute node.

[0083] In one embodiment, reconfiguring a rack assembly, including rebooting the compute nodes of the rack assembly to a different operating system, may include loading new operating system software or other software components into the rack assembly's corresponding network storage, or into the cloud management controller or other storage accessible by the cloud management controller.

[0084] In one embodiment, reconfiguring a rack assembly may include changing a network configuration of the rack assembly, i.e., a reconfiguration of the network configuration of the rack assembly is required to provide communication between the rack assembly and a remote rack assembly, the network configuration defining a communication path between the rack assembly and another rack assembly. For example, external access of a compute node using a virtual local area network may need to be reconfigured and correspondingly, a firewall may need to be reconfigured to provide appropriate access to other remote rack assemblies (e.g., compute nodes located on the remote rack assemblies). As mentioned above, resources within the data center may be reconfigured during network configuration to include access to resources outside the cloud gaming rack assembly, such as providing network / internet connectivity to other networks of third-party cloud hosting companies (e.g., AWS Cloud Computing) or other networks. Resource reconfiguration may also be performed for security purposes, such as to isolate harmful workloads (e.g., DDOS, hacking workloads, or untrusted external companies requesting services, or untrusted customers) and / or prevent them from adversely affecting in any way the overall functionality and integrity of the data center (e.g., primarily providing cloud gaming services).

[0085] In one embodiment, reconfiguring the rack assembly may include modifying the boot storage architecture, for example, a network reconfiguration may provide faster access to boot or operating system files for use during boot and / or execution of the corresponding operating system.

[0086] In one embodiment, reconfiguring the rack assembly may include reconfiguring a PCIe (or equivalent) fabric to enable high speed communication between computing nodes to form a "supercomputer" or AI computer where high speed communication between the computing nodes is required. For example, the rack assembly may be initially configured for direct access between multiple computing nodes and at least one network storage of the rack assembly, such as for in-game implementation. However, during dark times, the rack assembly may be reconfigured in a second configuration. For example, the second configuration may configure a PCIe fabric for direct communication between multiple compute nodes in a rack assembly instead of providing direct access to network storage (e.g., in the first configuration). Thus, in the second configuration, the PCIe fabric facilitates direct communication between a first compute node and a second compute node in the rack assembly. Additionally, rack reconfiguration may be performed to establish a special network configuration between the rack assemblies. Typically, cloud gaming rack assemblies operate independently of each other, but in some use cases, two or more rack assemblies may be interconnected and operated in conjunction through reconfiguration. Typically, without reconfiguration, network settings such as firewalls and / or cluster level switches prevent such cooperation.

[0087] FIG. 6 illustrates components of an exemplary device 600 that can be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 6 illustrates an exemplary hardware system suitable for providing remote boot control of computational nodes, such as computational nodes of a streaming array of computational threads in a rack assembly, and network reconfiguration of a rack assembly in a data center, according to embodiments of the present disclosure. The block diagram illustrates device 600 that can incorporate or is any of a personal computer, a server computer, a gaming console, a mobile device, or other digital device, each of which is suitable for practicing embodiments of the present invention. Device 600 includes a central processing unit (CPU) 602 for executing software applications and optionally an operating system. CPU 602 may be comprised of one or more homogeneous or heterogeneous processing cores.

[0088] According to various embodiments, CPU 602 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture specifically adapted for highly parallel, computationally intensive applications (e.g., media and interactive entertainment applications) for applications configured to perform graphics processing during game execution.

[0089] Memory 604 stores applications and data used by CPU 602 and GPU 616. Storage 606 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 608 communicate user input from one or more users to device 600 and may include, for example, a keyboard, a mouse, a joystick, a touchpad, a touchscreen, a still or video recorder / camera, and / or a microphone. The network interface 609 enables the device 600 to communicate with other computer systems over electronic communications networks, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 612 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 602, memory 604, and / or storage 606. The components of the device 600, including the CPU 602, the graphics subsystem including the GPU 616, the memory 604, the data storage 606, the user input devices 608, the network interface 609, and the audio processor 612, are connected via one or more data buses 622.

[0090] A graphics subsystem 614 is further connected to the data bus 622 and the components of the device 600. The graphics subsystem 614 includes at least one graphics processing unit (GPU) 616 and a graphics memory 618. The graphics memory 618 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 618 may be integrated in the same device as the GPU 616, connected as a separate device from the GPU 616, and / or incorporated within the memory 604. The pixel data may be provided to the graphics memory 618 directly from the CPU 602. Alternatively, the CPU 602 provides data and / or instructions defining a desired output image to the GPU 616, from which the GPU 616 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 604 and / or the graphics memory 618. In an embodiment, GPU 616 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 616 may further include one or more programmable execution units capable of executing shader programs.

[0091] Graphics subsystem 614 periodically outputs pixel data for images from graphics memory 618 to be displayed on display device 610 or projected by a projection system (not shown). Display device 610 can be any device capable of displaying visual information in response to signals from device 600, including CRT, LCD, plasma, and OLED displays. Device 600 can provide analog or digital signals to display device 610, for example.

[0092] In other embodiments, the graphics subsystem 614 includes multiple GPU devices that are coupled to perform graphics processing for a single application running on a corresponding CPU. For example, multiple GPUs may perform multi-GPU rendering of geometry for an application by pre-testing geometry against a screen region (which may be interleaved) before rendering objects for an image frame. In another example, multiple GPUs may perform an interleaved style of frame rendering, where GPU1 renders a first frame, GPU2 renders a second frame, and so on for successive frame periods until the last GPU is reached. The first GPU then renders the next video frame (e.g., if there are only two GPUs, GPU1 renders the third frame). That is, the GPUs perform rotation operations as they render frames. Rendering operations may overlap, such that GPU2 may begin rendering the second frame before GPU1 finishes rendering the first frame. In another implementation, rendering and / or different shader operations in the graphics pipeline may be assigned to multiple GPU devices. A master GPU does the main rendering and compositing. For example, in a group including three GPUs, master GPU1 may perform main rendering (e.g., first shader operations) and compositing of outputs from slave GPU2 and slave GPU3, slave GPU2 may perform second shader (e.g., fluid effects such as rivers) operations, slave GPU3 may perform third shader (e.g., particulate smoke) operations, and master GPU1 may composite the results from each of GPU1, GPU2, and GPU3. In that way, various GPUs may be assigned to perform various shader operations (e.g., flag flapping, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs may be assigned to different objects and / or parts of a scene corresponding to a video frame. In the above embodiments and implementations, these operations may occur in the same frame cycle (concurrently in parallel) or in different frame cycles (sequentially in parallel).

[0093] Accordingly, this disclosure describes methods and systems configured to provide remote boot control of computing nodes, such as computing nodes of a computing thread of a streaming array of a rack assembly, and network reconfiguration of a rack assembly in a data center.

[0094] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations that use various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various implementations that are possible by combining various elements to define more implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0095] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0096] With the foregoing embodiments in mind, it should be understood that embodiments of the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. Embodiments of the present disclosure also relate to devices or apparatus for performing these operations. The apparatus can be specially constructed for the required purpose or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0097] The present disclosure can also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data, which can later be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible media distributed across network-connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0098] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0099] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments of the present disclosure are not to be limited to the details provided herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. 1. A method for performing a system boot, comprising: a board management controller (BMC) configured to receive startup configuration instructions for booting a compute node with an operating system, the compute node being placed into a thread that includes a plurality of compute nodes, the BMC configured to manage a plurality of communication interfaces that provide communication to the plurality of compute nodes; transmitting a boot command from the BMC to a boot controller of the computing node via a communications interface to execute a basic input / output system (BIOS) firmware stored external to the computing node; executing the BIOS firmware on the computing node to initiate loading of the operating system to be executed by the computing node.

2. receiving, from the computing node, a request to access a storage address storing the BIOS firmware at the BMC; facilitating access by the computing node to the storage address to obtain the BIOS firmware; the storage address is included in the boot instructions from the BMC; The method of claim 1.

3. The sending of the boot command comprises: accessing, by the BMC, the BIOS firmware at a storage address, the storage address being included in the boot configuration instructions; transmitting the BIOS firmware from the BMC to the computing node in association with the boot instructions for the execution of the BIOS firmware by the computing node; The method of claim 1.

4. The execution of the BIOS firmware includes: loading at least a portion of the operating system from a storage address into a system memory of the compute node; The method of claim 1.

5. Loading at least a portion of the operating system from a storage address into a system memory of the compute node, comprising: loading a first portion of the operating system into the system memory of the compute node for execution; loading a second portion of the operating system into network storage for access by the compute node over a PCIe fabric during the execution of the operating system by the compute node. The method according to claim 4.

6. Selecting the communication interface based on the operating system; the BMC is configured as a switch to deliver a control signal to each of a plurality of operating systems via a corresponding communication interface selected from the plurality of communication interfaces; said operating system control signals being delivered to said compute node via said communications interface; The method of claim 1.

7. and managing, at a cloud management controller at a data center, one or more images of the operating system on the compute nodes. The method of claim 1.

8. pausing a job being executed by the compute node while the compute node is executing an application using a first operating system that is different from the operating system that is loaded for execution; capturing a state of the compute node and storing the state; transferring the state of the compute node to another compute node; running the application on the other compute node to resume the job; the job is paused at a selected pause point which is not a predetermined pause point; The method of claim 1.

9. the computational thread is one of a plurality of computational threads of a streaming array; Each rack assembly of the plurality of rack assemblies includes one or more network storage and one or more streaming arrays; Each streaming array includes one or more computational threads; Each computational thread includes one or more computational nodes. The method of claim 1.

10. sending a reconfiguration message from a cloud management controller to a rack controller of the rack assembly to reconfigure the rack assembly from a first configuration to a second configuration, the rack assembly being configured to facilitate first priority services implemented by a first plurality of applications, the second configuration being configured to facilitate second priority services implemented by a second plurality of applications, the second priority services having a lower priority than the first priority services; configuring the rack assembly in the second configuration; The method, wherein the rack assembly includes one or more network storage and one or more streaming arrays, each streaming array including one or more computing threads, each computing thread including one or more computing nodes.

11. detecting, at the cloud management controller, a decrease in demand for the first priority service supported by a data center including a plurality of rack assemblies; each of the plurality of rack assemblies is configured in the first configuration to facilitate servicing the first priority; the cloud management controller managing configuration of the plurality of rack assemblies; The method of claim 10.

12. The configuration of the rack assembly in the second configuration includes: reconfiguring the PCIe fabric in the second configuration for direct communication between a plurality of compute nodes in the rack assembly, such that the PCIe fabric facilitates direct communication between a first compute node and a second compute node in the rack assembly; In the first configuration, the PCIe fabric is configured for direct access between the plurality of compute nodes and at least one network storage of the rack assembly. The method of claim 10.

13. The configuration of the rack assembly in the second configuration includes: rebooting each of the plurality of compute nodes of the rack assembly to execute an operating system configured to support execution of the second priority service; Rebooting the compute node a board management controller (BMC) configured to receive startup configuration instructions for booting the compute node with the operating system, the compute node being placed into a thread that includes a plurality of compute nodes, the BMC configured to manage a plurality of communication interfaces that provide communication to the plurality of compute nodes; sending a boot instruction from the BMC to a boot controller of the computing node via a communications interface to execute a basic input / output system (BIOS) firmware; executing the BIOS firmware on the computing node to load the operating system for execution by the computing node; The method of claim 10.

14. The configuration of the rack assembly in the second configuration includes: changing a network configuration between the rack assembly and another rack assembly, the network configuration defining a communication path between the rack assembly and the other rack assembly; The method of claim 10.

15. A non-transitory computer-readable medium storing a computer program for performing booting of a system, comprising: program instructions for receiving, at a board management controller (BMC), startup configuration instructions for booting a compute node with an operating system, the compute node being arranged in a thread that includes a plurality of compute nodes, the BMC configured to manage a plurality of communication interfaces providing communication to the plurality of compute nodes; program instructions for transmitting boot instructions from the BMC via a communications interface to a boot controller of the computing node to execute a basic input / output system (BIOS) firmware stored external to the computing node; and program instructions for executing the BIOS firmware on the computing node to initiate loading of the operating system for execution by the computing node.

16. program instructions for receiving, at the BMC, a request from the computing node to access a storage address storing the BIOS firmware; and program instructions for facilitating access to the storage address by the computing node to obtain the BIOS firmware; the storage address is included in the boot instructions from the BMC; 16. The non-transitory computer-readable medium of claim 15.

17. The program instructions for transmitting the boot command include: including program instructions for accessing, by the BMC, the BIOS firmware at a storage address, the storage address being included in the boot configuration instructions; and program instructions for transmitting the BIOS firmware from the BMC to the computing node in association with the boot instructions for the execution of the BIOS firmware by the computing node.

16. The non-transitory computer-readable medium of claim 15.

18. The program instructions for executing the BIOS firmware include: program instructions for loading at least a portion of the operating system from a storage address into a system memory of the compute node; 16. The non-transitory computer-readable medium of claim 15.

19. further comprising program instructions for selecting the communication interface based on the operating system; the BMC is configured as a switch to deliver a control signal to each of a plurality of operating systems via a corresponding communication interface selected from the plurality of communication interfaces; said operating system control signals being delivered to said compute node via said communications interface; 16. The non-transitory computer-readable medium of claim 15.

20. program instructions for pausing a job being executed by the compute node during execution of an application by the compute node using a first operating system that is different from the operating system that is loaded for execution; program instructions for capturing a state of the compute node and storing the state; program instructions for transferring the state of the compute node to another compute node; and program instructions for executing the application on the other compute node to resume the job; the job is paused at a selected pause point which is not a predetermined pause point; 16. The non-transitory computer-readable medium of claim 15.

Images