Workload system with failover capacity reservation based on required threshold reliability and cluster reliability - Patents.com

Abstract

The method (600) includes determining a cluster confidence (162) of a computing cluster (142) that includes a maximum computing capacity (144). The method includes receiving a provisioning request (162) for the computing cluster that includes a threshold confidence (15) for the computing cluster. In response to the provisioning request, the method includes using the cluster confidence to determine a reserved computing capacity (172) for the computing cluster based on the threshold confidence. The reserved computing capacity is less than the maximum computing capacity. Based on the reserved computing capacity and the maximum computing capacity, the method includes determining an unreserved computing capacity (174) for the computing cluster. The method includes provisioning the computing cluster for execution of a user workload (20). The user workload executes on the unreserved computing capacity. The reserved computing capacity is initially unavailable for execution of the user workload.

Description

[Technical field]

[0001] The present disclosure relates to an adaptive workload system. [Background technology]

[0002] Service Level Objectives (SLOs) are an essential element of Service Level Agreements (SLAs) that specify the means by which performance is measured. For example, SLOs may be used to specify how performance is measured for a service provider that provides hardware and / or software to customers that run workloads. Such SLOs are generally structured for services associated with a platform (i.e., hardware and software) and have no specific application to the workloads running on the service. Thus, workloads need to be adapted to meet the platform's uptime guarantees. Uptime refers to the time the platform is up and running. Uptime typically remains fixed for workloads running with SLOs. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method for an adaptive workload system that, when executed by data processing hardware, causes the data processing hardware to perform operations. The operations include determining a cluster reliability of a computing cluster having a maximum computing capacity. The cluster reliability represents a reliability of the computing cluster when utilizing the entirety of the maximum computing capacity. The operations also include receiving a provisioning request from a user requesting provisioning of the computer cluster. The provisioning request includes a threshold reliability of the computing cluster. In response to receiving the provisioning request, a reserved computing capacity of the computing cluster is determined based on the threshold reliability of the computing cluster using the cluster reliability of the computing cluster. The reserved computing capacity is less than the maximum computing capacity. The operations also include determining an unreserved computing capacity of the computing cluster based on the reserved computing capacity of the computing cluster. The computing cluster is provisioned for execution of a user workload associated with the user, and the execution of the user workload is executed by the unreserved computing capacity of the computing cluster. The operations further include reserving the reserved computing capacity of the computing cluster. The reserved computing capacity of the computing cluster is initially unavailable for execution of the user workload.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the threshold confidence includes an uptime ratio of the computing cluster. Optionally, the operations include detecting a failure of the computing cluster. The failure affects unreserved computing capacity. In these examples, in response to detecting the failure, the operations include designating at least a portion of the reserved computing capacity as available for execution of user workloads.

[0005] In some examples, the computing cluster includes a plurality of components, and determining a cluster confidence of the computing cluster includes determining a respective component confidence for each component of the plurality of components and aggregating the respective component confidences. In some implementations, determining the reserved computing capacity of the computing cluster includes receiving a threshold confidence update request from a user, the threshold confidence update request including a second threshold confidence. Optionally, the threshold confidence may be adjusted based on the second threshold confidence.

[0006] After provisioning the computing cluster, in some examples, the operations include monitoring for failures affecting unreserved computing capacity. In some examples, the computing cluster includes a plurality of nodes, and reserving reserved computing capacity of the computing cluster includes tainting one or more nodes of the plurality of nodes. Each node of the plurality of nodes may include a maximum node computing capacity, and reserving reserved computing capacity of the computing cluster may include establishing, for each each node of the plurality of nodes, a computing capacity limit that is less than the maximum node computing capacity of each node. In some implementations, the operations further include detecting a failure of a respective one of the plurality of nodes, and removing the computing capacity limit from at least the respective one of the plurality of nodes in response to detecting the failure. Optionally, the operations of receiving the provisioning request include receiving a user input indication indicating a selection of a graphical element that sets the threshold confidence level in a graphical user interface executing on the data processing hardware for display on a screen in communication with the data processing hardware.

[0007] Another aspect of the disclosure provides a system for adaptive workloads. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include determining a cluster reliability of a computing cluster that includes a maximum computing capacity. The cluster reliability represents a reliability of the computing cluster when utilizing the entirety of the maximum computing capacity. The operations also include receiving a provisioning request from a user requesting provisioning of the computer cluster. The provisioning request includes a threshold reliability of the computing cluster. In response to receiving the provisioning request, a reserved computing capacity of the computing cluster is determined based on the threshold reliability of the computing cluster using the cluster reliability of the computing cluster. The reserved computing capacity is less than the maximum computing capacity. The operations also include determining an unreserved computing capacity of the computing cluster based on the reserved computing capacity of the computing cluster. The computing cluster is provisioned for execution of a user workload associated with the user, and the execution of the user workload is performed on the unreserved computing capacity of the computing cluster. The operations further include reserving the reserved computing capacity of the computing cluster. The reserved computing capacity of the computing cluster is initially unavailable for executing user workloads.

[0008] This aspect may include one or more of the following optional features: In some implementations, the threshold confidence includes an uptime ratio of the computing cluster. Optionally, the operations include detecting a failure of the computing cluster. The failure affects unreserved computing capacity. In these examples, in response to detecting the failure, the operations include designating at least a portion of the reserved computing capacity as available for execution of user workloads.

[0009] In some examples, the computing cluster includes a plurality of components, and determining the reliability of the computing cluster includes determining a respective component reliability for each of the plurality of components and aggregating each of the respective component reliability. In some implementations, determining the reserved computing capacity of the computing cluster includes receiving a threshold reliability update request from a user, the threshold reliability update request including a second threshold reliability. Optionally, the threshold reliability may be adjusted based on the second threshold reliability.

[0010] After provisioning the computing cluster, in some examples, the operations include monitoring for failures affecting unreserved computing capacity. In some examples, the computing cluster includes a plurality of nodes, and reserving reserved computing capacity of the computing cluster includes tainting one or more nodes of the plurality of nodes. Each of the plurality of nodes may have a maximum node computing capacity, and reserving reserved computing capacity of the computing cluster may include establishing, for each of the plurality of nodes, a computing capacity limit that is less than the maximum node computing capacity of each node. In some implementations, the operations further include detecting a failure of a respective one of the plurality of nodes, and removing the computing capacity limit from at least the respective one of the plurality of nodes in response to detecting the failure. Optionally, the operations of receiving the provisioning request include receiving a user input indication indicating a selection of a graphical element that sets the threshold confidence level in a graphical user interface executing on the data processing hardware for display on a screen in communication with the data processing hardware.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an example system for adaptive workloads. [Figure 2A] FIG. 2 is a schematic diagram of a computing cluster having unreserved and reserved computing capacity. [Figure 2B] FIG. 2 is a schematic diagram of a computing cluster having unreserved and reserved computing capacity. [Diagram 3] 1 is a schematic diagram of a component that monitors for at least one failure of a computing cluster. [Figure 4] FIG. 1 is a schematic diagram of a node of a computing cluster that has failed. [Figure 5A] 1 is a schematic diagram of an exemplary user device having a graphical user interface for adjusting the computational capacity of a computing cluster. [Figure 5B] 1 is a schematic diagram of an exemplary user device having a graphical user interface for adjusting the computational capacity of a computing cluster. [Figure 6] 4 is a flowchart illustrating an exemplary flow of operations for a method of providing an adaptive workload system. [Figure 7] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference numbers in the various drawings indicate like elements. A service level objective (SLO) is an essential element of a service level agreement (SLA) that specifies the means by which performance is measured. For example, a service level objective (SLO) may control the measurement of the performance of a container orchestration platform for one or more of the computing clusters. The SLO may control the monitoring and provisioning of the computing clusters to meet a particular uptime, where the uptime is the time that the computing cluster is operational (i.e., available to customers). However, traditional SLOs are typically applied to services and not to user workloads that may run on the services. This results in SLOs being monolithic services, whereby user workloads are adapted to stay within the uptime parameters defined in the SLO. For example, the uptime is typically fixed for workloads running in the SLO. Adapting user workloads to the SLO may limit the customizability of the SLO, and while some user workloads may easily adapt to the service limits, some user workloads may end up running slowly or inefficiently.

[0014] Embodiments herein are directed to an adaptive workload system that automatically adapts to implement dynamic SLOs (e.g., adjusted by the user) for executing user workloads. In other words, the system allows, for example, a user to adjust SLOs to user workloads rather than services. Through the adaptive workload system, the user can select the SLOs that best suit the user workloads, and the system adjusts the capacity of the services in response. For example, the system can adapt to selectively prioritize user workloads when resources are limited. In particular, hardware failures cause the system to adapt the services to accommodate the capacity requirements of the user workloads. In other words, SLOs are properties that the user can modify based on the needs of the current workload. This configuration of SLOs can result in dynamic changes that adapt to help provide, for example, uptime guarantees set by the user. For example, a user may use a service governed by SLOs for development, which requires less uptime guarantees compared to when the service is used for production workloads.

[0015] When a user adjusts an SLO, the service is configured as part of the adaptive workload system, which identifies the difference (e.g., in the uptime guarantee) based on the SLO adjustment. For example, the system can automatically adapt the service as the user adjusts the SLO based on the current workload needs, effectively updating and adjusting the uptime guarantee. This allows the user to effectively select the SLO that best fits the current workload requirements. Services may be configured via both the control plane and the data plane.

[0016] In some examples, the adaptive workload system dynamically adjusts both the control plane and data plane aspects of the service. In other examples, the adaptive workload system dynamically adjusts only a portion of the system (e.g., the data plane). For example, the data plane of a service refers to the computational capacity of the computing cluster that a user uses to run a workload. In this example, the data plane runs on a sliding scale of uptime based on the node capacity on the computing cluster. In other words, the system determines the appropriate computational capacity based on the SLOs adjusted or selected by the user and adjusts the available capacity of the computer cluster appropriately.

[0017] 1, in some implementations, an exemplary adaptive workload system 100 includes a remote system 140 that communicates with one or more user devices 10 over a network 112. The remote system 140 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic resources 114 including computing resources 116 (e.g., data processing hardware) and / or storage resources 118 (e.g., memory hardware). A data store 120 (i.e., a remote storage device) may be overlaid on the storage resources 118 to enable scalable use of the storage resources 118 by one or more of the clients (e.g., user devices 10) or computing resources 116.

[0018] The remote system 140 is configured to receive, for example, via the network 112, 112a, a provisioning request 14 from a user device 10 associated with a respective user 12. The user device 10 may correspond to any computing device, such as a desktop workstation, a laptop workstation, or a mobile device (i.e., a smartphone). The user device 10 includes computing resources 16 (e.g., data processing hardware) and / or storage resources 18 (e.g., memory hardware).

[0019] A user 12 executes one or more user workloads 20 on a computing cluster 142 as part of a service 110 provided by the adaptive workload system 100. The computing cluster 142 includes one or more nodes 200, 200a-n, each including computing resources 146 (e.g., data processing hardware) and / or storage resources 148 (e.g., memory hardware). A node data store 149 may be overlaid on the storage resources 148. In some examples, the computing cluster 142 includes on-premise hardware associated with the user 12. In this example, the user device 10, the remote system 140, and the computing cluster 142 communicate over one or more networks 112, such as a public network 112a and a private or local network 112, 112b. In other examples, the computing cluster 142 is part of the remote system 140 (i.e., cloud computing). A control plane of the service 110 may execute on the remote system 140, while a data plane of the service 110 may execute on the computing cluster.

[0020] The computing cluster 142 has a maximum computing capacity 144, and each node 200 of the computing cluster 142 has a maximum node computing capacity 210. Each node 200 may have a different maximum node computing capacity 210, and in general, the maximum computing capacity 144 of the computing cluster 142 is the sum of the maximum node computing capacity 210 of each node 200. When executing, a user workload 20 uses a portion of the maximum computing capacity 144 and one or more portions of the maximum node computing capacity 210 depending on the number of nodes 200 across which the user workload 20 is distributed.

[0021] The remote system 140 (or, in some examples, the computing cluster 142) executes the adaptive workload controller 150. The adaptive workload controller 150 includes a reliability module 160 that determines a cluster reliability 162 of the computing cluster 142 and / or the nodes 200. For example, the reliability module 160 receives cluster information 50 that includes information about the computing cluster 142 and / or the nodes 200 of the computing cluster 142 (such as the type, manufacturer, model number, and quantity of various components of the nodes 200 and / or the computing cluster 142). Using the cluster information 50, the reliability module 160 determines the cluster reliability 162 of the computing cluster (e.g., the mean time to failure (MTTF)). In some implementations, the reliability module 160 determines a node reliability 202, 202a-n for each node 200, and the cluster reliability 162 is an aggregate of each node reliability 202. For simplicity, the adaptive workload controller 150, and more generally the adaptive workload system 100, will be described with reference to a computing cluster 142. However, it is contemplated that the operations described herein may also be applied to the nodes 200 of the computing cluster 142.

[0022] The cluster reliability 162 of a computing cluster 142 may represent the cluster reliability 162 of the computing cluster 142 when the entirety of the maximum computing capacity 144 of the computing cluster 142 is needed (i.e., any event that reduces the maximum computing capacity 144 is considered a failure). For example, when the computing cluster 142 includes multiple nodes 200, the failure of any single node 200 results in a reduction in the maximum computing capacity 144. In general, it is contemplated that the cluster reliability 162 of a computing cluster 142 may be related to the MTTF of the computing cluster 142. The adaptive workload controller 150 automatically adjusts the availability of a portion of the maximum computing capacity 144 of the computing cluster 142 based on the determined cluster reliability 162, as described further below.

[0023] 1 , the adaptive workload controller 150 receives a provisioning request 14 from a user 12 over a network 112 to request provisioning of a computing cluster 142. The provisioning request 14, in some examples, includes a threshold confidence 15 of the computing cluster 142. The threshold confidence 15 may include an uptime percentage of the computing cluster 142.

[0024] In addition to the confidence module 160, the adaptive workload controller 150 of the adaptive workload system 100 includes a computing capacity module 170 and a provisioner 180. The computing capacity module 170 receives provisioning requests 14 from users 12 and cluster confidence 162 from the confidence module 160, and determines reserved computing capacity 172 for the computing clusters 142. As described in more detail below, the reserved computing capacity 172 represents the amount of computing capacity that must be reserved (based on the cluster confidence 162) to meet the threshold confidence 15.

[0025] The reserved computing capacity 172 is less than the maximum computing capacity 144, and the computing capacity module 170 determines the reserved computing capacity 172 based on the threshold confidence 15 and the cluster confidence 162 of the computing cluster 142. In other words, in response to receiving the provisioning request 14, the computing capacity module 170 determines the reserved computing capacity 172 of the computing cluster 142 using the cluster confidence 162 and the threshold confidence 15 of the computing cluster 142. The computing capacity module 170 also determines the unreserved computing capacity 174 based on the maximum computing capacity 144 and the reserved computing capacity 172. For example, to determine the unreserved computing capacity 174 of the computing cluster 142, the reserved computing capacity 172 is subtracted from the maximum computing capacity 144. It is understood that the computing capacity module 170 can alternatively first determine the unreserved computing capacity 174 and derive the reserved computing capacity 172 from the unreserved computing capacity 174.

[0026] The provisioner 180 provisions the computing cluster 142 for executing the user workload 20 associated with the user 12. The user workload 20 is initially executed only on the unreserved computing capacity 174 of the computing cluster 142. That is, the provisioner 180 provisions the computing cluster 142 such that only an amount of computing capacity of the computing cluster 142 equal to the unreserved computing capacity 174 is available for executing the user workload 20. The reserved computing capacity 172 of the computing cluster 142 is reserved for potential later use (e.g., in case of failure). That is, the reserved computing capacity 172 of the computing cluster 142 is initially unavailable for executing the user workload 20 (i.e., to meet the threshold confidence 15), but may be made available later. The provisioner 180 issues one or more provisioning commands 182 to provision the computing cluster 142.

[0027] 2A, in this example, the computing cluster 142 includes six nodes 200a-f, although in practice the cluster 142 may include any number of nodes 200. The reliability module 160 (FIG. 1) may determine a respective node reliability 202, 202a-f for each of the one or more nodes 200. When determining the cluster reliability 162 of the computing cluster 142, the respective node reliability 202 of each of the nodes 200a-f may be aggregated. In some examples, the reliability module 160 determines the node reliability 202 of each node 200 based on one or more component reliability of the components (e.g., CPU, RAM, non-volatile storage, network interface, etc.) that make up the node 200. The respective node reliability 202 (and / or component reliability) may then be utilized by the computing capacity module 170 (FIG. 1) to determine the reserved computing capacity 172 and the unreserved computing capacity 174.

[0028] In some implementations, the reserved computing capacity 172 refers to reserving at least one of the plurality of nodes 200a-f from executing the user workload 20. Reserving the node 200 may be accomplished by "tainting" the node 200. In some examples, tainting one or more of the nodes 200a-f results in the reserved computing capacity 172. For example, FIG. 2A shows two nodes 200e, 200f that the adaptive workload controller 150 has reserved to represent the reserved computing capacity 172. Meanwhile, the four nodes 200a-d of the computing cluster 142 represent the unreserved computing capacity 174 that may execute the user workload 20. That is, in this example, the reserved computing capacity 172 is created by reserving the entirety of one or more nodes 200 from executing the user workload 20. It is contemplated that the reserved computing capacity 172 may include more or less than two nodes 200 and the unreserved computing capacity 174 may include more or less than four nodes 200 .

[0029] Additionally or alternatively, as shown in FIG. 2B, the reserved computing capacity 172 and the unreserved computing capacity 174 may be implemented or applied individually to each node 200 of the computing cluster 142. In these implementations, a portion or percentage of the maximum node computing capacity 210 of each node 200 of the computing cluster 142 is designated as the reserved computing capacity 172, and the remainder is designated as the unreserved computing capacity 174. That is, in these examples, each node 200 reserves a respective portion of its maximum node computing capacity 210. For example, when reserving 20% ​​of the maximum computing capacity 144 of the computing cluster, each node 200 reserves 20% of their respective maximum node computing capacity 210. The unreserved computing capacity 174 of the computing cluster 142 is distinguished from the reserved computing capacity 172 in FIG. 2C by the dotted lines shown on each node 200 of the computing cluster 142. For example, reserved computing capacity 172 is shown as an upper region of node 200, and unreserved computing capacity 174 is shown as a lower region of node 200. Although not shown here, in addition to each node reserving a portion of maximum node computing capacity 210, one or more nodes 200 may be tainted or one or more entire nodes 200 may be reserved from operation in other ways.

[0030] 3, the adaptive workload controller 150, in some implementations, includes a monitor 310 and a resource controller 312. The monitor 310 monitors for failures of the unreserved computing capacity 174 of the computing cluster 142 (e.g., during execution of the user workload 20). For example, the monitor 310 receives operational data 314 from the computing cluster 142 regarding the operation of each node 200. The failures may include hardware and / or software failures of the node 200 (e.g., any components that make up the node 200) or hardware and / or software failures of the network that maintains communication with the node 200.

[0031] When the monitor 310 detects a failure affecting the unreserved computing capacity 174 (i.e., the ability of the computing cluster 142 to provide the user 12 with access to the entirety of the unreserved computing capacity 174), the resource controller 312 may designate at least a portion of the reserved computing capacity 172 as available for execution of the user workload 20. For example, the resource controller 312 may send a capacity adjustment command 316 to the computing cluster 142 to modify or adjust the designation of the reserved computing capacity 172 to make at least a portion of the reserved computing capacity 172 of the computing cluster 142 available for execution of the user workload 20. In some examples, a failure affecting the unreserved computing capacity 174 will cause the resource controller 312 to send a re-provisioning request to the adaptive workload controller 150. The re-provisioning request may re-provision the reserved computing capacity 172 and the unreserved computing capacity 174 of the respective node 200.

[0032] 4, in this example continuing from FIG. 2A, monitor 310 detects the failure of node 200a. Resource controller 312, in response to monitor 310 detecting the failure, removes the computational capacity limit from node 200e to compensate for the failed node 200a, thus "restocking" unreserved computational capacity 174 to its initial capacity. That is, resource controller 312 modifies the designation (e.g., removes the taint) of one of the nodes 200 (e.g., node 200e) currently designated as part of reserved computational capacity 172. Here, node 200e is identified by a dashed box indicating the change in designation from reserved computational capacity 172 to unreserved computational capacity 174. In this example, after the failure, five of the nodes 200a-e are designated as unreserved computing capacity 174 (with the failed node 200a not contributing to its respective maximum node computing capacity 210), and only one of the nodes 200f remains designated as reserved computing capacity 172. It is also contemplated that the failure of one node 200a may cause the resource controller 312 to individually adjust the reserved computing capacity 172 of each of the nodes 200b-f such that the percentage of unreserved computing capacity 174 is greater than the initially designated unreserved computing capacity 174 (FIG. 2B).

[0033] 5A, a schematic diagram 500a includes a user device 10 executing a graphical user interface (GUI) 510 for display on a screen 501 of the user device 10. The GUI 510 enables a user 12 to interact with the adaptive workload controller 150 via the user device 10. The GUI 510 includes a graphical representation 512 of a threshold confidence 15 of a computing cluster 142, which indicates the threshold confidence 15 along a graphical axis 514. The user 12 can provide a user input indication (e.g., via a touch screen, voice command, mouse, keyboard, etc.) to the GUI 510 indicating a selection of a graphical element 522 to set a value of the threshold confidence 15, and can cause the user device 10 to transmit the value of the threshold confidence 15 to the adaptive workload controller 150. For example, as illustrated, the user provides a user input indication to set a value of the threshold confidence 15 by manipulating a graphic slider 522, 522a along the graphical axis 514 to a first position 524a. That is, movement of the graphic slider 522a in a first direction (relative to the schematic 500a) along the graphical axis 514 may increase the value of the threshold confidence 15, while movement of the graphic slider 522a in an opposite second direction along the graphical axis 514 may decrease the value of the threshold confidence 15. In other examples, the user 12 may provide other types of user input indications indicating entry / selection of other types of graphical elements 522 to set the value of the threshold confidence 15 via a text box, button, dial, switch, etc.

[0034] For example, the GUI 510 may include a table 520 (in addition to or instead of the axis 514) from which the user 12 selects the amount of unreserved nodes 200 and thereby the reserved computing power 172 and / or threshold confidence 15 desired for operation of the computing cluster 142. Here, the GUI 510 may receive another user input indication of a selection of another graphical element (e.g., button) 522, 522b in the table 520, which may cause the GUI 510 to translate along the graphical axis 514 of the graphical representation 512 to move a graphic slider 522a from a first position 524a to a different position corresponding to an adjusted value of the threshold confidence 15. As an example, table 520 displayed in GUI 510 illustrates that selecting four nodes 200a-d as unreserved computing power 174 results in an expected computing power of 99.9923 percent, while selecting five nodes 200a-e as unreserved computing power 174 results in an expected computing power of 99.8958 percent. That is, reserving less computing power results in less estimated updates. It is also contemplated that user 12 may incrementally adjust threshold confidence 15 by manipulating a graphics slider 522a along graphical axis 514, which is a graphical representation 512 of threshold confidence 15, to set a second threshold confidence 518.

[0035] 5B, schematic diagram 500b is a continuation of the example of FIG. 5A. Here, a user may manipulate a graphic slider 522a and / or provide an additional user input indication of selecting another graphical element 522b to adjust the value of threshold confidence 15. For example, as shown in the example, a user provides an additional user input indication of setting an updated value of threshold confidence 15 by manipulating graphic slider 522a to a second position 524b along graphical axis 514. That is, user 12 manipulates graphic slider 522a and / or selects graphical element 522b in table 520 to change the position of threshold confidence 15 from a first position 524a (FIG. 5A) to a second position 524b. User device 10 transmits a threshold confidence update request 516 to adaptive workload controller 150 including a second threshold confidence 518. Computational capacity module 170 adjusts threshold confidence 15 based on the selected second threshold confidence 518. In other words, when determining the unreserved computing capacity 174 of the computing cluster 142, the computing capacity module 170 may receive a threshold confidence update request 516 from the user 12, which includes a second threshold confidence 518. Here, the user 12 may adjust the graphical representation 512 to move a first position 524a to a second position 524b representing the second threshold confidence 518. Specifically, the user 12 manipulates a portion of the graphical representation 512 along the graphical axis 514, whereby the graphical axis 514 and the graphical representation 512 of the threshold confidence 15 act as a sliding scale. Here, the user 12 is adjusting the threshold confidence from 99.99% uptime (FIG. 5A) to 99.89% uptime (FIG. 5B).

[0036] 6 is a flow chart illustrating an example flow of operations for a method 600 of an adaptive workload system. The computer-implemented method 600, when executed by the data processing hardware 116, causes the data processing hardware 116 to perform operations. The method 600 includes, at operation 602, determining a cluster confidence 162 of a computing cluster 142 that includes a maximum computing capacity 144. The cluster confidence 162 represents the cluster confidence 162 of the computing cluster 142 when utilizing the entirety of the maximum computing capacity 144. The method includes, at operation 604, receiving a provisioning request 14 from a user 12. The provisioning request 14 requests that the computing cluster 142 be provisioned and includes a threshold confidence 15 of the computing cluster 142. The method includes, at operation 606, in response to receiving the provisioning request 14, determining a reserved computing capacity 172 of the computing cluster 142 based on the threshold confidence 15 using the cluster confidence 162 of the computing cluster 142. The reserved computing capacity 172 is less than the maximum computing capacity 144. The method, at operation 608, determines an unreserved computing capacity 174 of the computing cluster based on the reserved computing capacity 172 and the maximum computing capacity 144 of the computing cluster 142. At operation 610, the method includes provisioning the computing cluster 142 for execution of a user workload 20 associated with the user 12. The user workload 20 is executed on the unreserved computing capacity 174 of the computing cluster 142. The method, at operation 612, includes reserving the reserved computing capacity 172 of the computing cluster 142. The reserved computing capacity 172 of the computing cluster 142 is initially unavailable for execution of the user workload 20.

[0037] 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. Computing device 700 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are intended to be illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0038] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 that connects to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 that connects to a low-speed bus 770 and storage device 730. Each of components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. Processor 710 is capable of processing instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730 for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories, as desired. Also, multiple computing devices 700 may be connected together (eg, as a bank of servers, a group of blade servers, or a multi-processor system) with each device providing a portion of the required operations.

[0039] The memory 720 stores information non-transiently within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0040] The storage device 730 can provide mass storage for the computing device 700. In some embodiments, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In further embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0041] The high-speed controller 740 manages the bandwidth-intensive operations of the computing device 700, and the low-speed controller 760 manages the bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is coupled to the memory 720, the display 780 (e.g., via a graphics processor or accelerator), and is coupled to a high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and the low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, via a network adapter.

[0042] The computing device 700 may be implemented in a number of different forms, as shown in the figure: for example, it may be implemented as a standard server 700a or as a repetition within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0043] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0044] Such computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0045] The processes and logic flows described herein may be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0046] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user (e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser).

[0047] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (600) that, when executed by data processing hardware (116), causes the data processing hardware (116) to perform operations, the operations comprising: receiving a cluster confidence (162) of a computing cluster (142) that includes a maximum computing capacity (144), the cluster confidence (162) representing a confidence of the computing cluster (142) when utilizing the entirety of the maximum computing capacity (144); receiving a provisioning request (14) from a user (12) requesting provisioning of said computing cluster (142), said provisioning request (14) including a threshold confidence level (15) of said computing cluster (142); in response to receiving the provisioning request (14), using the cluster confidence (162) of the computing cluster (142) to determine a reserved computing capacity (172) of the computing cluster (142) based on the threshold confidence (15) of the computing cluster (142), the reserved computing capacity (172) being less than the maximum computing capacity (144); determining an unreserved computing capacity (174) of the computing cluster (142) based on the reserved computing capacity (172) and the maximum computing capacity (144) of the computing cluster (142); provisioning the computing cluster (142) for execution of a user workload (20) associated with the user (12), the execution of the user workload (20) being performed on the unreserved computing capacity (174) of the computing cluster (142); A computer-implemented method (600) for reserving the reserved computing capacity (172) of the computing cluster (142), wherein the reserved computing capacity (172) of the computing cluster (142) is initially unavailable for execution of the user workload (20).

2. The method (600) of claim 1, wherein the threshold confidence (15) comprises an uptime ratio of the computing cluster (142).

3. The operation further comprises: detecting a failure of the computing cluster (142), the failure affecting the unreserved computing capacity (174) of the computing cluster (142); 3. The method (600) of claim 1 or 2, further comprising: in response to detecting the failure, designating at least a portion of the reserved computing capacity (172) as available for execution of the user workload (20).

4. The computing cluster (142) includes a number of components: Receiving the cluster trust (162) of the computing cluster (142) comprises: determining a respective component reliability for each of the plurality of components; A method (600) according to any of claims 1 to 3, comprising: aggregating each of the component confidences.

5. Determining the reserved computing capacity (172) of the computing cluster (142) comprises: receiving a threshold confidence update request (516) from the user (12), the threshold confidence update request including a second threshold confidence (15); The method (600) according to any of claims 1 to 4, further comprising: adjusting said threshold confidence (15) based on said second threshold confidence (15).

6. The method (600) of any of claims 1 to 5, wherein the operations further comprise, after provisioning the computing cluster (142), monitoring for failures affecting the unreserved computing capacity (174).

7. The computing cluster (142) includes a plurality of nodes (200); The method (600) of any preceding claim, wherein reserving the reserved computing capacity (172) of the computing cluster (142) comprises tainting one or more nodes (200) of the plurality of nodes (200).

8. The computing cluster (142) includes a plurality of nodes (200), each node (200) of the plurality of nodes (200) including a maximum node computing capacity (210); The method (600) of any of claims 1 to 7, wherein reserving the reserved computing capacity (172) of the computing cluster (142) comprises establishing, for each node (200) of the plurality of nodes (200), a computing capacity limit that is less than the maximum node computing capacity (210) of the respective node (200).

9. The operation further comprises: Detecting a failure of a respective one of the plurality of nodes; 10. The method of claim 8, further comprising: in response to detecting the failure, removing the computational capacity limit from at least a respective one of the plurality of nodes.

10. A method (600) according to any preceding claim, wherein receiving the provisioning request (14) comprises receiving a user input indication of a selection of a graphical element (522) that sets the threshold confidence (15) in a graphical user interface (510) executing on the data processing hardware (116) for display on a screen (501) in communication with the data processing hardware (116).

11. Data processing hardware (116); and memory hardware (118) in communication with the data processing hardware (116), the memory hardware (118) storing instructions that, when executed on the data processing hardware (116), cause the data processing hardware (116) to perform operations, the operations including: receiving a cluster confidence (162) of a computing cluster (142) that includes a maximum computing capacity (144), the cluster confidence (162) representing a confidence of the computing cluster (142) when utilizing the entirety of the maximum computing capacity (144); receiving a provisioning request (14) from a user (12) requesting provisioning of said computing cluster (142), said provisioning request (14) including a threshold confidence level (15) of said computing cluster (142); in response to receiving the provisioning request (14), using the cluster confidence (162) of the computing cluster (142) to determine a reserved computing capacity (172) of the computing cluster (142) based on the threshold confidence (15) of the computing cluster (142), the reserved computing capacity (172) being less than the maximum computing capacity (144); determining an unreserved computing capacity (174) of the computing cluster (142) based on the reserved computing capacity (172) and the maximum computing capacity (144) of the computing cluster (142); provisioning the computing cluster (142) for execution of a user workload (20) associated with the user (12), the execution of the user workload (20) being performed on the unreserved computing capacity (174) of the computing cluster (142); A system (100) reserving the reserved computing capacity (172) of the computing cluster (142), wherein the reserved computing capacity (172) of the computing cluster (142) is initially unavailable for execution of the user workload (20).

12. The system (100) of claim 11, wherein the threshold confidence (15) comprises an uptime ratio of the computing cluster (142).

13. The operation further comprises: detecting a failure of the computing cluster (142), the failure affecting the unreserved computing capacity (174) of the computing cluster (142); and in response to detecting the failure, designating at least a portion of the reserved computing capacity as available for execution of the user workload.

14. The computing cluster (142) includes a number of components: Receiving the cluster trust (162) of the computing cluster (142) comprises: determining a respective component reliability for each of the plurality of components; and aggregating each of the component confidences.

15. Determining the reserved computing capacity (172) of the computing cluster (142) comprises: receiving a threshold confidence update request (516) from the user (12), the threshold confidence update request including a second threshold confidence (15); Adjusting the threshold confidence (15) based on the second threshold confidence (15).

16. The system (100) of any of claims 11 to 15, wherein the operations further comprise, after provisioning the computing cluster (142), monitoring for failures affecting the unreserved computing capacity (174).

17. The computing cluster (142) includes a plurality of nodes (200); The system (100) of any one of claims 11 to 16, wherein reserving the reserved computing capacity (172) of the computing cluster (142) comprises tainting one or more nodes (200) of the plurality of nodes (200).

18. The computing cluster (142) includes a plurality of nodes (200), each of the plurality of nodes (200) having a maximum node computing capacity (210); The system (100) of any one of claims 11 to 17, wherein reserving the reserved computing capacity (172) of the computing cluster (142) comprises establishing, for each node (200) of the plurality of nodes (200), a computing capacity limit that is less than the maximum node computing capacity (210) of the respective node (200).

19. The operation further comprises: Detecting a failure of a respective one of the plurality of nodes; and removing the computing power limit from at least a respective one of the plurality of nodes in response to detecting the failure.

20. A system (100) as described in any of claims 11 to 19, wherein receiving the provisioning request (14) includes receiving a user input indication indicating a selection of a graphical element (522) that sets the threshold confidence level (15) in a graphical user interface (510) executing on the data processing hardware (116) for display on a screen (501) in communication with the data processing hardware (116).

Images