Techniques for orchestrating load shedding

Abstract

The disclosed technology relates to orchestrating power consumption reduction across multiple hosts. A plurality of response levels may be used, each response level associated with a corresponding set of reduction actions. An impact on the customer, the host, and / or the workload may be calculated at runtime based on current and / or predicted conditions and the workload, and a particular level of response may be selected based on the calculated impact. These techniques enable sufficient but minimal impact responses to be employed.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 423,762, titled "Orchestrated DC - Scale Load Shedding", filed on November 8, 2022; U.S. Provisional Application No. 63 / 439,576, titled "Orchestrated DC - Scale Load Shedding", filed on January 18, 2023; and U.S. Non - Provisional Application No. 18 / 338,962, titled "Techniques for Orchestrated Load Shedding", filed on June 21, 2023. The content of these applications is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to techniques for orchestrating the reduction of power consumption in a data center. The disclosed systems, methods, devices, and services implement dynamic orchestration methods to limit power consumption within the data center while ensuring a least - impact response for customers, hosts, and / or workloads. Background Art

[0004] Data centers are configured with a power infrastructure that provides various safety features in accordance with a hierarchical power distribution. Power is provided by a local power utility and is distributed to the various components of the data center, including power distribution units (PDUs) (e.g., transformers, switchboards, bus ducts, rack PDUs, etc.) and power - consuming devices (e.g., servers, network devices, etc.), according to this power distribution hierarchy. This ensures that the power consumed by all downstream devices complies with the power limits of each upstream device. When demand reaches a peak and / or when a component fails, the data center may no longer be able to effectively handle the demand, which may lead to a widespread power failure, causing at least severe damage to downstream devices, resulting in a reduction in processing capacity within the data center and / or a data center power failure, which in turn leads to a poor user experience. In the worst - case scenario, a power failure in one data center may trigger a cascading power failure of other devices within the same data center or other data centers as workloads are redistributed in an attempt to recover from the initial power failure. Additionally, in some embodiments, external factors (e.g., government regulations) may require the reduction of power consumption in a particular data center at a specific time.

[0005] Conventional methods of managing load shedding include manual tasks where an operator shuts down host racks or individual devices one by one to reduce power consumption. Alternatively, conventional techniques include pressing an emergency shutdown switch or button that is configured to shut off the power to an entire data center or a majority of the data center. The operator determining which components to shut down typically has no knowledge of the workloads or customers affected by these actions or the extent of the impact of taking these actions. Conventional methods like those mentioned here result in suboptimal approaches to affected customers and workloads. More sophisticated methods may have a lesser impact on customers and / or equipment / workloads while still providing the desired power reduction for the situation. Accordingly, it is desirable to improve power management techniques, particularly with respect to load shedding, to reduce power consumption to a sufficient impact and amount and avoid the drawbacks of conventional methods. Summary of the Invention

[0006] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is apparent that the various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

[0007] Some embodiments may include a method. The method may include identifying, by a computer system, multiple corresponding sets of workloads being executed on each of a plurality of hosts. The method may include identifying, by the computer system, a plurality of response levels that specify the applicability of a corresponding set of reduction actions to the plurality of hosts. In some embodiments, a first response level of the plurality of response levels may specify the applicability of a first set of reduction actions to the plurality of resources. The method may include determining, at least based on (a) the multiple corresponding sets of workloads being executed on each of the plurality of hosts and (b) the applicability of the first set of reduction actions to the plurality of hosts according to the first response level, a first estimate of the power reduction caused by the first response level. The method may include selecting, at least based on the first estimate of the power reduction caused by the first response level, the first response level from the plurality of response levels. The method may include causing the first set of reduction actions to be applied to the plurality of hosts according to the selected first response level.

[0008] In some embodiments, the method may further include determining, at least based on (a) the multiple corresponding sets of workloads being executed on each of the plurality of hosts and (b) the applicability of a second set of reduction actions to the plurality of hosts according to a second response level, a second estimate of the power reduction caused by the second response level of the plurality of response levels. In some embodiments, the selection of the first response level from the plurality of response levels is at least based on the first estimate of the power reduction caused by the first response level.

[0009] In some embodiments, determining a first estimate of the power reduction caused by a first response level includes any suitable combination of the following: 1) determining that a first set of reduction actions is applicable to a first subset of a plurality of hosts, 2) identifying multiple corresponding sets of workloads executed on each host in the first subset of hosts, and / or 3) determining the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts.

[0010] In some embodiments, determining a first estimate of the power reduction caused by a first response level further includes: determining the sum of the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts as the first estimate of the power reduction caused by the first response level.

[0011] In some embodiments, determining a first estimate of the power reduction caused by a first response level further includes any suitable combination of the following: 1) after applying the first set of reduction actions to the first subset of hosts, determining the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts, and / or 2) determining a first estimate of the power reduction caused by the first response level based on (a) the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts and (b) the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts after applying the first set of reduction actions to the first subset of hosts.

[0012] In some embodiments, the method further includes: 1) determining the difference between the current value of the total power consumption of the plurality of hosts and the current value of the total power threshold of the plurality of hosts; and 2) determining that a first value of the power reduction caused by the first response level is greater than the difference. In some embodiments, the first response level is selected at least based on determining that a first value of the power reduction caused by the first response level is greater than the difference.

[0013] In some embodiments, the multiple corresponding sets of workloads executed on each host in the plurality of hosts and the multiple response levels specifying the applicability of a corresponding set of reduction actions to the plurality of hosts are at least partially identified based on at least one of the following: 1) identifying degradation or failure of a temperature control system of a physical environment including the plurality of hosts, or 2) identifying a government reduction in power supply, or 3) identifying an increase in external temperature.

[0014] This disclosure presents a second method. The second method may include a computer system identifying multiple corresponding sets of workloads executed on each of multiple hosts. The second method may include a computer system identifying multiple response levels that specify the applicability of a corresponding set of reduction actions to the multiple hosts. In some embodiments, a first response level among the multiple response levels specifies the applicability of a first set of reduction actions to multiple resources, such that a first reduction action in the first set of reduction actions is applicable to a first resource among the multiple resources. The second method may include determining a first estimate of a predicted impact caused by the first response level based at least on (a) the multiple corresponding sets of workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level. In some embodiments, the workload impact is determined based on at least one of the host priority, the number of affected hosts, the number of affected workloads, the priority level of the affected workloads, the number of affected customers, or the priority level of the affected customers. The second method may include selecting the first response level from the multiple response levels based at least on the first estimate of the workload impact caused by the first response level. The second method may include applying the first set of reduction actions to the multiple hosts according to the selected first response level.

[0015] In some embodiments, the second method may include determining a second estimate of the workload impact caused by a second response level among the multiple response levels based at least on (a) the multiple corresponding sets of workloads executed on each of the multiple hosts and (b) the applicability of a second set of reduction actions to the multiple hosts according to the second response level. In some embodiments, the first response level is selected from the multiple response levels based at least on the first estimate of the workload impact caused by the first response level and the second estimate of the workload impact caused by the second response level.

[0016] In some embodiments, the first set of reduction actions is more stringent than the second set of reduction actions, and the first estimate of the workload impact caused by the first response level is less than the second estimate of the workload impact caused by the second response level.

[0017] In some embodiments, determining the first estimate of the workload impact caused by the first response level includes at least one of the following: 1) determining that the first set of reduction actions is applicable to a first subset of the multiple hosts, 2) identifying the multiple corresponding sets of workloads executed on each host in the first subset of hosts as the affected workloads caused by the first response level, or 3) determining at least one of (a) the number of affected workloads caused by the first response level, or (b) the type of priority of the affected workloads caused by the first response level.

[0018] In some embodiments, determining a first estimate of the workload impact caused by a first response level includes at least one of the following: 1) determining that a first set of reduction actions is applicable to a first subset of a plurality of hosts, 2) identifying multiple corresponding sets of workloads executed on each host in the first subset of hosts, 3) identifying the corresponding customers of the multiple corresponding sets of workloads executed on each host in the first subset of hosts as affected customers caused by the first response level, or 4) determining at least one of the following: (a) the number of affected customers caused by the first response level, or (b) the type of priority of the affected customers caused by the first response level.

[0019] In some embodiments, the second method further includes determining a second estimate of the power reduction caused by the first response level based at least on (a) multiple corresponding sets of workloads executed on each of the plurality of hosts and (b) the applicability of the first set of reduction actions to the plurality of hosts according to the first response level. In some embodiments, the first response level is selected from the multiple response levels based at least on the first estimate of the workload impact caused by the first response level and the second estimate of the power reduction caused by the first response level.

[0020] In some embodiments, the selected first response level is the response level among the multiple response levels associated with the minimum workload impact, and the power reduction achieved by the minimum workload impact is greater than or equal to the difference between the current value of the total power consumption of the plurality of hosts and the current value of the total power threshold of the plurality of hosts.

[0021] A third method is disclosed herein. The third method may include a computer system determining multiple corresponding sets of predicted workloads to be executed on each of the plurality of hosts during a future time period. The third method may include a computer system identifying multiple response levels that specify the applicability of a corresponding set of reduction actions to the plurality of hosts. In some embodiments, the first response level among the multiple response levels specifies the applicability of a first set of reduction actions to multiple resources. The third method may include determining a first estimate of the power reduction caused by the first response level based at least on (a) multiple corresponding sets of predicted workloads to be executed on each of the plurality of hosts and (b) the applicability of the first set of reduction actions to the plurality of hosts according to the first response level. The third method may include selecting the first response level from the multiple response levels based at least on the first estimate of the power reduction caused by the first response level. The third method may include identifying one or more workloads that (a) are currently being executed on the plurality of hosts and (b) will be affected by applying the first set of reduction actions to the plurality of hosts according to the selected first response level. The third method may include migrating the affected workloads from the plurality of hosts to one or more other hosts prior to the future time period.

[0022] In some embodiments, determining multiple sets of corresponding predicted workloads to be executed on each of a plurality of hosts during a future time period is based on historical patterns of workloads executed on the plurality of hosts.

[0023] In some embodiments, determining multiple sets of corresponding predicted workloads to be executed on each of a plurality of hosts during a future time period is performed using a machine learning model that has been previously trained using a supervised learning algorithm to predict a set of workloads based at least in part on historical workload data provided as training data.

[0024] In some embodiments, a third method includes at least one of the following: 1) obtaining, by a computer system, a predicted value of the total power consumption of a plurality of hosts during a future time period, 2) obtaining, by the computer system, a predicted value of the total power threshold of the plurality of hosts during a future time period, or 3) selecting, in response to determining that the predicted value of the total power consumption exceeds the predicted value of the total power threshold, from a plurality of response levels.

[0025] In some embodiments, a third method includes at least one of the following: 1) identifying, by a computer system, a predicted failure of a temperature control system associated with a plurality of hosts during a future time period, 2) obtaining, by the computer system, a predicted value of the total power threshold of the plurality of hosts during a future time period based at least in part on identifying the predicted failure, or 3) selecting, in response to identifying the predicted failure, from a plurality of response levels.

[0026] In some embodiments, identifying the predicted failure of the temperature control system utilizes a machine learning model that has been previously trained using training data including historical data associated with the plurality of hosts. In some embodiments, the machine learning model is trained using a supervised learning algorithm to identify the predicted failure from input data.

[0027] In some embodiments, the historical data of the training data includes temperature control system data, historical power consumption data corresponding to a set of hosts, and historical total power thresholds.

[0028] Systems, devices, and computer media are disclosed, where each of the systems, devices, and computer media may include one or more memories in which instructions corresponding to the methods disclosed herein may be stored. These instructions may be executed by one or more processors of the disclosed systems and devices to perform the methods disclosed herein. One or more computer programs may be configured to perform specific operations or actions corresponding to the described methods by including instructions that, when executed by a data processing device, cause the device to perform these actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Depicts an example physical environment (e.g., a data center or a portion thereof) including various components according to at least one embodiment.

[0030] Figure 2 Shows a simplified diagram of an exemplary power distribution infrastructure including various components of a data center according to at least one embodiment.

[0031] Figure 3 Illustrates an example architecture of an exemplary power orchestration system according to at least one embodiment, the power orchestration system being configured to orchestrate power consumption reduction of various resources applicable to a physical environment (e.g., a data center, a room in a data center, etc.).

[0032] Figure 4 Illustrates an example architecture of a power management service for detecting consumption overages and constraining power consumption overages according to at least one embodiment.

[0033] Figure 5 Illustrates according to at least one embodiment the Figure 2 corresponding example power distribution hierarchy of the arrangement of components.

[0034] Figure 6 Is a flowchart illustrating an example method for managing excessive power consumption according to at least one embodiment.

[0035] Figure 7 Illustrates an example architecture of a VEPO orchestration service for orchestrating power consumption constraints and / or power-off tasks according to at least one embodiment.

[0036] Figure 8 Is a table illustrating a set of example response levels and multiple sets of corresponding reduction actions for each response level according to at least one embodiment.

[0037] Figure 9 Is a diagram illustrating an example scope of components affected by one or more response levels used by a VEPO orchestration service for orchestrating power consumption constraints and / or power-off tasks according to at least one embodiment.

[0038] Figure 10 Is a block diagram illustrating an example use case of applying multiple response levels based on current conditions of a set of hosts according to at least one embodiment.

[0039] Figure 11 Is a schematic diagram of an example user interface according to at least one embodiment.

[0040] Figure 12 Illustrates the flow of an example method for training one or more machine learning models according to at least one embodiment;

[0041] Figure 13 is a block diagram illustrating an example method for achieving a response level among multiple response levels based at least in part on an estimated power reduction;

[0042] Figure 14 is a block diagram illustrating an example method for achieving a response level among multiple response levels based at least in part on a predicted impact;

[0043] Figure 15 is a block diagram illustrating an example method for pre - migrating workloads affected by a selected response level;

[0044] Figure 16 is a block diagram illustrating a mode for implementing an Infrastructure as a Service (IaaS) system of a cloud;

[0045] Figure 17 is a block diagram illustrating another mode for implementing an Infrastructure as a Service (IaaS) system of a cloud;

[0046] Figure 18 is a block diagram illustrating another mode for implementing an Infrastructure as a Service (IaaS) system of a cloud;

[0047] Figure 19 is a block diagram illustrating another mode for implementing an Infrastructure as a Service (IaaS) system of a cloud;

[0048] Figure 20 is a block diagram illustrating an example computer system according to at least one embodiment; Detailed Description

[0049] In the following description, various embodiments will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, those skilled in the art will also understand that the embodiments can be practiced without specific details. Additionally, well - known features may be omitted or simplified so as not to obscure the described embodiments.

[0050] The present disclosure relates to managing power consumption and orchestrating power consumption reduction within an environment (e.g., a host or a power distribution unit of a data center, or a portion thereof). More particularly, techniques for enabling orchestrated load shedding to achieve power consumption reduction due to current conditions and according to a total power threshold are described.

[0051] It is desirable to balance the power supply and consumption within a data center. When the consumed power exceeds the available supply, the balance can be restored by increasing the power supply or reducing the power consumption rate. If this balance is not maintained and components attempt to consume more power than is available, then the circuit breaker may trip, disconnecting the components from the power source.

[0052] The maximum total power threshold for a particular environment may depend on many factors, including but not limited to the current power consumption value at the hosts within the environment, the operating conditions of associated components (such as temperature control systems (e.g., HVAC, coolers, etc.)), environmental conditions (e.g., ambient temperature, external temperature outside the environment), etc. Conventional systems are not configured to manage power consumption based on these factors. When power consumption peaks and power failures are impending, conventional techniques mainly utilize manual efforts to shut down or suspend hosts. This results in suboptimal load shedding strategies because the operators implementing the load shedding typically do not understand the impact of their actions. For example, when choosing which devices to shut down or suspend, the operator rarely (if ever) knows the customers, workloads, instances, or hosts or their respective priorities. A power failure can cause interruptions to the operations performed by the components of the data center. For example, if a circuit breaker trips, disconnecting a server from the power source of the data center, then the websites hosted on the data center server will crash. Suboptimal load shedding strategies can be inefficient and may potentially exceed the amount of power reduction sufficient to reduce the total power consumption to the desired amount. Additionally or alternatively, suboptimal load shedding strategies may have a broader / greater impact on the underlying hosts, instances, workloads, and corresponding customers than might be expected.

[0053] The balance between the power supply and power consumption of a data center can be managed by maintaining a balance between the available supply and the demand and / or consumption. Sometimes, it may not be feasible to expand the power supply in a data center because power is typically statically allocated in long-term contracts with power utilities. Although the supply may be statically allocated, the power consumption in a data center can vary, sometimes significantly. As a simple example, when the number of threads executed by a server's processor increases, the power consumption of the server may increase. The ambient temperature within the data center affects the power consumed by the data center's cooling system. When the cooling system works harder, it consumes more power as it tries to reduce the ambient temperature experienced in the data center. When the cooling system fails, the data center may no longer be able to withstand the heat generated by the current power consumption level. The demand caused by some components in the data center, such as uninterruptible power supplies, power distribution units, cooling systems, or bus ducts, may be difficult to adjust. However, some components, such as servers, virtual machines, and / or bare-metal instances, may be more easily constrained. Additionally, workloads and / or instances may be migrated to other hosts and / or instances to concentrate resources on a smaller subset of hosts, resulting in more idle and / or vacant hosts. Applying power reduction to the idle / vacant hosts thereafter can ensure that the impact (e.g., the number of affected hosts, customers, instances, workloads) is minimized.

[0054] Many conventional power management methods utilize power capping to constrain the operation of power-consuming devices (e.g., servers, network devices, etc.) within a data center. When power capping is used, a power ceiling limit can be used to constrain the power consumed by a device. The power ceiling limit is used to constrain (e.g., throttle) the operation at a server to ensure that the power consumption of the server does not exceed the power ceiling limit. Using power capping ensures that the allocated power limit of upstream devices is not breached, resulting in each upstream device being supplied according to the worst-case scenario, where it is assumed that each downstream device consumes its corresponding maximum allocated power. However, these downstream devices may typically consume less power than their maximum allocated power, leaving at least a portion of the power allocated to the upstream devices unutilized. These methods waste valuable power and limit the density of power-consuming devices that can be used within the data center.

[0055] An efficient power infrastructure within a data center is necessary to increase a provider's profit margin, manage scarce power resources, and make the services provided by the data center more environmentally friendly. A data center that includes components of a hosted multi-tenant environment (e.g., a public cloud) may experience above-average consumption because not all cloud tenants use it simultaneously. To push power consumption closer to the statically allocated capacity to increase the efficiency and resource utilization of the data center, the data center provider can increase servers and / or leases so that the power consumption across all power-consuming devices is closer to the allocated power capacity of the data center. However, in some cases, narrowing the gap between the allocated power capacity and power consumption increases the risk of circuit breaker trips and the loss of the ability to utilize computing resources. The techniques described herein minimize the frequency at which operations at downstream devices are constrained, enabling these devices to utilize previously unutilized power while maintaining a high level of safety in avoiding power failures.

[0056] Technical Effects

[0057] The disclosed systems and methods provide an automated and dynamic orchestration approach to constrain power consumption (e.g., via power capping, migrating workloads, etc.), suspend hosts, migrate instances and / or hosts, and / or shut down hosts to achieve a desired power reduction. In some embodiments, at least a portion of the functions performed for these actions are user-selectable and / or based on user input. The techniques described provide multiple response levels. Each response level can be associated with a set of reduction actions (referred to herein as "actions" for brevity). Each response level (referred to herein as a "level" for brevity) can provide multiple sets of increasingly stringent reduction actions to be applied. When a total power threshold is breached, a response level can be selected based on the estimated power reduction corresponding to each response level (e.g., by the system based on user input, etc.). The least stringent level and / or the level with the least impact can be selected (e.g., by the system based on user input, etc.) that, if implemented, is sufficient to cause the total power consumption of the data center to drop below the total power threshold.

[0058] The estimated impact of applying level-related reduction actions can be determined at runtime based on the current properties of the workloads implemented on virtual machine and / or bare-metal instances, the customers associated with the workloads and / or affected hosts, and the priorities corresponding to these workloads, hosts, and / or customers. The demand (e.g., corresponding to a total power threshold) can be dynamically modified based on the operational status data and environmental data corresponding to the various components of the data center. These factors can be used to identify changes in the total power threshold (e.g., the threshold amount of total power consumption / heat / demand that the components of the data center can jointly manage) as the power management capabilities within the data center occur. The response level selection can be triggered by real-time demand changes, via a request (e.g., a user request, a request submitted by a government entity to enforce power consumption reduction, etc.), or any suitable trigger, and can be based on the impact of implementing the corresponding reduction action.

[0059] Using the techniques described herein, the power constraints, migration tasks, and / or pause or shutdown tasks employed can be adjusted according to the current conditions and resources (e.g., hosts, instances, workloads) in the data center, thereby mitigating or completely avoiding over-reduction. These techniques provide real-time capabilities for reducing the impact of power management response actions on customers, hosts, instances, and / or workloads, while ensuring effective avoidance of the risk of power failures. The systems and methods described herein provide a more efficient and effective power orchestration method than conventional systems, which can lead to a more satisfactory user experience. The specific actions to be implemented can be selected based at least in part on any suitable combination (e.g., automatically by the system, through user selection, etc.): 1) maximizing the overall power consumption reduction, 2) identifying / implementing the least stringent response level, 3) identifying / implementing the response level with the least impact, and / or 4) identifying the response level whose estimated power reduction is sufficient to bring the current total power consumption value below the current or predicted total power threshold, while also providing the least amount of over-power reduction (e.g., reduction exceeding that required to bring the current power consumption below the current total power threshold). Thus, these techniques provide a more flexible and intelligent approach to shedding load in various contexts and / or due to various trigger events.

[0060] Figure 1Illustrates an example environment (e.g., environment 100) including various components according to at least one embodiment. Environment 100 can be a physical environment, such as a data center (e.g., data center 102) or a portion thereof (e.g., a room of the data center, such as room 110A). Environment 100 can include a dedicated space for hosting any suitable number of servers, such as servers 104A-P (collectively referred to as "servers 104"), and the infrastructure for hosting these servers, such as network hardware, a cooling system (also referred to as a "temperature control system"), and storage devices. Servers 104A-P can also be referred to as "hosts". The networking hardware of data center 100 (not depicted here) can enable remote users to interact with the servers via a network (e.g., the Internet). Any suitable number of servers 104 (e.g., 10, 14, 21, 42, etc.) can be maintained in various racks, such as racks 106A-H (collectively referred to as "racks 106"). Racks 106 can include a framework or enclosure in which a corresponding set of servers is placed and / or installed.

[0061] Various subsets of racks 106 can be organized into groups referred to as "rows" (e.g., rows 108A-D, collectively referred to as "rows 108"). In some embodiments, rows 108 can include any suitable number of racks (e.g., 5, 8, 10, up to 10, etc.) that are co-located (e.g., within a threshold distance of each other). In other embodiments, a row can be an organizational unit, and the racks having a given row can be placed in different locations (not necessarily within a threshold distance of each other). As an example, rows 108 can be located in a room (e.g., room 110A, room 110N, etc.). A room (e.g., room 110A) can be a partition or physical enclosure or physical environment of a building in which any suitable number of racks 106 are placed. In other embodiments, a room can be an organizational unit, and rooms can be located in different physical locations, or multiple rooms can be located in a single partition of a building.

[0062] Various temperature control systems (e.g., one or more temperature control systems 112A-N, one or more temperature control systems 114A-N, etc.) can be configured to manage the ambient temperature in a data center or a portion thereof. As a non-limiting example, one or more temperature control systems 112A-N can be associated with room 110A, while one or more temperature control systems 114A-N can be associated with room 110N. Any suitable number of temperature control systems can be associated with the data center and / or portions thereof. In some embodiments, these temperature control systems can be any suitable heating, ventilation, and air conditioning (HVAC) equipment (e.g., air conditioning units), coolers (e.g., chilled water circulation equipment that controls temperature by circulating a liquid such as water), etc. In some embodiments, each temperature control system can be associated with the corresponding amount of heat that it is capable of and / or configured to manage (e.g., the amount of heat generated by the corresponding power consumption of server 104).

[0063] Figure 2 FIG. shows a simplified diagram of an exemplary power distribution infrastructure 200 of a data center 102 including various components (e.g., Figure 1 components of the data center 102). The power distribution infrastructure 200 can be connected to a utility power source (not shown), and power can initially be received at one or more uninterruptible power supplies (uninterruptible power supplies ((one or more) UPSs) 202). In some embodiments, power can be received from the utility at the (one or more) UPSs via a substation on site (not shown) that is configured to establish an appropriate voltage level for distributing power through the data center. The (one or more) UPSs 202 can individually include dedicated batteries or generators that provide emergency power in the event of a failure of the input power source. The (one or more) UPSs 202 can monitor the input power and provide backup power when a drop in the input power is detected.

[0064] The power distribution infrastructure 200 may include any suitable number of intermediate power distribution units ((one or more) PDUs) (e.g., (one or more) intermediate PDUs 204), which are connected to (one or more) UPSs 202 and receive power / electricity therefrom. Any suitable number of (one or more) intermediate PDUs 204 may be deployed between the UPS (in (one or more) UPSs 202) and any suitable number of row PDUs (e.g., row PDU 206). The power distribution units (e.g., (one or more) intermediate PDUs 204, row PDU 206, (one or more) rack PDUs 208, etc.) may be any suitable devices configured to control and distribute power / electricity. Example power distribution units may include, but are not limited to, main distribution boards, distribution boards, remote distribution boards, busbars, power strips, transformers, etc. Power may be provided from (one or more) UPSs 202 to (one or more) intermediate PDUs 204. The (one or more) intermediate PDUs 204 may distribute power to downstream components of the power distribution infrastructure 200 (e.g., row PDU 206).

[0065] The power distribution infrastructure 200 may include any suitable number of row power distribution units (including row PDU 206). The row PDU may include any suitable PDU (e.g., remote distribution board, busbar / raceway, etc.) that is deployed between an intermediate PDU (e.g., the PDU in (one or more) intermediate PDUs 204) and one or more rack PDUs (e.g., rack PDU 208A, rack PDU 208N, collectively referred to as "(one or more) rack PDUs 208"). A "row PDU" refers to a PDU configured to distribute power to one or more rows of devices (e.g., row 210, including servers 212A-D, collectively referred to as "servers 212"). As described above, a row (e.g., row 210) may include any suitable number of racks (e.g., racks 214A-N, collectively referred to as "racks 214") in which the servers 212 are located.

[0066] The power distribution infrastructure 200 may include any suitable number of rack power distribution units (including (one or more) rack PDUs 208). The rack PDU may include any suitable PDU that is deployed between a row PDU (e.g., row PDU 206) and a rack (e.g., rack 214A, Figure 1Between one or more servers (e.g., server 212A, server 212B, etc.) corresponding to the rack 106 of the example). A "rack PDU" refers to any suitable PDU configured to distribute power to one or more servers within a rack. A rack (e.g., rack 214A) can include any suitable number of servers 212. In some embodiments, the (one or more) rack PDUs 208 can include intelligent PDUs, which are also configured to monitor, manage, and control power consumption at multiple devices (e.g., rack PDU 208A, server 212A, server 212B, etc.).

[0067] Each of the servers 212 (for each example of the server 104 of Figure 1 can separately include a power controller ((one or more) power controllers 216A-D, collectively referred to as "power controller 216"). A power controller refers to any suitable hardware or software component operating at a device (e.g., a server) configured to monitor and / or manage power consumption at the device. The power controllers 216 can separately monitor the power consumption of the respective servers on which they operate. The power controllers 216 can each be configured to enforce a power capping limit to constrain the power consumption at the respective servers. Enforcing the power capping limit can include any suitable combination of the following: monitoring the power consumption at the server, determining whether to constrain (e.g., limit, define, etc.) the power consumption at the server (e.g., at least in part based on a comparison between the current power consumption of the server and the stored power capping limit), and limiting / defining the power consumption at the server (e.g., using processor and memory dynamic voltage and frequency scaling to suppress server power consumption). Enforcing the power capping limit can be referred to as "power capping".

[0068] Figure 1 The data center 102 in can include various components depicted in the power distribution infrastructure 200. For example, room 110A can include one or more bus ducts (each bus duct being an example of a row PDU 206). A busbar (also referred to as a "bus duct") refers to a conduit of conductive material that can distribute power (e.g., within room 110A). The bus duct can receive power from the power distribution unit of the (one or more) intermediate PDUs 204 and supply power to one or more racks (e.g., Figure 1 associated with the row 108A of Figure 1 rack 106A of Figure 1 rack 106B, etc.). Each power infrastructure component that distributes / provides power to other components also consumes a portion of the power passing through it. This loss can be due to heat loss generated when power flows through the component or power directly consumed by the component (e.g., power consumed by the processor in a rack PDU).

[0069] Figure 3 Illustrates an example architecture of an example power orchestration system 300 according to at least one embodiment, the power orchestration system being configured to orchestrate power consumption reduction of various resources applicable to a physical environment (e.g., a data center, a room of a data center, etc.). The term "resource" can be considered to include hosts, workloads (e.g., virtual machines and / or bare-metal instances running these workloads), and / or customers associated with these hosts and / or workloads / instances. The power orchestration system 300 can be configured to monitor power consumption levels (e.g., current individual and / or total power consumption values corresponding to various hosts such as (one or more) hosts 324, each of which is an example of (one or more) servers 104). The power orchestration system 300 can manage power consumption within the physical environment at least in part based on this monitoring such that a total power threshold is enforced. As used herein, the total power threshold represents the maximum power consumption amount that the components of the system 300 can manage under current conditions. The total power threshold can be dynamically adjusted as conditions change (e.g., based on an order from a government entity to reduce power consumption, based on environmental conditions (actual or predicted), based on current power consumption values (actual or predicted), based on the operating status (actual or predicted) of various components such as the temperature control system of the physical environment, etc.). The specific impact (e.g., scope, applicability) of the changes made to enforce the current total power threshold may vary depending on real-time conditions. Some example actions that can be employed to enforce the current total power threshold can be power capping (e.g., enforcing a maximum power consumption ceiling at a particular device such as one or more hosts 324) or otherwise allocating a budgeted amount of power to any appropriate device (e.g., Figure 1 one or more of the hosts 324 in, (one or more) PDUs 202, 204, 206, 208, etc.). Figure 2 Suspending or shutting down hosts and / or instances (e.g., VMs or BMs) and / or workloads running at the instances, migrating a workload from one instance to another, migrating a workload from one host to another, etc.

[0070] The power orchestration system 300 can include various components such as Figure 3 the components depicted in. For example, the power orchestration system 300 can include a VEPO orchestration service 302. The VEPO orchestration service 302 can be configured to obtain various data based on which impact and reduction actions can be determined. By way of example, the VEPO orchestration service 302 can be configured to obtain any suitable combination of the following: power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318.

[0071] Power data 310 may include any suitable budget / assignment value, power ceiling value, and / or current power consumption value of (one or more) hosts 324 or (one or more) PDUs (including rack PDUs 320 ( Figure 2 example of the rack PDU 208 of)), which indicates the amount of power currently consumed by the device. The power data 310 may be stored in a location accessible to the VEPO orchestration service 302, and / or the power data 310 may be obtained from a source and / or service configured to manage and / or obtain such data. In some embodiments, the power management service 304 may be configured to obtain the power data 310 and store such data in a location from which the VEPO orchestration service 302 can retrieve the data. In some embodiments, the power management service 304 or another component configured to obtain such data may provide the power data 310 to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or directly via a request.

[0072] Account data 312 may include any suitable attributes of the customer, any suitable attributes of the (one or more) hosts and / or (one or more) instances associated with the customer, etc. Some example attributes may include the category associated with the customer account (e.g., free tier category, which indicates that the customer is using the service in a non-paying manner that may be provided with limited features or resources; free trial category, which indicates that the customer is using the service in a non-paying manner for a limited time). The category may correspond to the priority associated with the customer, or the priority may be assigned to the customer, host, or workload in other ways. The priority may indicate the degree of relationship importance, such as low priority, medium priority, and high priority, but other priority schemes may also be considered. Another example attribute may include an identifier associated with the customer, host, instance, and / or workload. The account data 312 may be stored in a location accessible to the VEPO orchestration service 302, and the account data 312 may be retrieved from that location, or the account data 312 may be obtained from a service and / or source configured to manage and / or obtain such data (e.g., an account service of a cloud computing environment not shown). In some embodiments, the account data 310 is provided directly from the source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0073] Environmental data 314 can include any suitable data associated with the physical environment or environmental data indicating external conditions. By way of example, environmental data 314 can include environmental temperature readings of the physical environment, external temperature outside the physical environment, design points indicating the external temperature that the physical environment is designed to withstand, and the like. At least a portion of environmental data 314 can be collected from one or more sensors (not shown) configured to measure specific conditions such as environmental temperature within the physical environment, external temperature outside the physical environment, and the like. In some embodiments, at least a portion of environmental data 314 can be provided by a weather source (such as a weather forecast stored in the storage device of the power orchestration system 300) or obtained from an external source (such as a weather forecast website). In some embodiments, environmental data 314 can include tables and / or protocols from which a reduced capacity / capability can be determined / identified. By way of example, a table or mapping can be included in environmental data 314 indicating that a particular difference between the environmental temperature and the temperature outside the physical environment (referred to as the "external" temperature) is associated with a particular reduced capacity of power consumption that the components of the system or individual components can manage. Environmental data 314 can be stored in a location accessible to the VEPO orchestration service 302, and environmental data 314 can be retrieved from that location, or environmental data 314 can be obtained from services and / or sources configured to manage and / or obtain such data (e.g., a metrics service of a cloud computing environment, the power management service 304, etc.). In some embodiments, environmental data 314 is provided to the VEPO orchestration service 302 directly from the source or a component configured to obtain such data according to a predefined schedule, frequency, or periodicity or via a request.

[0074] Host / instance data 316 can include any suitable data associated with a host and / or an instance. Host / instance data 316 can include workload metadata that identifies the workloads executed at the host and / or via a particular instance. In some embodiments, host / instance data 316 can identify the corresponding customer, host, and / or instance and / or the corresponding priority of the workload, and the like. At least a portion of host / instance data 316 can initially be obtained and / or maintained by a separate service (e.g., the compute service 306, an example of a compute service control plane of a cloud computing environment). Host / instance data 316 can be stored in a location accessible to the VEPO orchestration service 302, and host / instance 316 can be retrieved from that location, or host / instance 316 can be obtained from services and / or sources configured to manage and / or obtain such data (e.g., the compute service 306, etc.). In some embodiments, host / instance data 316 is provided to the VEPO orchestration service 302 directly from the source or a component configured to obtain such data according to a predefined schedule, frequency, or periodicity or via a request.

[0075] The operational data 318 can include any suitable data associated with the operational status or conditions of one or more devices or components corresponding to a physical environment. As a non-limiting example, the operational data 318 can include the operational status of one or more temperature control systems (e.g., Figure 1 the temperature control system(s) 112) of Figure 1 . In some embodiments, the operational data 318 can indicate which temperature control systems are operational and / or the operational capacity of these systems. In some embodiments, the environmental data 314 can include tables and / or protocols from which a reduced capacity / capability can be determined / identified. By way of example, a table or mapping can be included in the operational data 318 indicating that a failure (e.g., complete or partial) of a particular component (e.g., a particular temperature control system) is associated with a particular reduced capacity of power consumption that can be managed by the components of the overall system or a single component of the system. At least some portion of the operational data 318 can be initially obtained and / or maintained by a separate service (e.g., the power management service 304, etc.). The operational data 318 can be stored in a location accessible by the VEPO orchestration service 302, and the operational data 318 can be retrieved from that location, or the operational data 318 can be obtained from a service and / or source (e.g., the compute service 306, etc.) configured to manage and / or obtain such data. In some embodiments, the operational data 318 can be provided directly to the VEPO orchestration service 302 from a source or a component configured to obtain such data according to a predefined schedule, frequency, or periodicity or via a request.

[0076] It should be appreciated that the power data 310, account data 312, environmental data 314, host / instance data 316, or operational data 318 can include current data indicating current values and / or historical data indicating corresponding historical values. In some embodiments, the future attributes of the power data 310, account data 312, environmental data 314, host / instance data 316, or operational data 318 can be predicted at least in part based on these historical values. In some embodiments, machine learning can be utilized to predict any suitable portion of these future attributes. Example methods for predicting the future values of the power data 310, account data 312, environmental data 314, host / instance data 316, or operational data 318 are discussed in more detail with respect to Figure 12 Figure 12 . In some embodiments, the VEPO orchestration service 302 can be configured to aggregate data of any suitable combination of the power data 310, account data 312, environmental data 314, host / instance data 316, or operational data 318 into a table, mapping, or database from which the data can be filtered or sorted to identify a subset of resources (e.g., hosts, instances, workloads, customers) for a given set of constraints (e.g., low-priority workloads, free-tier customers, etc.).

[0077] The VEPO orchestration service 302 can be configured to identify and / or modify the total power threshold associated with the physical environment based at least in part on any suitable combination of the following: current / total power consumption values (actual or predicted) associated with any suitable combination of (one or more) hosts 324; a received request associated with a government entity (e.g., a local power authority) that commands / requests a specific power reduction or an overall power reduction, possibly for a specific time period (e.g., the next 24 hours); current or predicted environmental conditions (e.g., current or predicted ambient temperature, current or future external temperature, etc.); current or predicted operating conditions corresponding to a temperature control system (e.g., current or predicted total / partial failure), etc.

[0078] The VEPO orchestration service 302 can be configured to manage the actual power consumption values corresponding to the hosts / instances / workloads of the physical environment through associated functions or functions provided by other systems and / or services. Managing the actual power consumption values can include power capping, pausing workloads, instances, hosts, shutting down workloads / instances / hosts, migrating an instance from one host to another host, migrating a workload from one instance and / or host to another instance and / or host, etc.

[0079] The VEPO orchestration service 302 can obtain configuration data (e.g., maps, protocol sets, rules, etc.) corresponding to multiple response levels. Each response level can be associated with a corresponding set of reduction actions (e.g., power capping, migration, pause, shutdown, etc.) that can be performed on components of the physical environment (e.g., (one or more) servers 104, (one or more) PDUs 202, 204, 206, 208, etc.). In some embodiments, each set of reduction actions corresponding to a specific level is associated with potentially different reductions in the total power consumption of the data center. In some embodiments, the multiple response levels indicate an increasing severity of performing reduction actions to reduce the total power consumption of the data center. "Severity" is intended to refer to the relative degree of destructiveness of an action. For example, the action of power capping a server may be considered less severe than the action of completely shutting down the server because in the former, the server can still provide some processing capacity, albeit with reduced capacity, while in the latter, the server provides no processing capacity. A set of example response levels is discussed in more detail. Figure 9 A set of example response levels is discussed in more detail.

[0080] The VEPO orchestration service 302 can be configured to determine the impact of applying a reduction action of a given level to the hosts / instances / workloads (collectively referred to as "resources") of a physical environment using configuration data corresponding to the specification of the (one or more) response levels and their corresponding reduction actions. The impact of applying a given action may depend on the current conditions of the runtime resources. In some embodiments, identifying the impact of a given reduction action may include identifying a set of resources and / or customers for which the action is applicable if implemented. Thus, the estimated impact is intended to refer to the scope or applicability of a given action or level. Identifying the scope and / or applicability of a given action or level may include identifying the specific resources and / or customers that will be affected by implementing the action or level and / or identifying any appropriate attributes associated with the applicable resources and / or customers. As a simple example, the VEPO orchestration service 302 may identify that if the action is taken to shut down all idle hosts, then X idle hosts among the (one or more) hosts 324 will be affected, and these (one or more) hosts are associated with a specific customer or a specific number of customers, and / or the (one or more) hosts and / or customers are associated with other attributes such as category (e.g., free tier), priority (e.g., high priority), etc.

[0081] The VEPO orchestration service 302 can be configured to estimate the impact and / or actual power reduction that one or more levels may experience, at least in part, based on any suitable combination of configuration data specifying the levels and corresponding actions, as well as power data 310, account data 312, host / instance data, etc. The estimated impact can be identified for one or more response levels, one or more reduction actions corresponding to a given level, etc. (e.g., which (one or more) hosts, (one or more) instances, (one or more) workloads, (one or more) customers may be affected, what attributes are associated with the affected (one or more) hosts, (one or more) instances, (one or more) workloads, and / or customers, how many of the affected (one or more) hosts / (one or more) instances / (one or more) workloads / (one or more) customers there are, etc.). In some embodiments, the estimated impact of a given action can be aggregated with the estimated impact of all actions of a given level to identify the estimated impact of the given level.

[0082] The VEPO orchestration service 302 can be configured to determine the estimated impact and / or actual power reduction that one or more levels may experience, at least in part, based on current data, historical data, or predictive data (e.g., current values, historical values, or predictive values corresponding to power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318).

[0083] In some embodiments, the VEPO orchestration service 302 may be configured to present, recommend, or automatically select a given response level from a set of possible response levels based at least in part on any suitable combination of the estimated impact and / or estimated power reduction that may be experienced if a reduction action corresponding to a given level is implemented (e.g., carried out). Thus, in some embodiments, the power orchestration system 300 may recommend and / or select a particular level based on any suitable combination of the following: 1) determining a level that, if implemented, would likely result in a sufficient but not excessive reduction in power consumption at the resources of the physical environment (e.g., the minimum power consumption reduction amount sufficient to bring the current power consumption value below the current total power threshold), 2) determining a level that includes a set of the least stringent actions, or 3) determining the level or action with the least impact (e.g., the set of resources and / or customers that would likely be affected the fewest, having a priority or category indicating the lowest degree of collective importance). Determining the least stringent level or action, or the level or action with the least impact, may be determined independently of the estimated power consumption reduction that may be experienced by implementing that level and / or action, or determining the least stringent level / action and / or the level / action with the least impact may be determined from one or more levels that, if implemented under the given current conditions, would likely result in a sufficient reduction in power consumption such that the power consumption value drops to an aggregate value below the current total power threshold).

[0084] The power management service 304 may be configured to provide functionality for identifying power ceiling values for one or more hosts and / or instances based at least in part on the total power threshold, the budget / assigned power threshold associated with the (one or more) PDUs, and / or the current individual and / or total power consumption values. In some embodiments, the power management service 304 may be configured to identify the resources to which power ceilings are to be applied and / or the specific values of these power ceilings based at least in part on any suitable combination of current, historical, or predicted values of the power data 310, the account data 312, the environmental data 314, and the host / instance data 316. In some embodiments, the VEPO orchestration service 302 may trigger and utilize the functionality provided by the power management service 304 for identifying / modifying the budget / assigned power corresponding to one or more resources (e.g., hosts, instances, PDUs, etc.), identifying which or how many resources are applicable, and any suitable combination of the estimated power reduction values that may be experienced if a given level of (one or more) actions is implemented. In some embodiments, the power management service 304 may perform some or all of these functions.

[0085] The VEPO orchestration service 302 and / or the power management service 304 may host the user interface 308. The user interface 308 may be configured to provide any suitable application metadata corresponding to a combination of: current individual and / or power consumption values corresponding to any suitable number or type of resources (e.g., hosts, instances, workloads, etc.) or customers, any suitable attributes associated with these resources or customers (e.g., priority, category, etc.), prediction conditions (e.g., changes in predicted total power thresholds), current conditions (e.g., current total power thresholds), estimated impacts (e.g., the number, identifiers, or other attributes associated with affected host / instance / workload customers), and / or estimated power reduction (e.g., the estimated reduction expected to be experienced if an action of that level is applied), and / or the identification of the action(s) corresponding to a given level. The user interface 308 may present any suitable combination of this application metadata. The application metadata may correspond to any suitable combination of available response levels. In some embodiments, the application metadata corresponding to the current / predicted power consumption values and / or the current / predicted total power thresholds may be displayed. In some embodiments, the application metadata presented at the user interface 308 may correspond to any suitable number of levels of estimated power consumption reduction and / or estimated impact. In some embodiments, the VEPO orchestration service 302 may provide a recommendation at a particular level (e.g., via the provided application metadata corresponding to that level or the reduction action associated with that level), and may input the confirmation / or rejection of the recommended level via an optional option at the user interface 308. If confirmed, the VEPO orchestration service 302 may be configured to perform operations to effectuate the implementation of the confirmed level and its corresponding action. In some embodiments, the user interface 308 may present the application metadata of any suitable number of response levels and / or corresponding reduction actions, and may provide one or more options at the user interface 308 to enable the user to select a particular level and / or action.

[0086] Receiving user input that indicates a selection of a particular level and / or action or that confirms a selection of a particular level and / or action can cause the VEPO orchestration service 302 to perform operations to effectuate the implementation of the confirmed level and its corresponding action. Effectuating or implementing the confirmed level and / or its corresponding action can include instructing any suitable components of the power orchestration system (e.g., the power management service 304, the compute service 306) to effectuate any suitable portion of the level and / or the corresponding (one or more) actions. Instructing one or more components can include providing any suitable data, such as any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and operation data 318, to the instructed (one or more) components. In some embodiments, the instructed components can be configured to effectuate the (one or more) actions or level based on the data provided by the VEPO orchestration service 302, or the instructed components can obtain any suitable portion of the data from a location storing the power data 310, account data 312, environmental data 314, host / instance data 316, or operation data 318.

[0087] As a non-limiting example, if the (one or more) actions for the level of reduction or impact to be estimated include power capping, then the power management service 304 can be configured to estimate a power ceiling, estimate individual or total power consumption reduction, and / or effectuate the estimated power ceiling to achieve the estimated individual / total power consumption reduction. In some embodiments, the power ceiling can be effectuated directly by the power management service 304, or by instructing the compute service 306. Other actions, such as migrating instances or workloads, pausing instances, workloads, or hosts, shutting down hosts, or preventing (at least for a period of time) future assignment or startup of instances and / or workloads, can be identified by the VEPO orchestration service 302 and / or the compute service 306 and effectuated by the VEPO orchestration service 302 through the functionality provided by the compute service 306. In some embodiments, the compute service 306 can be configured to identify potentially affected hosts / instances / workloads / customers and provide such information to the VEPO orchestration service 302 and / or the power management service 304.

[0088] The compute service 306 can be configured to communicate with the baseboard management controller (BMC) / integrated lights-out manager (ILOM) 322 of the rack PDU 320. ILOM is an example of a BMC for illustrative purposes. In some embodiments, the BMC / ILOM 322 can be a dedicated service processor that monitors the physical state of the host machine, which in this case is the rack PDU 320. Similarly, the (one or more) hosts 324 (e.g., the (one or more) hosts of rack 319, to which the rack PDU 320 is coupled in accordance with Figure 5The power distribution hierarchy under discussion that manages its power can include a BMC / ILOM 328, which can be a dedicated service processor, that monitors the physical state of a host, which in this example is one of the host(s) 324. The BMC / ILOM 322 and the BMC / ILOM 328 can be configured to manage and / or enforce, etc., the budgeted power and / or power capping on the device on which the BMC / ILOM 328 operates or on downstream devices. For example, the BMC / ILOM 322 can receive a power capping value for use by the BMC / ILOM 328 to constrain the power consumption of a given host. The BMC / ILOM 322 can distribute the power capping to the corresponding BMC / ILOM 328 associated with that power capping. In some embodiments, the BMC / ILOM 322 can distribute the (one or more) power cappings stored at the applicable host(s), but the enforcement of these power cappings is not carried out until the BMC / ILOM 322 provides further instructions to the BMC / ILOM 328. The BMC / ILOM 328 can enforce the power capping at the host level and / or instance level and / or at the workload level of one or more workloads (not shown) executed at the instance.

[0089] The BMC / ILOM 322 to the BMC / ILOM 328 can be used to instruct a device (e.g., the rack PDU 320 and / or the host(s) 324) to pause operation or shut down the host / instance / workload. In some embodiments, the BMC / ILOM 322 to the BMC / ILOM 328 can be instructed to resume the operation of the host / instance / workload. If shut down, then the device (e.g., via its BMC / ILOM) can be instructed to start from the shut-down state (e.g., by the compute service 306 and / or by an upstream device such as its PDU). This can include the compute service 306 sending the (one or more) instructions to the BMC / ILOM 322 to the BMC / ILOM 328 to pause and / or shut down the host / instance / workload.

[0090] In some embodiments, the compute service 306 can perform any appropriate operations to migrate a workload from one instance to another (e.g., migrate to an instance on the same host, migrate to a different instance operating on a different host, etc.), migrate an instance from one host to another (e.g., hosts in the same rack, hosts in different racks, etc.), migrate the instance / workload back to the host that initially hosted the previously migrated instance / workload, etc. In some embodiments, the compute service 306 can be configured to ensure that a host / instance / workload is not assigned to a host and / or instance to which a level and / or action applicable to the implementation or ongoing implementation pertains.

[0091] Any suitable combination of (one or more) devices (e.g., (one or more) hosts 324, rack PDU 320) may include one or more power controllers (e.g., BMC / ILOM 328, BMC / ILOM 322, respectively). A power controller may be any suitable hardware or software component operating at a device (e.g., a server) that is configured to monitor and / or manage power consumption at the device. A power controller may individually monitor the power consumption of the corresponding device on which it operates. Each power controller may be configured to enforce a power capping limit to constrain the power consumption at the corresponding device. Enforcing a power capping limit may include monitoring the power consumption at the device, determining whether to constrain (e.g., limit, cap, etc.) the power consumption at the device (e.g., at least in part based on a comparison of the device's current power consumption to a stored power capping limit), and limiting / capping the power consumption at the device (e.g., using dynamic voltage and / or frequency scaling of the (one or more) processors and / or memory of the device to throttle the power consumption at the device). Any suitable operation associated with power capping may be implemented by the power controller.

[0092] In some embodiments, a power controller (e.g., BMC / ILOM 322) may communicate with another power controller (e.g., BMC / ILOM 328) corresponding to one of the (one or more) hosts 324 via a direct connection (e.g., via a cable) and / or via a (one or more) network. A power controller (e.g., BMC / ILOM 328) may provide power consumption data indicative of the current power consumption of the device (e.g., cumulative power consumption over a period of time, current power consumption rate, etc.). The power consumption data may be provided to a power controller (e.g., BMC / ILOM 322) at any suitable frequency, periodically, or according to a predefined schedule or event (e.g., upon breaching a predefined consumption threshold, upon a threshold amount change in the consumption rate, upon determining that one or more predefined conditions are met, upon determining the thermal properties of the device, etc.).

[0093] In some embodiments, a power controller (e.g., BMC / ILOM 328) may receive a power capping value (also referred to as a "power cap") from a power controller (e.g., BMC / ILOM 322). In some embodiments, additional data may be provided with the power capping value. By way of example, an indicator may be included in the power capping value that indicates whether the power capping value is to be applied immediately. In some embodiments, it may be default to immediately apply / enforce the received power cap. In other embodiments, it may be default not to immediately apply / enforce the received power cap.

[0094] When applying / enforcing a power limit, a power controller (e.g., BMC / ILOM 328) can monitor the power consumption at a device. This can include using metering devices or software that are configured to identify / calculate power consumption data for the device (e.g., cumulative power consumption over a period of time, current power consumption rate, change in current power consumption rate within a time window, etc.). As part of applying / enforcing a power limit (also referred to as "power capping"), the power controller (e.g., BMC / ILOM 328) can determine whether to constrain (e.g., limit, cap, etc.) the power consumption at the device (e.g., at least in part based on comparing the device's current consumption rate to a stored power limit value). When constraining the power consumption at the device, the power controller (e.g., BMC / ILOM 328) can limit / cap the power consumption at the device (e.g., using dynamic voltage and frequency scaling of (one or more) processors and / or memory to throttle the power consumption at the device). In some embodiments, when the device's current consumption data (e.g., cumulative consumption, consumption rate within a time window, etc.) approaches the enforced power limit value (e.g., breaches a threshold less than the power limit value), the power controller (e.g., BMC / ILOM 328) can execute instructions to limit / cap the device's power consumption. When constraining / capping the power consumption at the device (also referred to as "throttling"), the power controller (e.g., BMC / ILOM 328) ensures that the device's power consumption remains below the power consumption indicated by the power limit value. The power controller (e.g., BMC / ILOM 328) can be configured to generally constrain the power at a host, or to constrain the power at a host based on instances and / or workloads for which a power limit may apply.

[0095] In some embodiments, the power controller for (one or more) hosts 324 can be configured to allow (one or more) hosts 324 to run unconstrained (e.g., not capped based on power consumption and a power limit) until the power controller (e.g., BMC / ILOM 328) is instructed to apply / enforce a power limit. In some embodiments, the power controller (e.g., BMC / ILOM 328) can receive a power limit value that may or may not later be instructed to be enforced, but when received, the power limit value can be stored in memory and not used for power management at the device. Thus, in some embodiments, the power controller (e.g., BMC / ILOM 328) can avoid starting a power capping operation (e.g., the comparison and determination described above) until it is instructed to do so (e.g., via an indicator provided by power controller 448). The indication to start enforcing a power limit can be received together with the power limit value, or can be received as a separate communication from power controller 448.

[0096] According to the power distribution hierarchy 500, each power controller of one or more PDUs (e.g., BMC / ILOM 322) can be configured to distribute, manage, and monitor power for any suitable number of (one or more) devices (e.g., (one or more) hosts 324 of rack 319). In some embodiments, the power controller (e.g., BMC / ILOM 322) can be a computing agent or program installed at a given PDU (e.g., Figure 3 the rack PDU 320). The power controller (e.g., BMC / ILOM 322) can be configured to obtain power consumption data from (one or more) hosts 324 of rack 319 corresponding to the devices (e.g., the devices for which the given PDU is configured to distribute, manage, or monitor power) associated therewith. The power controller (e.g., BMC / ILOM 322) can receive the consumption data according to a predefined frequency, periodicity, or schedule implemented by the power controller (e.g., BMC / ILOM 328), and / or the power controller (e.g., BMC / ILOM 322) can receive the consumption data in response to a request for the consumption data to the power controller (e.g., BMC / ILOM 328). The power controller (e.g., BMC / ILOM 322) can be configured to request the consumption data from the power controller (e.g., BMC / ILOM 328) according to a predefined frequency, periodicity, or schedule implemented by the power controller (e.g., BMC / ILOM 322).

[0097] The power controller (e.g., BMC / ILOM 322) can transmit the received consumption data (directly or through the computing service 306) to the power management service 402 at any suitable time according to any suitable frequency, periodicity, or schedule, or due to the satisfaction of one or more predefined conditions (e.g., the individual / cumulative / collective consumption rate of (one or more) hosts 324 changes by more than a threshold). In some embodiments, the power controller (e.g., BMC / ILOM 322) can aggregate the power consumption data received from (one or more) hosts 324 (e.g., via the power controller (e.g., BMC / ILOM328) of each device) before transmitting the aggregated consumption data to the power management service 402. In some embodiments, the consumption data can be aggregated by host, instance type / category / priority, workload type / category / priority, or customer.

[0098] A power controller (e.g., BMC / ILOM 322) can receive one or more power ceiling values corresponding to one or more hosts 324 at any appropriate time (e.g., from the execution manager 410 of the power management service 402). These one or more power ceiling values can be calculated by the power management service 402 (e.g., via the constraint identification manager 408). In some embodiments, the power controller (e.g., BMC / ILOM 322) can receive a timing value, where the power ceiling value indicates the duration to be used for a timer. The power controller (e.g., BMC / ILOM 322) can be configured to generate and initiate a timer having an associated duration / period corresponding to the timing value. When the timer expires, indicating that the period corresponding to the time period has passed, the power controller (e.g., BMC / ILOM 322) can transmit data to one or more hosts, including an indicator or other appropriate data indicating that the one or more hosts continue power capping using the power ceiling values previously stored at each device. In some embodiments, the power ceiling value can be provided with an indicator. In other embodiments, the power ceiling value can be provided by the power controller (e.g., BMC / ILOM 322) to the power controller of the device (e.g., BMC / ILOM 328) immediately after receiving the one or more power ceiling values from the power management service 402. Thus, in some embodiments, the power ceiling value can be transmitted by the power controller (e.g., BMC / ILOM 322) to the power controller (e.g., BMC / ILOM 328), stored in memory, but not enforced by the power controller (e.g., BMC / ILOM 322) until the power controller (e.g., BMC / ILOM 328) receives subsequent data from the power controller (e.g., BMC / ILOM 322) indicating that the power controller (e.g., BMC / ILOM 328) should start power capping.

[0099] The VEPO orchestration service 302, the power management service 304, the computing service 306, the rack PDU 320, one or more hosts 324 (collectively referred to as " Figure 3The device(s) can communicate via one or more wired or wireless networks (e.g., network(s) 808). The storage device storing power data 310, account data 312, environmental data 314, host / instance data 316, and operation data 318 can also communicate with the VEPO orchestration service 302, power management service 304, computing service 306, rack PDU 320, host(s) 324 via one or more wired or wireless connections (e.g., via one or more networks). In some embodiments, the network(s) can include any one or combination of various different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks.

[0100] Figure 3 The device(s) can be any suitable type of computing device, such as but not limited to server devices, networking devices, or any suitable device within a data center. In some embodiments, some devices (e.g., rack PDU 320 and host(s) 324) are arranged in a power distribution hierarchy (such as the power distribution hierarchy 500 discussed in conjunction with Figure 5 In some embodiments, host(s) 324 can correspond to and be represented by level 1 nodes of the power distribution hierarchy 500. The rack PDU 320 ("rack leader") can correspond to and be represented by a higher-level node (e.g., level 2 node) of the power distribution hierarchy.

[0101] Figure 3 Each of the device(s) can include at least one memory. The processor(s) can each be suitably implemented in hardware, computer-executable instructions, firmware, or a combination thereof. Figure 3 The computer-executable instruction or firmware implementation of the processor(s) of the device(s) can include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0102] Figure 3 The memory of the device(s) can store program instructions that can be loaded and executed on the corresponding processor(s) of a given device, as well as data generated during the execution of these programs. Depending on the configuration and type of the user computing device, the memory can be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). Figure 3The (one or more) devices may also include additional removable and / or non-removable storage devices, including but not limited to magnetic storage devices, optical disks, and / or tape storage devices. The disk drive and its associated computer-readable medium can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some embodiments, the memory may separately include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM.

[0103] Turning to the contents of the more detailed memory, the memory may include an operating system, one or more data repositories, and one or more applications, modules, instances, workloads, or services, such as the VEPO orchestration service 302, the power management service 304, the computing service 306, etc.

[0104] Figure 3 The (one or more) devices may include (one or more) communication connections that allow the (one or more) devices to communicate with each other via (one or more) networks (not shown). Figure 3 The (one or more) devices may also include I / O devices, such as keyboards, mice, pens, voice input devices, touch input devices, displays, speakers, printers, etc.

[0105] Figure 4 Illustrated is an example architecture 400 of a power management service 402 ( Figure 3 an example of the power management service 304) according to at least one embodiment. The power management service may include an input / output processing manager 404, which may be configured to receive and / or transmit any appropriate data between the power management service 402 and Figure 3 any other appropriate devices. As depicted, the power management service 402 may include a consumption monitoring manager 406, a constraint identification manager 408, and an enforcement manager 410, but more or fewer computing components or subroutines may be similarly utilized.

[0106] The consumption monitoring manager 406 may be configured to determine current individual and / or total power consumption values (e.g., corresponding to Figure 3 the (one or more) hosts 324, each host being Figure 1(example(s) of one or more of the servers 104). The constraint recognition manager 408 may be configured to perform any suitable functions corresponding to identifying one or more power capping values for a host, instance, or workload (e.g., (one or more of) the hosts 324). According to at least one embodiment, the enforcement manager 410 may be configured to enforce and / or trigger the enforcement of a power cap at any suitable device. Allocating power may refer to the process of assigning a budgeted amount of electrical power (e.g., an expected load / power draw) to any suitable component. A power cap may be an example of a budgeted amount of electrical power.

[0107] In some embodiments, the functions described in connection with the power management service 402 may be performed by one or more virtual machines implemented in a hosted computing environment (e.g., on (one or more of) the hosts 324, corresponding to Figure 1 any suitable number of the servers 104). The hosted computing environment may include one or more rapidly provisioned and released computing resources, which may include computing, networking, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. Multiple example cloud computing environments are provided and discussed in more detail below. Figures 16 - 19 Provide and discuss multiple example cloud computing environments in more detail.

[0108] The power management service 450 (e.g., the consumption monitoring manager 406) may be configured to receive / obtain consumption data from (one or more of) the PDUs (e.g., from a rack PDU 320 of a power controller (such as Figure 3 the BMC / ILOM 322). As described above, the consumption data may be aggregated or accumulated for one or more devices. As a non-limiting example, an instance of the consumption data received by the power management service 402 may include the cumulative / aggregated consumption data for each device associated with a given PDU (e.g., managed by a given PDU). For example, the PDU providing the consumption data may be a rack PDU (e.g., the rack PDU 320), and the consumption data provided by that PDU may include the cumulative / aggregated and / or individual consumption data values for each device in the rack (e.g., a host in (one or more of) the hosts 324 associated with a given rack 319). In some embodiments, the cumulative / aggregated consumption data values may additionally include the power consumption of the PDU. The power management service 402 may access the individual and / or aggregated or cumulative power consumption data for individual devices or all devices associated with that instance of the consumption data from the consumption data. The power management service 402 may be configured to be at least partially based on representing Figure 5Perform any suitable operations (such as aggregating consumption data, calculating power ceiling values, calculating timing values, determining budget power values, identifying whether power capping is required (such as when the aggregated consumption breaches / may breach the budget power, etc.)) on the data of the power distribution hierarchy 500 and / or any suitable configuration data indicating the arrangement of components within the data center (e.g., indicating which devices distribute power to which devices). For example, the configuration data may include data representing the power distribution hierarchy 500.

[0109] Figure 5 Illustrates an example of a power distribution hierarchy 500 corresponding to the arrangement of components of Figure 2 The power distribution hierarchy 500 can represent the arrangement of any suitable number of components of a power system (such as the power distribution infrastructure components discussed in connection with the power distribution infrastructure 200 of Figure 2 The power distribution hierarchy 500 can include any suitable number of nodes organized according to any suitable number of levels. Each level may include one or more nodes. The root level (e.g., level 5) may include a single root node of the power distribution hierarchy 500. Each node of the power distribution hierarchy 500 can represent a corresponding component of the power distribution infrastructure 200. A set of one or more nodes at a given level can descend from a particular node corresponding to a higher level of the power distribution hierarchy 500. A set of lower-level components represented by nodes at a lower level (e.g., level 1) can receive power distributed by a higher-level component represented by a node at level 2 (e.g., a component upstream of the lower-level components), which in turn receives power from a higher-level component represented by a node at level 3, which receives power from a higher-level component represented by a node at level 4. In some embodiments, each component of the power system receives power initially distributed by a component corresponding to a node at level 5 (e.g., the highest level of the power distribution hierarchy 500) (e.g., node 502, the root node). Node 502 can receive power from a utility power source (e.g., a local power utility system).

[0110] As depicted, the power distribution hierarchy 500 includes node 502 at level 5. In some embodiments, node 502 can represent Figure 2The uninterruptible power supply of one or more UPSs 202. Components corresponding to node 502 can distribute / supply power to components corresponding to node 504 at level 4. A component (e.g., a lower-level component) that receives power from a higher-level component (a component represented by a node in power distribution hierarchy 500 that is at a higher level than the level of the node representing the lower-level component) can be considered subordinate to the higher-level component. In some embodiments, node 304 can represent a component subordinate to the UPS represented by node 502, such as Figure 2 One of the intermediate PDUs 204 (e.g., a distribution panel). The component represented by node 504 can be configured to distribute / supply power to the components represented by nodes 506 and 508, respectively.

[0111] Nodes 506 and 508 at level 3 can each represent Figure 2 The corresponding components (e.g., the corresponding one or more row PDUs). The component corresponding to node 506 (e.g., row PDU 206, busbar) can distribute / supply power to the component corresponding to node 510 (e.g., Figure 2 The rack PDU 208A) and the component corresponding to node 512 (e.g., Figure 2 The rack PDU 208N). The component corresponding to node 510 (e.g., rack PDU 208A) can distribute / supply power to the components corresponding to nodes 514 and 516 at level 1 (representing Figure 2 The servers 212A and 212B), and these components can be monitored / managed by components of servers 212A and 212B, such as power controllers 216A and 216B. The components corresponding to nodes 514 and 516 can be arranged in the same rack.

[0112] The component corresponding to node 512 (e.g., rack PDU 208A) can distribute / supply power to the component corresponding to node 518 at level 1 (e.g., Figure 2 The server 214C including power controller 216C). Nodes 514, 516, and 518 at level 1 can be arranged in the same row (e.g., Figure 2 Row 210) and / or be associated with the same row.

[0113] Returning to node 508 at level 3, node 508 (e.g., a different row PDU, such as a remote power panel) can distribute / supply power to the component corresponding to node 510 (e.g., Figure 2 The rack PDU 208A) and the component corresponding to node 512 (e.g., Figure 2The rack PDU 208N distributes / supplies power. Components corresponding to node 520 (e.g., the rack PDU) can distribute / supply power to components corresponding to nodes 522 and 526 at level 1 (e.g., corresponding to respective servers including respective power controllers). Components corresponding to nodes 524 and 526 can be arranged in the same rack. Components corresponding to node 522 (e.g., another rack PDU) can distribute / supply power to components corresponding to nodes 528 and 530 at level 1 (e.g., corresponding to respective servers including respective power controllers). Components corresponding to nodes 528 and 530 can be arranged in the same rack.

[0114] The specific number of components (e.g., corresponding to level 1 nodes) that receive power distributed from a higher-level component (e.g., corresponding to a node at level 2) may be different from Figure 5 the number shown. The specific number of levels within the power distribution hierarchy may vary depending on the specific arrangement of components used in a given data center. It is contemplated that each non-root level (e.g., in the Figure 5 example, levels 1 - 4) can include a different number of nodes, which represent a different number of components than the number of nodes depicted for each non-root level in Figure 5 . The nodes can be arranged in a configuration different from but similar to the configuration depicted in Figure 5 .

[0115] Return Figure 4 , the power management service 402 (e.g., the constraint identification manager 408) can calculate aggregated / cumulative consumption data to identify the aggregated / cumulative consumption data for one or more devices at a higher level of the power distribution hierarchy 500. For example, the power management service 402 (e.g., the constraint identification manager 408) can utilize the consumption data provided by one or more rack PDUs and calculate the aggregated / cumulative consumption data for row devices. For example, the consumption data corresponds to the devices represented by the level 1 nodes of Figure 5 which share a common set of nodes up to level 3 of the hierarchy. In this case, the common level 3 nodes represent row PDUs, such as busbars. In some embodiments, the consumption data indicating the power consumption of one or more rack PDUs (e.g., an example of a component corresponding to level 2 of the power distribution hierarchy 500) can be included in the consumption data provided to the power management service 402. The consumption data provided by the rack PDUs can be provided according to any suitable frequency, periodicity, or schedule, or in response to a request transmitted by the power management service 402. The power management service 402 can obtain data related to Figure 3Consumption data associated with any suitable number (one or more) of devices (such as (one or more) hosts 324 and / or racks of (one or more) devices corresponding to one or more racks (e.g., rack 319)) can be aggregated or calculated for any suitable number of components represented by any suitable nodes and / or levels of the power distribution hierarchy 500.

[0116] The power management service 402 (e.g., the constraint identification manager 408) can be configured to calculate one or more power caps (e.g., power cap values for one or more servers to which power is distributed by the higher-level device (e.g., the row-level device in this example)) at least in part based on the power values (e.g., budgeted power amounts) assigned to higher-level components. The higher-level components can correspond to any suitable level (e.g., levels 2 - 5) of the power distribution hierarchy 500 other than the lowest level (e.g., level 1). By way of example, the power management service 402 can calculate the power cap value for a server to which power is distributed by a given row device (e.g., a busbar). These calculations can be based on consumption data provided by the rack PDU to which power is distributed by the row device. In some embodiments, the power management service 402 can store the consumption data for subsequent use. The power management service 402 can utilize historical consumption data when calculating these power cap values. In some embodiments, the power management service 402 can obtain, utilize, and / or train one or more machine learning models to identify specific power cap values for one or more hosts 324 (e.g., components corresponding to level 1 of the power distribution hierarchy) from historical consumption data. These techniques will be discussed in more detail with respect to Figure 12 be discussed in more detail.

[0117] The power management service 402 (e.g., the enforcement manager 410) can calculate the timing value of a timer (e.g., a timer that can be initiated and managed by (one or more) PDUs (such as rack PDU 320)). The timing value can be calculated based on at least any suitable combination of the following: the change in the power consumption rate of the higher-level component, the direction of change (increase / decrease) in the power consumption of the higher-level component, the power spike tolerance of the higher-level component and / or the entire system 300, or the enforcement time associated with a lower-level device (e.g., the time between when each of (one or more) devices (e.g., (one or more) hosts 324) is instructed to enforce a power cap value and the time when each of (one or more) devices (e.g., (one or more) hosts 324) will actively enforce the cap (e.g., the time to first constrain / throttle the power consumption at that device, or at least determine whether to constrain / throttle)).

[0118] The power management service 402 (e.g., enforcement manager 410) can transmit the calculated timing value to the (one or more) PDUs 402 at any appropriate time. In some embodiments, the power management service 450 can initially determine the power ceiling for a given higher-level component (e.g., row component) without considering the consumption that occurs relative to other components at the same level (e.g., other row components). In some embodiments, when the timer is initialized or elapses, or at any appropriate time, the power management service 402 can process the consumption data corresponding to other components at the same level to determine whether it is more desirable to perform power capping on one or more downstream components of these same-level components. In some embodiments, the power management service 402 can utilize the priorities associated with the (one or more) devices and / or the workloads running on the (one or more) devices to determine the power ceiling value (e.g., at least partially based on Figure 3 the priority identified by the host / instance data 316). The power management service 402 can be configured to preferentially perform power capping on the (one or more) low-priority devices / (one or more) workloads while leaving the (one or more) devices / (one or more) workloads with higher priority values unconstrained. In some embodiments, the power management service 402 can be configured to preferentially perform power capping on a group of the highest-consuming devices (e.g., across one row, across multiple rows, etc.) while leaving the lower-consuming devices to operate unconstrained. In some embodiments, if the priority associated with a device and / or workload is high (or higher than the priorities associated with other devices and / or workloads), then the specific consumption of that device can be left unconstrained even if the device is included in the group of the highest-consuming devices. Thus, the power management service 402 can be configured to prioritize the priority of the device / workload over the power consumption at that particular device.

[0119] In some embodiments, the power ceiling values can be initially determined by the power management service 402 (e.g., by the constraint identification manager) for a given row. These power ceiling values can be provided to the power controller of the rack PDU (e.g., BMC / ILOM 320, or another agent or component of the rack PDU 320), which in turn can distribute the power ceiling values to the power controllers (e.g., BMC / ILOM 328) at the host(s) 324 for storage. The power management service 402 (e.g., the constraint identification manager 408) can process the consumption data of other row devices on the same row to determine the power ceiling values for the devices corresponding to different rows. This can be advantageous because the devices managed by another row may not consume their budgeted power, leaving some unutilized power at the devices at that row level. In some embodiments, the power management service 402 (e.g., the constraint identification manager 408) can be configured to determine whether it is more beneficial to cap the power of the devices of one row while allowing at least some of the devices of another row to operate without constraint. The benefits of determining a set of power ceiling values can be at least partially based on minimizing the estimated impact (e.g., the number of devices to be power-capped), minimizing the priority values associated with the devices to be power-capped, maximizing the number of devices associated with a particular high-priority value that will not be power-capped, etc. The power management service 402 can utilize a predefined protocol (e.g., a set of rules) to determine whether enforcing the power ceiling values it has sent to one PDU is more or less advantageous / beneficial than the different power ceiling values it has identified based on processing the consumption data of multiple PDUs associated with one or more other rows.

[0120] The power management service 402 (e.g., the enforcement manager 410) can be configured to formulate the enforcement of a set of power ceiling values determined to be the most beneficial (based on a predefined set of rules) to ensure that one or more circuit breakers within the data center do not trip and / or the total power threshold is enforced. If the current / total power consumption level exceeds the current total power threshold, then the power management service 402 can be configured to perform or trigger an action to bring the current / total power consumption level to a value below the current total power threshold.

[0121] In some embodiments, the monitoring, identification, and / or constraint based on the power ceiling values can be determined directly by the power management service 402 based on the power data 310, account data 312, environmental data 314, host / instance data 316, operation data 318, etc.

[0122] In some embodiments, the power management service 402 may be configured to identify and / or enforce a power ceiling value based at least in part on instructions and / or constraints or other applicable data provided by the VEPO orchestration service 302. As a non-limiting example, the VEPO orchestration service 302 may filter aggregated data of any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318 based on a reduction action associated with a level (e.g., "reduce the power ceiling of all low-priority hosts by 10%") to determine the estimated impact of implementing that action, including the number and / or subset of resources (e.g., hosts, instances, workloads, customers) to which the action would be applied if the action were implemented. For example, the VEPO orchestration service 302 may identify the number and / or identifiers corresponding to all low-priority hosts from the aggregated data of any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the identifier and / or instruction (10% power ceiling reduction) may be provided to the power management service 402, which receives it through the input / output processing module 404 and is implemented by the enforcement manager 410. As another non-limiting example, any suitable portion of the metadata associated with a reduction action (e.g., "10% power ceiling reduction for all low-priority hosts") may be provided to the power management service 402, and the constraint identification manager 408 may be configured to identify specific hosts / instances / workloads from the metadata associated with the reduction action. In other words, any suitable combination of the VEPO orchestration service 302 and / or the power management service 402 (e.g., power management service 304) may be configured to identify the estimated impact of a given action. In some embodiments, the power management service 402 (e.g., the constraint identification manager 408) may be configured to identify an estimated power reduction value corresponding to the individual or total power consumption of one or more devices (e.g., (one or more) hosts 324), assuming hosts / instances / workloads / customers that are expected to be affected if the action (or level) is implemented.

[0123] If it is determined that a previously provided set of power caps to the power controller (e.g., BMC / ILOM 322 and / or BMC / ILOM 322) is most beneficial, then the power management service 402 can transmit data to the power controller to cause the power controller to transmit an indicator to the (one or more) devices to start power capping based on the previously distributed power caps. Alternatively, if the power management service 402 determines that a new set of power caps is more (or most) beneficial from a power management perspective, then it can transmit data to the power controller (e.g., BMC / ILOM 322) to cancel the timer. In some embodiments, canceling the timer may cause the previously distributed power caps to be deleted by an instruction from the power controller (e.g., transmitted in response to canceling the timer), or these power caps may default to timing out (e.g., according to a predefined time period). The power management service 402 can transmit the new power caps to the appropriate PDU (which may include or exclude the same PDU to which the previous set of power caps was transmitted), and indicate that the corresponding devices should immediately enforce these power caps. These PDUs can transmit these power caps along with instructions to the receiving devices to immediately start the power capping operation (e.g., including monitoring the consumption related to the power cap value, determining whether the cap is based on the power cap value and the current consumption of the device, and capping or unconstraining the operation of the device based on the determination).

[0124] In some embodiments, if the timer expires at the original PDU (e.g., the time period corresponding to the timing value has passed) and no cancellation has been received from the power management system 402, then the power controller (e.g., BMC / ILOM 322) can automatically (e.g., via BMC / ILOM 328) indicate to the (one or more) devices to start the power capping operation using the stored power cap values. This technique ensures fail-safe when the power management service 402 fails to indicate the PDU or cancel the timer for any reason.

[0125] The above technique enables more lower-level devices to operate unconstrained by minimizing the number and / or frequency of capping the power consumption at these devices. Additionally, the peaks of downstream devices in a given device hierarchy can be allowed to exceed the budgeted power of the device while still ensuring that the maximum power capacity of the device (values above the budgeted value) is not exceeded. This allows the consumption of lower-level devices to cause the consumption levels at higher-level devices to operate within the power buffer that is not used in a conventional system. The described system and techniques allow for more efficient utilization of the power distribution components of system 400 and reduce waste while ensuring that power failures are avoided.

[0126] In some embodiments, the power management service 402 may select a set of power ceiling values determined to be most advantageous (e.g., based at least in part on any suitable combination such as maximum expected reduction and / or minimum estimated impact, etc.), or may receive from the VEPO orchestration service 302 and / or implement a specific set of power ceiling values provided by the power management service 402 (previously identified by the power management service 402 and provided to the VEPO orchestration service 302). In some embodiments, the identified specific power ceiling values may be identified based at least in part on maximizing power consumption reduction, identifying a reduction associated with a sufficient power ceiling, which reduction associated with the sufficient power ceiling is identified alone or in combination with other operations corresponding to reduction actions or levels associated with reducing the total power consumption value of the (one or more) devices below the current total power threshold.

[0127] Figure 6 is a flowchart illustrating an example method 600 for managing excessive power consumption according to at least one embodiment. Method 600 may include more or fewer operations than those depicted or described with respect to Figure 6 These operations may be performed in any suitable order. Any suitable portion of the operations described in connection with the power management service 606 may be additionally or alternatively performed by Figure 3 the VEPO orchestration service 302 and / or the compute service 306.

[0128] Method 600 may begin at 610, where the PDU 604 may receive and / or obtain consumption data from the (one or more) devices 602 (e.g., by request). The (one or more) devices 602 may be Figure 3 an example of the (one or more) hosts 324 (e.g., multiple servers), and may be arranged within a common server rack (e.g., rack 319). The PDU 604 is Figure 2 an example of the rack PDU 208A and / or Figure 3 the rack PDU 320. Each of the (one or more) devices 602 may be a device to which the PDU 604 distributes power. The (one or more) PDUs 608 may be an example of other rack PDUs associated with the same row as the PDU 604, and thus from the same row PDU as the PDU 604 (e.g., Figure 2 the row PDU 206, corresponding to Figure 5Node 506) receives power. The consumption received from the PDU(s) 608 may correspond to the power consumption of the device(s) 612. In some embodiments, at least some portions of the PDU(s) 608 and the device(s) 612 correspond to a different row than the row corresponding to the device(s) 602. In some embodiments, any suitable number of consumption data instances received by the PDU 604 (referred to as "device consumption data") may be associated with a single device among the device(s) 602. The same may be true for the device consumption data received by the PDU(s) 608. In some embodiments, the PDU 604 and the PDU(s) 608 may aggregate and / or perform calculations based on the received device consumption data instances to generate rack consumption data corresponding to each respective PDU. For example, the PDU 605 may generate rack consumption data from the device consumption data provided by the device(s) 602. The rack consumption data may include the device consumption data instances received by the PDU 604 at 610. Similarly, the rack consumption data provided by the PDU(s) 608 may include the device consumption instances received from the device(s) 612. The rack consumption data generated by the PDU 604 and / or the PDU(s) 608 may be generated at least in part based on the power consumption of the PDU 604 and / or the PDU(s) 608, respectively.

[0129] At 614, the power management service 606 ( Figure 4 the power management service 402 of Figure 3 an example of the power management service 304 of Figure 2 an example of the PDU(s) 208 of Figure 3 an example of the BMC / ILOM 328 of

[0130] At 616, the power management service 606 may be at least in part based on another PDU (e.g., Figure 5Line-level devices not depicted therein, such as busbars) associated maximum and / or budgeted power amounts and consumption data received from PDU 604 and / or (one or more) PDU 608 to determine whether to calculate the power ceiling value of (one or more) devices 602. In some embodiments, the power management service 606 may access from the stored data the maximum and / or budgeted power amounts associated with, for example, line-level PDUs (e.g., Figure 2 the line PDU 206 not depicted herein), or the power management service 606 may receive the maximum and / or budgeted amounts from a PDU (line-level PDU) associated with the maximum and / or budgeted power amounts at any appropriate time. The power management service 606 may aggregate the rack consumption data from PDU 604 and / or (one or more) PDU 608 to determine the cumulative power consumption value regarding the line-level devices (e.g., the total power consumption of the devices downstream of the line-level device within the previous time window, the change in the consumption rate from the perspective of the line-level device, the direction of the change in the consumption rate from the perspective of the line-level device, the time period required to start power capping at one or more devices 602 and / or 612, etc.). In some embodiments, the power management service 606 may generate additional cumulative power consumption data from at least some portion of the cumulative power consumption data value and / or using historical device consumption data (e.g., cumulative or individual consumption data). For example, the power management service 606 may calculate any appropriate combination of the following at least in part based on historical consumption data (e.g., historical device consumption data corresponding to (one or more) devices 602 and / or 512 and / or historical rack consumption data): the change in the consumption rate from the perspective of the line-level device, the direction of the change in the consumption rate from the perspective of the line-level device, etc.

[0131] Alternatively, in some embodiments, the power management service 606 may receive at 616 any appropriate data and / or instructions from the VEPO orchestration service 402 for identifying any appropriate portion of the reduction actions (e.g., power capping only for low-priority hosts) and / or any appropriate devices of any appropriate constraints and / or applicable constraints. The power management service 606 may identify a subset of (one or more) hosts for which to calculate the power ceiling (e.g., in this example, only from the low-priority host candidates). For example, the operation of power capping only for low-priority hosts may cause the power management service 606 to identify multiple power ceilings for the subset of these low-priority hosts from all low-priority hosts (e.g., a subset of (one or more) hosts 324).

[0132] If the cumulative power consumption (e.g., the power consumption corresponding to the device(s) 602 and 612) exceeds the budgeted power amount associated with the row-level PDU, then the power management service 606 can determine that a power ceiling value should be calculated. If the cumulative power consumption does not exceed the budgeted amount associated with the row-level PDU, then the power management service 606 can determine that a power ceiling should not be calculated, and method 600 can end. Alternatively, since the cumulative power consumption of the downstream devices (e.g., the device(s) 602 and 612) exceeds the budgeted amount associated with the row-level PDU, the power management service can determine that a power ceiling should be calculated and can proceed to 618.

[0133] In some embodiments, the power management service 606 can additionally or alternatively determine that the power ceiling value should be calculated at 616 based at least in part on providing historical consumption data (e.g., device consumption data and / or rack consumption data corresponding to the device(s) 602 and / or 612) as input data to one or more machine learning models. Any suitable supervised or unsupervised machine learning algorithm can be used to train the machine learning model(s) to identify, from the historical consumption data provided as input, the likelihood that the consumption corresponding to the device associated with the historical consumption data will exceed the budgeted amount (e.g., the budgeted power amount allocated to the row-level device). The machine learning model(s) can be trained using a corresponding training data set that includes instances of historical consumption data. These machine learning model(s) training is discussed in more detail. Using the above techniques, the power management service 506 can identify that the power ceiling value should be calculated based on the cumulative device / rack consumption level exceeding the budgeted power associated with the row-level device and / or based on determining from the output of the machine learning model(s) that the cumulative device / rack consumption level is likely to exceed the budgeted power associated with the row-level device. Determining that the device / rack consumption level is likely to exceed the budgeted power of the row-level device can be determined based on receiving an output from the machine learning model(s) indicating the likelihood (e.g., likely / unlikely), or by comparing an output value (e.g., percentage, confidence value, etc.) indicating the likelihood of exceeding the budgeted power of the row-level device with a predefined threshold. An output value indicating a likelihood exceeding the predefined threshold may cause the power management service 606 to determine that power capping is reasonable and that the power ceiling value should be calculated. Figure 12 The training of these machine learning model(s) is discussed in more detail. Using the above techniques, the power management service 506 can identify that the power ceiling value should be calculated based on the cumulative device / rack consumption level exceeding the budgeted power associated with the row-level device and / or based on determining from the output of the machine learning model(s) that the cumulative device / rack consumption level is likely to exceed the budgeted power associated with the row-level device. Determining that the device / rack consumption level is likely to exceed the budgeted power of the row-level device can be determined based on receiving an output from the machine learning model(s) indicating the likelihood (e.g., likely / unlikely), or by comparing an output value (e.g., percentage, confidence value, etc.) indicating the likelihood of exceeding the budgeted power of the row-level device with a predefined threshold. An output value indicating a likelihood exceeding the predefined threshold may cause the power management service 606 to determine that power capping is reasonable and that the power ceiling value should be calculated.

[0134] At 618, power management service 606 may calculate a power ceiling value for any suitable number of (one or more) devices 602 and / or (one or more) devices 612. For example, power management service 606 may utilize device consumption data and / or rack consumption data corresponding to each of (one or more) devices 602 and 612. In some embodiments, power management service 606 may determine the difference between the cumulative power consumption of a row of devices (e.g., (one or more) devices 602 and any (one or more) devices 612 in the same row) (calculated by aggregating device and / or rack consumption data corresponding to (one or more) devices 602 and (one or more) devices 612) and the budgeted power associated with the row-level devices. This difference may be used to identify the power consumption amount to be constrained at the devices associated with the row. Power management service 606 may determine a power ceiling value for any suitable combination of (one or more) devices 602 and / or devices 612 in the same row based on identifying a power ceiling value that, if enforced by the devices, would result in power consumption being reduced to a value less than the budgeted power associated with the row-level devices.

[0135] In some embodiments, power management service 606 may determine a specific power ceiling value for any suitable combination of (one or more) devices 602 and 612 corresponding to the same row based at least in part on consumption data and / or priority values associated with the devices and / or the workload associated with the power consumption of the devices. In some embodiments, these priority values may be provided as part of the device consumption data provided by the devices associated with the consumption data, or the priority values may be determined at least in part based on device type, workload type, etc. obtained from the consumption data or any other suitable source (e.g., separate data accessible to power management service 606). Using priorities associated with the devices or workloads respectively, power management service 606 may calculate the power ceiling value in a manner that favors power-capped devices associated with lower-priority workloads over power-capped devices associated with higher-priority workloads. In some embodiments, power management service 606 may calculate the power ceiling value based at least in part on power-capped devices that consume at a higher rate over other power-capped devices that consume at a lower rate (e.g., a group of highest-consuming devices). In some embodiments, power management service 606 may calculate the power ceiling value based at least in part on a combination of multiple factors, including the consumption value of each device and the priority associated with the device or the workload being performed by the device. Power management service 606 may identify power ceiling values for the devices that avoid fully capping high-consuming devices and / or high-priority devices, or that cap these devices to a lesser extent compared to devices that consume less power and / or are associated with lower priorities.

[0136] At 620, the power management service 620 can calculate a timing value corresponding to the duration of a timer initialized and managed by a PDU (e.g., PDU 604). The timing value can be calculated based at least in part on any suitable combination of the rate of change of power consumption from the perspective of the row-level device, the direction of change of power consumption from the perspective of the row-level device, or the time at which power capping is performed at each device related to the power ceiling calculated at 618. For example, the power management service 620 can determine the timing value based on determining a relatively large increase in power consumption from the perspective of the row-level device, and the timing value is greater than the timing value for a relatively small increase in power consumption from the perspective of the row-level device. Thus, the more the power consumption rate increases, the smaller the timing value (corresponding to a shorter timer), while the smaller the increase in the power consumption rate, the larger the timing value may be (corresponding to a longer timer). Similarly, when the time at which power capping is performed at each device related to the power ceiling is relatively short compared to the calculation of the first timing value, the calculated first timing value may be less than the calculated second timing value. Thus, the faster the device expecting power capping can enforce the ceiling, the shorter the timing value may be. In some embodiments, no timer is calculated for the power ceiling determined for the data provided or indicated via the VEPO orchestration service 302.

[0137] At 622, the calculated power ceiling value calculated at 618 and / or the timing value calculated at 620 can be transmitted to the PDU 604. Although not shown, the corresponding power ceiling value and / or timing value of any (one or more) devices 612 in the same row can be transmitted at 620. Although not shown, in some embodiments, the transmission at 622 can be based at least in part on an instruction received from the VEPO orchestration service 302 to implement / realize the power ceiling value calculated at 618.

[0138] At 624, when the timing value is provided at 622, the PDU 604 can use the timing value to initialize a timer having a duration corresponding to the timing value. As described above, in the case where the power management service 606 is instructed by the VEPO orchestration service 302, the timing value may not be provided at 622 or stored at 624.

[0139] At 626, the PDU 604 can transmit the power ceiling value (if received at 622) to the (one or more) devices 602. At 628, the (one or more) devices 602 can store the received power ceiling value. In some embodiments, the power ceiling value provided at 626 can include an indicator indicating that enforcement does not start, or the indicator may not be provided, and the (one or more) devices 602 can default to avoid starting power capping.

[0140] At 630, the power management service 606 can perform a higher level of analysis on device / rack consumption data received from any one or more devices 612 that correspond to rows different from the rows corresponding to the one or more devices 602. As part of this process, the power management service 606 can identify unused power associated with other row-level devices. If there is unused power, then the power management service 606 can calculate a new set of power ceiling values at least in part based on consumption data corresponding to at least one other row of devices. In some embodiments, any suitable power ceiling values identified by the power management service 606 can be provided to the VEPO orchestration service 302 at any suitable time. As described above, this new set of power ceiling values can be beneficial because devices managed by another row-level device may not consume their budgeted power, leaving unused power at that row-level device. In some embodiments, the power management service 606 can be configured to determine whether it may be more advantageous to cap the devices of another row while allowing at least some of the one or more devices 602 (and possibly some of the one or more devices 612 corresponding to the same row as the one or more devices 602) to operate without constraint. The benefits of each power capping method can be calculated at least in part based on minimizing the number of devices to be power capped, minimizing the priority values associated with the devices to be power capped, maximizing the number of devices associated with a particular high priority value that will not be power capped, and the like. The power management service 606 can utilize a predefined scheme or set of rules to determine whether enforcing the power ceiling values it has determined for a row (e.g., corresponding to the power ceiling values transmitted at 622) is more or less advantageous than different power capping values identified based on processing consumption data from multiple rows. In some embodiments, the benefits of each set of power ceiling values identified by the power management service 606 can be accessed, and the most advantageous set of power ceilings can be selected by the power management service 606 (e.g., according to a set of rules, according to user input selecting a particular response level, etc.).

[0141] The power management service 606 can be configured to enforce a set of power ceiling values that are determined to be more (or most) favorable (or the set of power ceiling values indicated by the VEPO orchestration service 302) to ensure that one or more circuit breakers within the data center do not trip and / or the total power threshold is enforced. If it is determined that a previously provided set of power ceilings to the PDU 604 is less favorable than the set of power ceiling values calculated at 630, then the power management service 606 can immediately transmit data to the PDU 604 to cause the PDU 604 to transmit an indicator to the device(s) 602 to immediately start power capping based on the previously distributed power ceilings. Alternatively, the power management service 606 can take no further action, which can allow the timer at the PDU 604 to elapse. Any suitable action performed by the power management service 606 based on the determination or identification made by the power management service 606 can alternatively be triggered by an instruction from the VEPO orchestration service 302.

[0142] If the power management service 606 determines that a new set of power ceilings is more (or most) favorable from a power management perspective, then it can transmit data to the power controller (e.g., Figure 3 the BMC / ILOM 322) to cancel the timer. The previously distributed power ceilings can be deleted by an instruction from the power controller (e.g., the BMC / ILOM 322) (e.g., transmitted in response to canceling the timer), or those power ceilings can default timeout based on a predefined time period. The power management service 606 can transmit the new power ceilings to a suitable PDU (which can include or exclude the same PDU to which the previous set of power ceilings was transmitted), indicating that the power ceilings will be immediately enforced by the corresponding devices. These PDUs can transmit these power ceilings along with the instruction to the receiving devices to immediately start the power capping operation (e.g., including monitoring the consumption with respect to the power ceiling value, determining whether to cap based on the power ceiling value and the current consumption of the device, and operating or leaving unconstrained the operation of the device based on the determination to cap).

[0143] If, from a power management perspective, the power limit value calculated at 630 is determined to be more favorable than the power limit value calculated at 618 and is ultimately stored at (one or more) devices 602 at 628, then method 600 can proceed to 632, where the power limit value calculated at 630 can be transmitted to (one or more) PDUs 608 (e.g., any (one or more) PDU 608 that manages the devices associated with the power limit value). In some embodiments, the power management service 606 can provide an indication that the power limit value transmitted at 632 will be enforced immediately. The operation described at 632 can be triggered by the power management service 606 based on a determination / identification made by the power management service 606 or in response to receiving an instruction from the VEPO orchestration service 302.

[0144] In response to receiving the power limit value and the indication at 632, the (one or more) PDUs 608 that distribute power to the devices associated with these power limit values can transmit the power limit value and the indication to the devices associated with these power limit values. At 636, the receiving devices of the (one or more) devices 612 can perform a power capping operation without delay based on receiving the indication. This includes 1) determining whether to limit / qualify the operation based on the current consumption of the device compared to the power limit value provided to the given device, and 2) limiting / qualifying or avoiding limiting / qualifying the power consumption of the device.

[0145] In some embodiments, at least in part based on identifying that the power limit value calculated at 630 is more (or most) favorable than the power limit value calculated at 618, the power management service 606 can transmit data to cancel the timer and / or the power limit value transmitted at 622. In some embodiments, the PDU 604 can be configured to cancel the timer at 640. In some embodiments, the PDU 604 can transmit at 642 such that the (one or more) devices 602 delete the data of the power limit value from the memory at 644.

[0146] At 646, in the case where a more favorable set of power ceiling values (as described above) is not found or the power management service 606 does not transmit a cancellation (as described in connection with 638 above), the PDU 604 may identify that the timer initiated at 624 has expired (e.g., the duration corresponding to the timing value provided at 622 has passed). Accordingly, the PDU 604 may transmit an indication to the device(s) 602 at 648, thereby instructing the device(s) 602 to start power capping based on the power ceiling value stored at the device(s) 602 at 628. Receiving the indication at 648 may cause the device(s) 602 to start power capping at 650 based on the power ceiling value stored at 628. In some embodiments, the VEPO orchestration service 302 may be configured to control the operations of 646 - 650, and the power management service 606 cannot perform the operations of 646 - 650 unless the VEPO orchestration service 302 instructs it to do so.

[0147] In some embodiments, if the PDU 604 identifies that the power consumption of the device(s) 602 has decreased, then the PDU 604 may be configured to cancel the timer at any appropriate time after the timer is initiated at 624. In some embodiments, this may cause the PDU 604 to transmit data to the device(s) 602 to cause the device(s) 602 to discard the previously stored power ceiling value from local memory.

[0148] In some embodiments, the power ceiling value enforced at any appropriate device may time out or may be replaced by a power ceiling value calculated by the power management service 606 at any appropriate time. In some embodiments, the power management service 606 may transmit a cancellation or replacement of the power ceiling value to any appropriate device via its corresponding rack PDU. If a cancellation of the power ceiling value is received, then the device may delete the previously stored power ceiling value, thereby allowing the device to resume operation without constraint. If a replacement power ceiling value is received, then the device may store the new power ceiling value and enforce the new power ceiling value immediately or when instructed by its rack PDU. In some embodiments, being instructed by its PDU to enforce a new power ceiling value may cause the device's device (e.g., the power controller 446) to replace the previously used power ceiling value for power capping with the new power ceiling value that the device is instructed to enforce.

[0149] Any suitable operation of method 600 can be continuously performed at any suitable time to manage over - consumption (e.g., the case where the consumption of a server corresponding to a row - level device exceeds the budgeted power of the row - level device), so as to enable more efficient use of previously unused power while avoiding power outages due to circuit breaker tripping. It should be recognized that the power management service 606 can perform similar operations for any suitable level of the power distribution hierarchy 500. Although an example of monitoring the power consumption corresponding to a row - level device (e.g., the device represented by the level 3 node of the power distribution hierarchy 500) has been provided, similar operations can be performed for higher - level devices at any suitable level (e.g., components corresponding to any level from level 2 to level 5 of the power distribution hierarchy 500). As described above, any suitable function / operation of the power management service 606 can be triggered at least in part based on the determination / identification performed by the power management service 606, or any suitable function / operation of the power management service 606 can be triggered via an instruction from the VEPO orchestration service 302.

[0150] Figure 7 Illustrates an example architecture of a VEPO orchestration service 702 ( Figure 3 an example of the VEPO orchestration service 302) according to at least one embodiment, which is configured to orchestrate the management of the power consumption of a host according to a total power threshold. The VEPO orchestration service 702 can include an input / output processing manager 704, which can be configured to receive and / or transmit any suitable data between the VEPO orchestration service 702 and Figure 3 any other suitable device. As shown, the power management service 402 can include a demand manager 710, an impact identification manager 706, and an enforcement manager 708, but more or fewer computing components or sub - routines can be similarly used.

[0151] The demand manager 710 can be configured to monitor and / or modify the total power threshold associated with a physical environment (e.g., Figure 1 the data center 102, room 110A, row 108A, etc.). Monitoring can include monitoring Figure 3Any suitable combination of power data 310, environmental data 314, host / instance data 316, and / or operation data 318. Modifying the total power threshold can be performed, and the amount by which the total power threshold is modified can be determined at least in part based on a plurality of trigger events. These trigger events can include, but are not limited to: 1) receiving a request from a government entity (e.g., a local power authority) to reduce power indefinitely or for a period of time; 2) determining current and / or predicted environmental conditions (e.g., current environmental or external temperature, predicted environmental or external temperature); 3) determining the current or predicted total power consumption corresponding to the devices in the physical environment; 4) determining the current and / or predicted operating conditions of one or more temperature control units in the physical environment. The amount by which the total power threshold can or will be modified can be determined at least in part based on a plurality of factors, including but not limited to: the difference between the environmental temperature and the external temperature, the power consumption attributed to or associated with a particular component (e.g., a particular temperature control unit), the amount or percentage specified in the request received from the government entity, or the difference between the current or predicted total power consumption value and the current or predicted total power threshold.

[0152] The impact identification manager 706 can be configured to generate (directly, or by invoking the functionality of another service or process such as the compute service 306 and / or the power management service 304) one or more instances of an estimated impact corresponding to a response level and / or a reduction action corresponding to a response level. The VEPO orchestration service 702 can obtain configuration data 712 that specifies a plurality of VEPO levels and their corresponding set of one or more reduction actions.

[0153] Figure 8 is Table 800 according to at least one embodiment, which illustrates a set of example response levels and the corresponding set of reduction actions for each response level. The data in Table 800 can be provided differently, such as by mapping or being embedded within the program code of the VEPO orchestration service 702. Each response level in the set of response levels can be associated with a corresponding set of reduction actions that can be performed on a plurality of resources (e.g., hosts, instances, workloads, etc.) in the data center, and each corresponding set of reduction actions is associated with a different reduction in the total power consumption of the data center. In some embodiments, the plurality of response levels indicate increasing severity and / or which reduction actions to perform to reduce the total power consumption of the data center.

[0154] By way of example, Table 800 includes four VEPO response levels, but any suitable number of response levels can be used. The VEPO response level 1 can be associated with one reduction action. Specifically, the reduction action specifies that all vacant and / or idle hosts and hypervisors that can be evacuated (e.g., via migration) will be shut down (e.g., powered off).

[0155] In some embodiments, the VEPO response level 1 may include stopping or prohibiting the start or assignment of instances and / or workloads to the affected hosts. In some embodiments, the VEPO response level 1 may include reduction actions, which include migrating instances and / or workloads from one host to another host, or from one instance to another instance (in the case of workload migration).

[0156] The VEPO response level 2 may be associated with multiple actions (such as any / all reduction actions discussed above in connection with the VEPO response level 1) and one or more additional reduction actions. For example, an additional reduction action corresponding to shutting down the hypervisor hosting the instances and / or workloads associated with customers of category 1 (e.g., free-tier customers, free-trial customers, etc.).

[0157] The VEPO response level 3 may be associated with multiple actions (such as any / all reduction actions associated with the VEPO response level 1 and / or 2) and one or more additional reduction actions. By way of example, the VEPO response level 3 may include additional reduction actions corresponding to shutting down the hypervisors for category 2 bare-metal instances (e.g., pay-as-you-go and / or enterprise BMs) and category 3 virtual machines (VMs) (e.g., VMs associated with pay-as-you-go and enterprise categories) (e.g., after evacuating higher-priority VMs). In some embodiments, the additional reduction actions may specify that BMs and / or VMs associated with customers and / or cloud services of priority level 1 (e.g., critical customers and cloud provider services) should be excluded from consideration.

[0158] The VEPO response level 4 may be associated with multiple actions (such as any / all reduction actions associated with the VEPO response levels 1, 2, and / or 3) and one or more additional reduction actions. By way of example, the VEPO response level 4 may include additional reduction actions corresponding to shutting down all compute hosts in the customer enclave, except those that remain in a critical state and / or are considered critical to recovery.

[0159] Figure 8 The number of response levels and / or specific reduction actions depicted is not intended to limit the scope of the present disclosure. Any suitable number of response levels may be employed, with each response level corresponding to any suitable number of reduction actions. The reduction actions may include or exclude hosts, instances, workloads, and / or customers based on any suitable attributes associated therewith. For example, based on priority, category, status, and / or based on future recovery requirements. In some embodiments, the hosts, instances, and workloads considered candidates for applying these reduction actions may be based on a limited scope described in more detail Figure 9 in more detail.

[0160] AlthoughFigure 8 The response levels depicted in Figure 8 include some overlap reduction actions, but this is not a requirement. In some embodiments, the reduction actions for each level may include lesser or greater degrees of overlap, including no overlap, resulting in completely unique actions associated with each level.

[0161] Figure 9 FIG. 900 is a schematic illustration of an example scope of components of a VEPO orchestration service (e.g., Figure 7 the VEPO orchestration service 702 of Figure 7 ) for orchestrating the impact of one or more response levels for power consumption constraints, migration, suspension, and / or power-off tasks. The schematic illustration depicts a user domain 902 and a cloud service domain 904. The user domain 902 may include computing instances operating within a customer footprint (e.g., footprint 908) that run workloads (or any suitable workloads) corresponding to one or more services 906. The instances and / or workloads corresponding to one or more services 906 may be associated with one or more users 910.

[0162] In contrast, the cloud service domain 904 may include computing instances that run workloads corresponding to one or more services 912 (e.g., control and / or data plane services, such as those discussed in connection with Figures 16 - 19 service leasing) and / or resources (such as object storage devices, virtual cloud networking, etc.).

[0163] In some embodiments, although one or more hosts 324 may host any suitable combination of instances and / or workloads corresponding to one or more services 906 associated with a user (e.g., a customer) and / or one or more services 912 and / or resources associated with a cloud provider, only the hosts, instances, and / or workloads (e.g., computing resources) corresponding to the customer domain may be considered candidates to which reduction actions may be applied. In some embodiments, reduction actions may be completely avoided (at least for some period of time) for services and / or resources associated with the cloud provider since these services and / or resources are necessary for later recovery.

[0164] Returning now Figure 7 to Figure 7 , in some embodiments, the functionality described in connection with the power management service 402 may be performed by one or more virtual machines implemented within a hosted computing environment (e.g., on one or more hosts 324, corresponding to Figure 1 any suitable number of servers 104 of Figure 1 ). The hosted computing environment may include one or more rapidly provisioned and released computing resources, which may include computing, networking, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. Below will be discussed with respect to Figures 16 - 19Provide and discuss in more detail multiple example cloud computing environments.

[0165] The VEPO orchestration service 702 can be configured to manage power consumption corresponding to hosts / instances / workloads in a physical environment through functions provided by associated functions or other systems and / or services. Managing power consumption values can include power capping, pausing workloads, instances, hosts, shutting down workloads / instances / hosts, migrating an instance from one host to another, migrating a workload from one instance and / or host to another instance and / or host, etc.

[0166] The impact identification manager 706 can be configured to utilize configuration data 712 corresponding to specifications of multiple response levels and their corresponding mitigation actions (e.g., Figure 8 table 800) to determine the estimated impact of applying a mitigation action of a given level to hosts / instances / workloads (collectively referred to as "resources") in a physical environment. The estimated impact of applying a given action and / or level can depend on the current conditions of runtime resources. In some embodiments, identifying the estimated impact of a given mitigation action can include identifying a set of resources and / or customers to which the action would apply if implemented. Thus, the estimated impact is intended to refer to the scope or applicability of a given action or level. Identifying the scope and / or applicability of a given action or level can include identifying the specific resources and / or customers that would be affected if the action or level were implemented and / or identifying any appropriate attributes associated with the applicable resources and / or customers. As a simple example, the impact identification manager 706 can identify that if the action of shutting down all idle hosts is taken, then X idle hosts among the (one or more) hosts 324 would be affected, and these (one or more) hosts are associated with a specific customer or a specific number of customers, and / or the (one or more) hosts and / or customers are associated with other attributes (such as category (e.g., free tier), priority (e.g., high priority), etc.). Mitigation actions can provide progressively more severe actions and / or progressively broader impacts. As an example of a progressively severe action, a lower level can specify an action of power capping a specific host associated with a specific attribute, while a higher level can specify shutting down the host completely. Examples of progressively broader impacts can include a lower level that affects a small number of hosts (e.g., 2, 5, 10) corresponding to a single low-priority customer, while a higher level may affect a larger number of hosts (e.g., 100) corresponding to the same or more customers, and these hosts are associated with a wider range of priorities, etc. Thus, a broader impact refers to the quantity, scope, or extent of the applicability of a given action to hosts, instances, workloads, or customers.

[0167] The impact identification manager 706 can be configured to estimate the impact that one or more levels may experience and / or estimate power reduction, at least in part, based on configuration data specifying levels and corresponding actions and any suitable combination of power data 310, account data 312, host / instance data, etc. The estimated impact can be identified for one or more response levels, one or more reduction actions corresponding to a given level, etc. (e.g., which (one or more) hosts, (one or more) instances, (one or more) workloads, (one or more) customers may be affected, which attributes are associated with the affected (one or more) hosts, (one or more) instances, (one or more) workloads, and / or customers, how many (one or more) hosts / (one or more) instances / (one or more) workloads / (one or more) customers are affected, etc.). In some embodiments, the estimated impact of a given action can be aggregated with the estimated impacts of all actions for a given level to identify the estimated impact of the given level.

[0168] In some embodiments, the impact identification manager 706 can be used to estimate the impact of each level and / or estimate power reduction. In some embodiments, determining the estimated power reduction can be an incremental process that includes identifying applicable resources (e.g., hosts, instances, and / or workloads), determining the current power consumption of the applicable resources, estimating the reduction for each response if the action is implemented / realized, and aggregating the individual estimated reductions of the individual responses to identify the total estimated power reduction corresponding to a given action. This process can be repeated for each action of a given level, and the final estimated power consumption reductions for each action can be aggregated to determine the total estimated power consumption reduction for the given level. The functionality of the impact identification manager 706 can be invoked by the enforcement manager 708.

[0169] In some embodiments, the demand manager 710 can invoke the functionality of the enforcement manager 708, at least in part, based on monitoring or determining a need to change the total power threshold. Monitoring can include monitoring Figure 3Any suitable combination of power data 310, environmental data 314, host / instance data 316, and / or operational data 318. The total power threshold can be modified, and the amount by which the total power threshold is modified can be determined at least in part based on a plurality of trigger events. These trigger events can include, but are not limited to: 1) receiving a request from a government entity (e.g., a local power authority) to reduce power indefinitely or for a period of time; 2) determining current and / or predicted environmental conditions (e.g., current environmental or external temperature, predicted environmental or external temperature); 3) determining the current or predicted total power consumption corresponding to the devices in the physical environment; 4) determining the current and / or predicted operating conditions of one or more temperature control units in the physical environment. The amount by which the total power threshold can or will be modified can be determined at least in part based on a plurality of factors, including but not limited to: the difference between the environmental temperature and the external temperature, the power consumption attributed to or associated with a particular component (e.g., a particular temperature control unit), the amount or percentage specified in the request received from the government entity, or the difference between the current or predicted total power consumption value and the current or predicted total power threshold.

[0170] In some embodiments, after determining the amount of change in the total power threshold, the demand manager 710 can invoke the functionality of the enforcement manager 708 to determine which response level is appropriate for a given change. As a non-limiting example, the enforcement manager 708 can incrementally invoke the functionality of the impact identification manager 706 (e.g., via a function call or other means) to determine the impact associated with a given level (e.g., Figure 8VEPO Response Level 1) corresponding estimated impact and / or estimated power reduction. The result estimated impact and / or estimated power reduction calculated by the impact identification manager can be returned to the enforcement manager 708. In some embodiments, the enforcement manager 708 can obtain and / or implement a predefined scheme to determine the appropriateness of selecting one response level over another. For example, the environment manager 708 can be configured to compare the estimated power reduction of a given level with the current total power threshold identified by the demand manager 710 to determine whether the estimated power consumption of the given level (e.g., VEPO Response Level 1) is sufficient (e.g., exceeds the change required to cause the current total power consumption (e.g., identified and provided by the power management service 402) to drop below the total power threshold identified by the demand manager 710). If sufficient, then in some embodiments, the enforcement manager 708 can evaluate the estimated impact according to the predefined scheme to determine whether the estimated impact is sufficient and / or acceptable. As a non-limiting example, even if the estimated power reduction is sufficient to cause the current total power consumption to drop below the current total power threshold, if the estimated impact exceeds the conditions of the predefined scheme (e.g., the number of affected hosts exceeds the threshold identified in the predefined scheme), then the enforcement manager 708 can reject the response level. If the estimated power consumption and / or estimated impact of the response level fails to meet the requirements / conditions of the predefined scheme, then the enforcement manager 708 can be configured to call the impact identification manager 706 to determine the estimated impact and / or estimated power reduction corresponding to a higher VEPO response level (e.g., VEPO Response Level 2).

[0171] The estimation of impact and / or power reduction can be invoked by the enforcement manager 708 and determined by the impact identification manager 706 one level at a time until the enforcement manager 708 finds the first level that meets the requirements / conditions of the predefined scheme regarding the estimated impact and / or estimated power reduction. The level of the lowest level determined to meet all the impact conditions of the predefined scheme (e.g., the level that appears higher in Table 800) can be referred to as the "least impactful resource level". The level of the lowest level determined to meet all the power reduction conditions of the predefined scheme and / or comply with (e.g., be below) the current total power threshold (e.g., the level that appears higher in Table 800) can be referred to as the "least reduced resource level". The level of the lowest level among the response levels determined to meet all the conditions regarding the estimated impact and estimated power reduction can be referred to as the "first sufficient response level". The enforcement manager 708 can be configured to select the response level, depending on the predefined scheme, that is the first to be identified as the least impactful, least reduced, or the first sufficient.

[0172] In some embodiments, the enforcement manager 708 may cause the impact recognition manager 706 to determine the estimated impact and / or estimated power reduction for each level (or some subset, such as the first five, the next five, etc.). The enforcement manager 708 may be configured to select a particular response level based at least in part on comparing the estimated impact and / or estimated power reduction for each level to determine the response level with the least impact (e.g., having a relatively minimal impact), the least reduction (e.g., resulting in the smallest amount of power reduction sufficient to bring the current total power consumption below the current total power threshold), or the least reduction and the least impact (e.g., at least in part based on a weighted algorithm).

[0173] The estimated impact and / or estimated reduction calculated by the impact recognition manager 706 (possibly by invoking Figure 3 the functions of the computer service 306 and / or the power management service 304), and / or any appropriate data on which these estimates depend (such as current values and / or predicted values) may be presented via the interface 11 discussed in more detail below.

[0174] As discussed herein, any appropriate determination and / or identification by the demand manager 710, the impact recognition manager 706, and / or the enforcement manager 708 (or any appropriate service or subroutine invoked thereby) may depend on current data, historical data, and / or predicted data. As a non-limiting example, the demand manager 710 may determine the amount of change needed and / or the amount by which the current total power threshold needs to be modified based on Figure 1 the current operating condition of the temperature control unit(s) 112 and / or the predicted operating condition of these units over a period of time in the future. The predicted condition may be identified based at least in part on historical data (e.g., indicating a partial or complete failure or shutdown of one or more parts of the temperature control unit) and / or based on the output provided by a machine learning model (e.g., a model trained in the manner described in Figure 12 . In some embodiments, the amount of change in the total power threshold determined by the demand manager 710 may be based on a predefined formula and / or table. For example, a formula and / or table may be obtained from which the capacity associated with the temperature control unit may be determined and / or calculated. The table (if used) may indicate the power consumption attributable to or associated with the temperature control unit. Thus, a failure of the temperature control unit (e.g., partial and / or complete failure) may be associated with all or a proportional share of the power consumption attributable to or associated with the temperature control unit (e.g., the power consumption in the form of heat that the temperature control unit is configured to dispose of). Thus, the demand manager 710 may determine that a complete failure of the temperature control unit requires reducing the total power threshold by an amount equal to the total power consumption attributable to or associated with the temperature control unit specified in the table.

[0175] In some embodiments, the enforcement manager 708 may be configured to present, recommend, or automatically select a given response level from a possible set of response levels based at least in part on any suitable combination of an estimated impact that may be experienced if a reduction action corresponding to a given level is implemented (e.g., effected) and / or an estimated power reduction. Thus, in some embodiments, the VEPO orchestration service 702 may recommend and / or select a particular level based on any suitable combination of: 1) a determination of a level that, if implemented, would likely result in a sufficient but not excessive reduction in power consumption at the resources of the physical environment (e.g., the minimum power consumption reduction sufficient to bring the current power consumption value below the current total power threshold), 2) a determination of a level that includes a least restrictive set of actions, or 3) a determination of a level or action having the least impact (e.g., a set of resources and / or customers that are likely to be affected the least, having a priority or category indicating the lowest degree of collective importance). Determining the least restrictive level or action, or the level or action having the least impact may be determined without regard to the estimated power consumption reduction that may be experienced by implementing the level and / or action, or determining the least restrictive level / action and / or the level / action having the least impact may be determined from one or more levels that, if implemented under a given current condition, are estimated to result in a sufficient reduction in power consumption such that the power consumption value drops to an aggregate value below the current total power threshold. In some embodiments, the enforcement manager 708 may cause any suitable data corresponding to the current, predicted, or estimated power consumption and / or impact to be presented on Figure 11 an interface. In some embodiments, the selection of a particular level to be effected may be driven by user input provided on a user interface. Thus, in some embodiments, the enforcement manager 708 may refrain from performing operations to effect a reduction action for a level until the user has identified and / or confirmed, via user input provided on the user interface, the level to be effected.

[0176] In some embodiments, performing these operations can include instructing the power management service 402 and / or the compute service 306 to perform operations. For example, the enforcement manager 708 can provide the power management service 402 with identifiers of affected resources (e.g., hosts, instances, workloads, etc.) and estimated power reduction and / or reduction action data specifying actions to be taken (e.g., resources specific to power capping, power capping all low-priority hosts / instances / workloads, power capping specific or all resources meeting a set of specific attributes by a specific amount (e.g., 10%, at least x amount, etc.)). As described above, this can be configured to enact / implement corresponding power caps as directed by the enforcement manager 708. Similarly, the compute service 306 can be provided with identifiers of affected resources (e.g., hosts, instances, workloads, etc.) and reduction action data specifying actions for the compute service 306 to take (e.g., shutting down all idle hosts, shutting down these specific hosts, pausing all low-priority workloads, migrating and / or compressing all low-priority workloads to maximize the number of idle / vacant hosts, etc.). As described above, this can be configured to enact / implement corresponding power caps as directed by the enforcement manager 708.

[0177] The demand manager 710 can also be configured to identify that a previous trigger that led to the achievement of a response level no longer occurs, and can modify the total power threshold based on the identification that the trigger condition no longer occurs. For example, the demand manager 710 (e.g., via monitoring the operation data 318) can identify that a temperature control unit that previously failed is no longer operating. Accordingly, the demand manager 710 can modify (e.g., increase) the total power threshold by the amount attributed to the temperature control unit when fully operational. Based on this change, the enforcement manager 706 can be configured to perform any appropriate operations to reverse, to the extent possible, any reduction actions taken previously in accordance with any response level selected in response to the original trigger condition. If power caps were used, then the enforcement manager 706 can instruct the power management service 402 to remove those previously applied power caps. If migrations of workloads and / or instances were performed, then the enforcement manager 706 can instruct the compute service 306 to attempt to migrate those resources back to the original hosts and / or instances that hosted the instance and / or workload, respectively. If resources were paused and / or shut down, then the enforcement manager 706 can instruct the compute service 306 to perform operations to resume the instance and / or workload and / or send a signal to the shut-down host to start. In this way, the enforcement manager 706 can perform any appropriate operations to recover from reduction actions taken based on a previously selected given response level.

[0178] Figure 10 is a block diagram illustrating an example use case of applying multiple response levels based on the current conditions of a set of hosts according to at least one embodiment.Figure 10 The examples provided in Figure 8 discussion of the VEPO response levels and corresponding actions are Figure 7 the current configuration data used by the VEPO orchestration service 702 of

[0179] · Servers 1004A and 1004C are vacant.

[0180] · Server 1004D is hosting resources

[0181] (e.g., instances) associated with a customer at category level 1.

[0182] · Server 1004I is hosting BM instances associated with a customer associated with priority level 2.

[0183] · Server 1004J is hosting the BM of a customer associated with priority level 3.

[0184] · Server 1004K is hosting the VM of a customer associated with priority level 1.

[0185] · Server 1004B is hosting one or more instances that are considered critical for recovery. In some embodiments, these instances may be associated with the service(s) 912 provided by the cloud provider.

[0186] As described above, the VEPO orchestration service 702 can identify the estimated impact and / or estimated power reduction corresponding to each response level step by step or in parallel. The VEPO orchestration service 702 can identify the responses applicable to a given level (e.g., the reduction actions corresponding to a given response level) (e.g., through the functions provided by the power management service 304 and / or the computing service 306). For example, in the ongoing example, the VEPO orchestration service 702 (through the power management service 304) can identify that servers 1004A and 1004C are a set of resources for the redundant action application of VEPO response level 1 based on (e.g., from non-cloud-provider-based resources such as resources associated with a user territory such as user territory 902), and thus identify that VEPO response level 1 is estimated to potentially affect 2 servers, namely servers 1004A and 1004C. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which the estimated power reduction of a given level (if formulated / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 10% reduction will have no impact on the customer. The idle BM capacity is reduced to 0%, and the idle CM capacity is severely reduced, etc.). Any appropriate part of this information can be presented on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0187] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources (e.g., from non-cloud-provider-based resources such as resources associated with a user's domain (such as user domain 902)) on which server 1004D is hosting redundant action applications at the VEPO response level 3, also identify that the VEPO response level 2, including a scaled-down action at the VEPO response level 1, is estimated to potentially affect servers 1004A and 1004C. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at the VEPO response level 2 (if instituted / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 20% reduction has a minimal impact on the customer. Up to 400 low-priority VMs are affected, etc.). Any appropriate portion of this information can be presented on the user interface 1100, with similar data associated with other VEPO response levels presented or not presented.

[0188] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources (e.g., from non-cloud-provider-based resources such as resources associated with a user's domain (such as user domain 902)) on which the BM hosted by server 1004I and the VM hosted by server 1004J are resources for redundant action applications at the VEPO response level 3, also identify that the VEPO response level 3, including scaled-down actions at the VEPO response levels 1 and 2, is estimated to potentially affect servers 1004A, 1004C, and 1004D. Resources hosted by server 1004K and / or server 1004K can be excluded based on meeting an exclusion associated with the evaluated response level. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at the VEPO response level 3 (if instituted / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 40% reduction has a moderate impact. Up to 200 class 1 or 2 BMs / VMs are affected, etc.). Any appropriate portion of this information can be displayed on the user interface 1100, with similar data associated with other VEPO response levels presented or not presented.

[0189] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources hosted by server 1004I and VM hosted by server 1004J as resources for which redundant actions at VEPO response level 4 are applied based on identification (e.g., from non-cloud-provider-based resources such as resources associated with a user premise (such as user premise 902)), also identify that a VEPO response 4 estimate may affect all servers other than server 1004B (e.g., a group of servers including servers 1004A and 1004C - 1004L) based on reduction actions including VEPO response levels 1 - 3. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at VEPO response level 4 (if enacted / implemented) will affect the total power consumption of resources in the physical environment (e.g., an 80% reduction has a severe impact, regional outages, etc.). Any appropriate portion of this information can be displayed on the user interface 1100, with similar data associated with other VEPO response levels presented or not presented.

[0190] Figure 11 FIG. is a schematic diagram of an example user interface 1100 according to at least one embodiment. The user interface 1100 can include any appropriate number and type of graphical interface elements (e.g., drop-down boxes 1102 - 1108) to respectively select and / or view VEPO data corresponding to a given region, availability domain (AD), building, and / or room. Other interface elements, such as radio buttons and / or edit boxes, can be used. The VEPO data presented on the user interface 1100 can correspond to the region, AD, building, and room selected via the graphical interface elements 1102 - 1108 (also referred to as "options" 1102 - 1108). As depicted, the VEPO data 1109 is displayed according to the values selected via the options 1102 - 1108.

[0191] As discussed above, any appropriate information used by the VEPO orchestration service 702, the power management service 304, and / or the compute service 306 can be presented via the user interface 1100. As a non-limiting example, any appropriate combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318 can be presented in the user interface 1100.

[0192] The specific content of the VEPO data 1109 displayed on the user interface 1100 may vary, but as depicted, the VEPO data 1109 includes VEPO level data (e.g., presented in column 1110 related to Figure 8The identifier corresponding to the VEPO response level), the current power consumption value (e.g., the current power consumption value presented in column 1112), the estimated reduction associated with each VEPO level (e.g., the estimated VEPO reduction presented in column 1114), and the estimated impact associated with each VEPO level (e.g., the estimated VEPO impact presented in column 1116).

[0193] The current total power threshold can be displayed at 1118 (e.g., the current total power threshold determined by Figure 7 the demand engine 710). The current power reduction required to meet the current total power threshold can be presented at 1120. This value (e.g., indicating a need to reduce by 5 kW) can be calculated based at least in part on identifying the difference between the current total power threshold (e.g., 11 kW) and the current power consumption presented at 1112 (e.g., 16 kW) (e.g., by the demand engine 710).

[0194] The user interface 1100 can be configured to present estimated impact data corresponding to each VEPO response level (referred to herein as "VEPO level" for brevity) within column 1116. In some embodiments, a region corresponding to a row of column 1116 can be selected to present an expanded view of the estimated impact data corresponding to a particular VEPO level. Similarly, a region corresponding to a row of column 1114 can be selected to present an expanded view of the estimated VEPO reduction corresponding to a particular VEPO level.

[0195] The user interface 1100 can include an indicator 1122 that indicates the recommended VEPO response level (as depicted, the system recommends VEPO level 2). As discussed above, the recommended VEPO level can be identified by the power orchestration service 702 based on the above factors. In some embodiments, the recommended VEPO level (e.g., VEPO level 2) can be selected based on identifying that VEPO level 2 is associated with the minimum estimated VEPO reduction amount required to meet the reduction indicated at 1120. In some embodiments, the VEPO level 2 can be recommended additionally or alternatively based at least in part on identifying that the estimated impact of applying VEPO level 2 provides a minimum impact relative to the estimated impact of VEPO level 104.

[0196] In some embodiments, a region corresponding to each row of VEPO data 1109 can be selected to select a given VEPO response level. Once selected, an additional menu or options can be provided to the user for indicating that the system should continue with the selected VEPO response according to the reduction actions associated with that level.

[0197] Figure 12The flow diagram depicts an example method 1200 for training one or more machine learning models according to at least one embodiment. In some embodiments, any suitable machine learning algorithm (e.g., supervised, unsupervised, etc.) and any suitable number of training data sets (e.g., training data 1208) can be used to train the (one or more) models 1202 by, for example, the power management service 304 of Figure 3 , Figure 3 the power orchestration service 302 of Figure 3 , Figure 3 the computing service 306 of Figure 3 , or different devices or systems. Supervised machine learning algorithms refer to machine learning tasks that involve learning an inference function that maps inputs to outputs based on a labeled training data set of known example input / output pairs. Unsupervised machine learning algorithms refer to a set of algorithms for analyzing and clustering unlabeled data sets (e.g., unlabeled data 1210). These algorithms are configured to identify patterns or data groupings without human intervention. In some embodiments, any suitable number of the (one or more) models 1202 can be trained during the training phase 1204. Figure 3 According to at least one embodiment, at least one of the (one or more) models can be trained to identify the likelihood or confidence that the total power consumption of a downstream device (e.g., the (one or more) hosts 324 of Figure 3 ) will exceed a budget threshold corresponding to an upstream device (e.g., a row-level device such as the row PDU 204 of Figure 2 ) that distributes power to any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. The training data 1208 used to train one or more of the (one or more) models 1202 can include any suitable combination of individual and / or total power consumption values (current or historical) of power data 310 corresponding to one or more hosts (e.g., the (one or more) hosts 324 of Figure 3 , the (one or more) servers 104 of Figure 1 , etc.). In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the (one or more) models can be trained to identify a prediction corresponding to the total power consumption of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 can include current or historical power caps applied to the corresponding hosts, and the output provided by such (one or more) models 1208 can identify a predicted power cap value for any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104.

[0198] According to at least one embodiment, at least one of the (one or more) models can be trained to identify the likelihood or confidence that the total power consumption of a downstream device (e.g., the (one or more) hosts 324 of Figure 3 ) will exceed a budget threshold corresponding to an upstream device (e.g., a row-level device such as the row PDU 204 of Figure 2 ) that distributes power to any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. The training data 1208 used to train one or more of the (one or more) models 1202 can include any suitable combination of individual and / or total power consumption values (current or historical) of power data 310 corresponding to one or more hosts (e.g., the (one or more) hosts 324 of Figure 3 , the (one or more) servers 104 of Figure 1 , etc.). In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the (one or more) models can be trained to identify a prediction corresponding to the total power consumption of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 can include current or historical power caps applied to the corresponding hosts, and the output provided by such (one or more) models 1208 can identify a predicted power cap value for any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. Figure 3 the (one or more) hosts 324) will exceed a budget threshold corresponding to an upstream device (e.g., a row-level device such as Figure 2 the row PDU 204 of Figure 2 ) that distributes power to any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. The training data 1208 used to train one or more of the (one or more) models 1202 can include any suitable combination of individual and / or total power consumption values (current or historical) of power data 310 corresponding to one or more hosts (e.g., Figure 3 the (one or more) hosts 324 of Figure 3 , Figure 1 the (one or more) servers 104 of Figure 1 , etc.). In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the (one or more) models can be trained to identify a prediction corresponding to the total power consumption of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 can include current or historical power caps applied to the corresponding hosts, and the output provided by such (one or more) models 1208 can identify a predicted power cap value for any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104.

[0199] According to at least one embodiment, at least one of the one or more models can be trained to identify a predicted workload for any suitable combination of one or more hosts 324 and / or one or more servers 104. Training data 1208 for training one or more of the one or more models 1202 to identify the predicted workload can include any suitable combination of current and / or historical data of host / instance data 316 corresponding to one or more hosts (e.g., Figure 3 one or more hosts 324 of, Figure 1 one or more servers 104 of, etc.). In some embodiments, the training data 1208 for training these one or more specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more models can be trained to identify the likelihood or confidence that a predicted set of workloads will be assigned to any suitable combination of one or more hosts 324 and / or one or more servers 104.

[0200] According to at least one embodiment, at least one of the one or more models can be trained to identify a predicted partial or complete failure of one or more temperature control units. Training data 1208 for training one or more of the one or more models 1202 to identify such failures can include current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where one or more hosts 324 and / or one or more servers 104 are located, current or past external temperatures occurring relative to a region outside the physical environment (e.g., the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, and any suitable combination of current and / or historical failures experienced by one or more temperature control units (e.g., Figure 1 one or more temperature control systems 112 of). In some embodiments, the training data 1208 for training these one or more specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more models can be trained to identify the likelihood or confidence that a specific failure (partial or complete) of one or more temperature control systems 112 will occur during a future time period. In some embodiments, one or more models can be trained to identify the degree of failure of one or more temperature control systems 112 (e.g., 80% failure, 100% failure, etc.).

[0201] According to at least one embodiment, at least one of the (one or more) models can be trained to provide an estimated total power reduction required due to a partial and / or complete failure of one or more temperature control units (e.g., the (one or more) temperature control systems 112). The training data 1208 used to train one or more of the (one or more) models 1202 to identify the required estimated total power reduction can include any suitable combination of the following: current and / or historical data of environmental data 314 and / or operational data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current and / or historical external temperatures that have occurred currently or in the past relative to a region outside the physical environment (e.g., in the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, current and / or historical failures experienced by one or more temperature control units (e.g., Figure 1 the (one or more) temperature control systems 112), current and / or historical total power thresholds, and / or current and / or historical power consumption levels of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operational data 318. In some embodiments, one or more of the (one or more) models can be trained to identify the likelihood or confidence that an estimated total power reduction will be required during a future time period.

[0202] According to at least one embodiment, at least one of the (one or more) models can be trained to predict future temperatures (e.g., ambient temperature and / or external temperature). The training data 1208 used to train one or more of the (one or more) models 1202 to identify future temperatures can include any suitable combination of the following: current and / or historical data of environmental data 314 and / or operational data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current and / or historical external temperatures that have occurred currently or in the past relative to a region outside the physical environment (e.g., in the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, one or more temperature control units (e.g., Figure 1The current and / or historical faults experienced by the (one or more) temperature control systems 112, the current and / or historical total power thresholds, and / or the current and / or historical power consumption levels of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 may include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, (one or more) models may be trained to identify the likelihood or confidence of experiencing a predicted temperature during a future time period.

[0203] Generally, the (one or more) models 1202 may include any suitable number of models. The (one or more) models 1202 may be individually trained to identify the likelihood or confidence or accuracy of the output provided by the (one or more) models 1202 from the training data 1208 discussed above. In some embodiments, the (one or more) models 1202 may be configured to determine values corresponding to the examples provided above, and the likelihood and / or confidence values may indicate the likelihood or confidence that the output provided by the (one or more) models is accurate.

[0204] Generally, the (one or more) models 1202 may be trained during the training phase 1204 using supervised learning algorithms and labeled data 1206 to identify the outputs described in the examples above. The likelihood value may be a binary indicator, a percentage, a confidence value, etc., which indicates the degree of likelihood. The likelihood value may be a binary indicator that indicates whether a specific budget amount is likely or unlikely to be breached, or the likelihood value may indicate the likelihood and / or confidence that a predicted output will occur in the future. The labeled data 1206 may be any suitable portion of the potential training data (e.g., training data 1208) that can be used to train the (one or more) models to produce the outputs described above. The labeled data 1206 may include any suitable number of examples of current and / or historical data corresponding to any suitable number of devices in the physical environment (e.g., data center 102, the (one or more) servers 104, the (one or more) temperature control systems 112, Figure 2 PDUs, etc.). In some embodiments, the labeled data 1206 may include labels identifying known likelihoods and / or actual values. Using the labeled data 1206, a model (e.g., an inference function) that maps inputs (e.g., one or more instances of training data) to outputs (e.g., predicted values or the likelihood / confidence value that the predicted value or output will occur during a future time period) can be learned. In some embodiments, Figure 3Any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306, or a separate service or system, can train any suitable combination of the (one or more) models 1202. In some embodiments, any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306 can obtain a pre-trained version of the (one or more) models 1202.

[0205] (One or more) models 1202 and the various types of models discussed above can include any suitable number of models that are trained using unsupervised learning techniques to identify probabilities / confidences and / or predictions corresponding to the examples provided above. Unsupervised machine learning algorithms are configured to learn patterns from unlabeled data. In some embodiments, the training phase 1204 can utilize unsupervised machine learning algorithms to generate one or more of the (one or more) models 1202. For example, the training data 1208 can include unlabeled data 1210 (e.g., any suitable combination of current values and / or historical values corresponding to power data 310, account data 12, environmental data 314, host / instance data 316, operational data 318, etc.). The unlabeled data 1210 can be used with an unsupervised learning algorithm to divide the entries of the unlabeled data 1210 into groups. The unsupervised learning algorithm can be configured to group similar entries together into common groups. Examples of unsupervised learning algorithms can include clustering methods such as k-means clustering, DBScan, etc. In some embodiments, the unlabeled data 1210 can be clustered with the labeled data 1206 such that the unlabeled instances of a given group can be assigned the same label as the other labeled instances within that group.

[0206] In some embodiments, any suitable portion of the training data 1208 can be used to train the model(s) 1202 during the training phase 1204. For example, 70% of the labeled data 1206 and / or the unlabeled data 1210 can be used to train the model(s) 1202. Once trained, or at any suitable time, the model(s) 1202 can be evaluated to assess its quality (e.g., the accuracy of the output(s) 1212 relative to the labels corresponding to the labeled data 1206). By way of example, a portion of the examples of the labeled data 1206 and / or the unlabeled data 1210 can be used as input to the model(s) 1202 to generate the output(s) 1212. By way of example, an example of the labeled data 1206 can be provided as input, and the corresponding output (e.g., the output(s) 1212) can be compared to the label known to be associated with that example. If a portion of the output (e.g., the label) matches the example label, then that portion of the output can be considered accurate. Any suitable number of labeled examples can be used, and the number of accurate labels can be compared to the total number of examples provided (and / or the total number of previously identified labels) to determine an accuracy value for the given model, which quantifies how accurate the model is. For example, if 90 out of 100 input examples generate output labels that match the previously known example labels, then the model being evaluated can be determined to be 90% accurate.

[0207] In some embodiments, when the model(s) 1202 is used for subsequent inputs, the subsequent output(s) generated by the model(s) 1202 can be added to the corresponding input and used to retrain and / or update the model(s) 1202 at 1216. In some embodiments, examples may not be used to retrain or update the model until the feedback process 1214 is performed. In the feedback process 1214, an example (e.g., an example including one or more historical consumption data instances corresponding to one or more devices and / or racks) and the corresponding output generated by one of the model(s) 1202 for the example are presented to the user, and the user identifies whether the generated output (e.g., the quantity and / or the likelihood confidence value) is correct for the given example.

[0208] Figure 12 The training process depicted (e.g., method 1200) can be performed any suitable number of times at any suitable interval and / or according to any suitable schedule such that the accuracy of the model(s) 1202 improves over time.

[0209] In some embodiments, any suitable number and / or combination of (one or more) models 1202 may be used to determine an output. In some embodiments, the power management service 402 may utilize any suitable combination of outputs provided by (one or more) models 1202 to determine whether the budget threshold for a given component (e.g., row-level PDU) is likely to be breached (and / or likely to be breached by a certain amount). Thus, in some embodiments, the power management service 402 may utilize models trained using any suitable combination of supervised and unsupervised learning algorithms.

[0210] Figure 13 FIG. is a block diagram illustrating an example method 1300 for achieving a response level among multiple response levels based at least in part on an estimated power reduction according to at least one embodiment. Method 1300 may be performed by Figure 3 one or more components of the power orchestration system 300 or in conjunction with Figure 4 and Figure 7 its sub-components discussed. By way of example. Method 1300 may be performed at least in part by Figure 3 any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the compute service 306 of Figure 13 . The operations of method 1300 may be performed in any suitable order. Method 1300 may include more or fewer operations than

[0211] those depicted in Figure 7 . Method 1300 may begin at 1302, where the multi-group of corresponding workloads executed on each of the multiple hosts may be identified (e.g., by Figure 3 the impact identification manager 706 of

[0212] . In some embodiments, the multi-group of corresponding workloads executed on each of the multiple hosts may be identified at least in part based on Figure 7 the host / instance data 316 of Figure 7 . At 1304, multiple response levels specifying the applicability of a corresponding set of reduction actions to the multiple hosts may be identified (e.g., by Figure 8 the impact identification manager 706 of Figure 8The VEPO response level in [the context] can specify that all vacant / idle hosts and hypervisors that can be fully evacuated are applicable to a certain action (e.g., power-off). Thus, the scaling action indicates that a subset of resources (e.g., vacant / idle hosts and hypervisors that can be fully evacuated) is applicable to a given scaling action (e.g., power-off), and indicates the corresponding response level by association.

[0213] At 1306, a first estimate of the power reduction caused by the first response level can be determined (e.g., by the impact identification manager 706). In some embodiments, the first estimate can be determined based at least on (a) multiple sets of corresponding workloads executed on each of the multiple hosts and (b) the applicability of the first set of scaling actions to the multiple hosts according to the first response level.

[0214] At 1308, a first response level can be selected from multiple response levels (e.g., by the Figure 7 enforcement manager 708) based at least on the first estimate of the power reduction caused by the first response level. In some embodiments, the enforcement manager 708 can select the first response level based at least in part on identifying that the corresponding set of scaling actions for the multiple hosts of the first response level is sufficient to reduce the power consumption level of the multiple hosts to a value below a total power threshold (e.g., the total power threshold determined by the Figure 7 demand manager 710). In some embodiments, for each response level (e.g., the Figure 8 VEPO response level), a corresponding estimate can be identified (e.g., by the impact identification manager 706), and the first response level can be selected based at least in part on having the least impact (e.g., affecting fewer workloads, hosts, customers, or any suitable combination of the above) while being sufficient to reduce the power consumption level of the multiple hosts to a value below the total power threshold.

[0215] At 1310, the first set of scaling actions can be applied to the multiple hosts according to the selected first response level. For example, instructions can be transmitted (e.g., by the Figure 7 enforcement manager 708) to cause the first set of scaling actions to be applied to the multiple hosts. In some embodiments, the instructions can be transmitted to the Figure 3 power management service 304 and / or the Figure 3 computing service 306. At least some of the instructions can cause the identification / application of power caps, migration of workloads and / or virtual machines, shutting down of (one or more) virtual machines and / or (one or more) hosts, migration of workloads, or any suitable combination of the above.

[0216] Figure 14is a block diagram illustrating an example method 1400 for achieving a response level in multiple response levels based at least in part on a predicted impact. Method 1400 may be performed by one or more components of the power orchestration system 300 of Figure 3 or in combination with subcomponents thereof discussed in Figure 4 and Figure 7 For example, method 1400 may be performed at least in part by any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306 of Figure 3 The operations of method 1400 may be performed in any suitable order. Method 1400 may include more or fewer operations than those depicted in Figure 14

[0217] Method 1400 may begin at 1402, where multiple corresponding sets of workloads executed on each of multiple hosts may be identified (e.g., by the impact identification manager 706 of Figure 7 ). In some embodiments, the multiple corresponding sets of workloads executed on each of multiple hosts may be identified at least in part based on the host / instance data 316 of Figure 3

[0218] At 1404, multiple response levels specifying the applicability of a corresponding set of reduction actions to the multiple hosts may be identified (e.g., by the impact identification manager 706 of Figure 7 ). In some embodiments, the multiple response levels may be identified from configuration data (e.g., the configuration data 712 of Figure 7 ). In some embodiments, the multiple response levels (e.g., the VEPO response levels of Figure 8 ) specify the applicability of a corresponding set of reduction actions to the multiple hosts (e.g., via one or more reduction actions corresponding to each response level). In some embodiments, a first response level among the multiple response levels specifies the applicability of a first set of reduction actions to the multiple hosts (e.g., at least in part based on one or more reduction actions associated with that response level). For example, Figure 8 the VEPO response levels in may specify that all vacant / idle hosts and hypervisors that may be fully evacuated are applicable to a certain action (e.g., power-off). Thus, a reduction action indicates that a subset of resources (e.g., vacant / idle hosts and hypervisors that may be fully evacuated) is applicable to a given reduction action (e.g., power-off), and by association indicates the corresponding response level.

[0219] ​​At 1406, a first estimate of the predicted impact caused by the first response level can be determined (e.g., by the impact recognition manager 706). In some embodiments, the first estimate can be determined based at least on (a) multiple sets of corresponding workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level. In some embodiments, the predicted workload impact can be determined based on at least one of the priority of the hosts, the number of affected hosts, the number of affected workloads, the priority level of the affected workloads, the number of affected customers, or the priority level of the affected customers. In some embodiments, the prediction of the workload impact can be at least partially based on historical host / instance data (e.g., Figure 3 host / instance data 316), account data (e.g., account data 312), and / or any suitable data discussed in connection with Figure 3 . In some embodiments, the prediction of the workload impact can be at least partially based on providing input data (e.g., Figure 12 host / instance data 316, account data 312, etc.) to a machine learning model (e.g., Figure 3 one or more models 1202).

[0220] At 1408, a first response level can be selected from the multiple response levels based at least on the first estimate of the workload impact caused by the first response level (e.g., by the enforcement manager 708 of Figure 7 ). In some embodiments, the enforcement manager 708 can select the first response level based at least in part on identifying that the first estimate of the workload impact indicates fewer workloads affected than the estimated workload impact corresponding to at least one other response level among the multiple response levels. In some embodiments, the workload impact can indicate fewer workloads, (one or more) hosts, and / or customers affected. In some embodiments, the enforcement manager 708 can also select the first response level based on identifying that applying a set of reduction actions corresponding to the first response level is expected to reduce the power consumption level of the multiple hosts to a value below the total power threshold.

[0221] At 1410, the first set of reduction actions can be applied to the multiple hosts according to the selected first response level. By way of example, instructions can be transmitted (e.g., by the enforcement manager 708 of Figure 7 ) to cause the first set of reduction actions to be applied to the multiple hosts. In some embodiments, the instructions can be transmitted to the Figure 3 power management service 304 and / or Figure 3computing service 306. At least some of the instructions may cause identification / application of a power limit, migration of workloads and / or virtual machines, shutting down one or more virtual machines and / or one or more hosts, migrating workloads, or any suitable combination of the foregoing.

[0222] Figure 15 is a block diagram illustrating an example method for pre-migrating workloads affected by a selected response level according to at least one embodiment. Method 1500 may be performed by Figure 3 one or more components of the power orchestration system 300 or in combination with Figure 4 and Figure 7 its sub-components discussed. For example, method 1500 may be performed at least in part by Figure 3 any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306 of Figure 15 . The operations of method 1500 may be performed in any suitable order. Method 1500 may include more or fewer operations than those depicted in

[0223] Method 1500 may begin at 1502, where it may be determined (e.g., by Figure 7 the impact identification manager 706 of Figure 3 ) the multiple sets of corresponding predicted workloads to be executed on each of the multiple hosts during a future time period. In some embodiments, the multiple sets of corresponding predicted workloads to be executed on each of the multiple hosts during a future time period may be identified at least in part based on historical data (e.g., included in Figure 12 the host / instance data 316 of Figure 3 ). In some embodiments, the multiple sets of predicted workloads may be predicted at least in part based on providing input data (e.g.,

[0224] the host / instance data 316, the account data 312, etc. of Figure 7 ) to a machine learning model (e.g., one or more of the one or more models 1202 of Figure 7 ). In some embodiments, the multiple response levels that specify the applicability of a corresponding set of reduction actions to the multiple hosts may be identified (e.g., by Figure 8The VEPO response level) specifies the applicability of a corresponding set of reduction actions to multiple hosts (e.g., via one or more reduction actions corresponding to each response level). In some embodiments, the first response level among the multiple response levels specifies the applicability of the first set of reduction actions to multiple hosts (e.g., at least in part based on one or more reduction actions associated with that response level). For example, Figure 8 The VEPO response level in can specify that all vacant / idle hosts and hypervisors that can be fully evacuated are applicable to each action (e.g., power off). Thus, the reduction action indicates that a subset of resources (e.g., vacant / idle hosts and hypervisors that can be fully evacuated) is applicable to a given reduction action (e.g., power off), and the corresponding response level is indicated by the association.

[0225] At 1506, a first estimate of the power reduction caused by the first response level can be determined (e.g., by the impact identification manager 706). In some embodiments, the first estimate can be determined based at least on (a) multiple sets of corresponding predicted workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level. In some embodiments, the power reduction can be determined based on at least one of multiple sets of predicted workloads, predicted power consumption values, predicted priorities, etc. In some embodiments, it can be at least in part based on (e.g., included in Figure 3 the power data 310 of) historical / current power data, (e.g., included in Figure 3 the host / instance data 316 of) historical / current host / instance data, (e.g., included in the account data 312) historical / current account data, and / or combined with Figure 3 any suitable historical and / or current data discussed to predict the predicted workloads, predicted power consumption values, predicted priorities, etc. In some embodiments, it can be at least in part based on providing input data (e.g., power data 310, host / instance data 316, account data 312, environment data 314, operation data 318, any suitable combination of the above) to a machine learning model (e.g., Figure 12 one or more of the (one or more) models 1202 of) to determine the power reduction caused by the first response level.

[0226] At 1508, the first response level can be selected from the multiple response levels (e.g., by the Figure 7 enforcement manager 708 of) at least based on the first estimate of the power reduction caused by the first response level. In some embodiments, the enforcement manager 708 can at least in part based on identifying that the corresponding set of reduction actions for the multiple hosts of the first response level is sufficient to reduce the power consumption level of the multiple hosts to below the total power threshold (e.g., byFigure 7 The demand manager 710 determines the total power threshold value to select the first response level. In some embodiments, each response level (e.g., Figure 8 the VEPO response level) can be identified (e.g., by the impact identification manager 706), and the first response level can be selected at least in part based on minimizing the impact (e.g., affecting fewer workloads, hosts, customers, or any suitable combination thereof) while being sufficient to reduce the power consumption level of multiple hosts to a value below the total power threshold.

[0227] At 1510, one or more workloads can be identified (e.g., by Figure 7 the impact identification manager 706) that (a) are currently being executed on multiple hosts and (b) will be affected by applying the first set of reduction actions to the multiple hosts according to the selected first response level. In some embodiments, one or more workloads can be predicted workloads. The impact identification manager 706 can predict one or more workloads at least in part based on historical data (e.g., included in Figure 3 the host / instance data 316) and / or at least in part based on the output provided by at least one of the Figure 12 (one or more) models 1202 (e.g., the output provided in response to providing input data including any suitable historical and / or current data including the host / instance data 316).

[0228] At 1512, before a future time period, the affected workloads can be pre-migrated from the multiple hosts to one or more other hosts. As a non-limiting example, the affected workloads (and / or the virtual machines on which the workloads are executed) can be migrated to other hosts to compress the total number of hosts in use and / or empty the hosts and / or virtual machines so that they can be shut down. In some embodiments, these migrations can be initiated at least in part by the enforcement manager 708 based on sending instructions to Figure 3 the computing service 306.

[0229] Figures 16 - 19 depicts multiple example environments that can be hosted by Figure 3 the (one or more) hosts 324. Figures 16 - 19 The environments depicted in depict a cloud computing, multi-tenant environment. As described above, cloud computing and / or multi-tenant environments and other environments can benefit from utilizing the power management techniques disclosed herein. These techniques enable data centers including components that implement environments described in conjunction with Figures 1 - 4 and Figure 7 etc. to utilize the power resources of the data center more efficiently than conventional techniques that result in a large amount of unused power.

[0230] IaaS Infrastructure Example

[0231] Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also provide various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Thus, since these services may be policy-driven, IaaS users can be able to implement policies to drive load balancing to maintain the availability and performance of applications.

[0232] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to an IaaS platform to create a virtual machine (VM), install an operating system (OS) on each VM, deploy middleware such as a database, create buckets for workloads and backups, and even install enterprise software into that VM. Then, the customer can use the provider's services to perform various functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, etc.

[0233] In most cases, the cloud computing model will require the participation of a cloud provider. The cloud provider can be but is not necessarily a third-party service that specifically provides (e.g., provisions, rents, sells) IaaS. An entity may also choose to deploy a private cloud and thus become its own infrastructure service provider.

[0234] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It can also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider and is below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Thus, the customer can be responsible for the processing of (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be launched on demand, etc.).

[0235] In some examples, IaaS provisioning can refer to obtaining computers or virtual hosts for use and even installing the required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0236] In some cases, there are two different challenges with IaaS provisioning. First, there is an initial challenge in provisioning the initial set of infrastructure before anything is running. Second, once everything has been provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two challenges can be addressed by enabling the configuration of the infrastructure to be defined in a declarative manner. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, a workflow can be generated to create and / or manage the different components described in the configuration files.

[0237] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shareable computing resources), also known as the core network. In some examples, there can also be one or more inbound / outbound traffic group rules that are provisioned to define how inbound and / or outbound traffic will be set up for the network, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. can also be provisioned. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve incrementally.

[0238] In some cases, continuous deployment techniques can be employed to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the techniques described can enable infrastructure management within these environments. In some examples, a service team can write code that is expected to be deployed to one or more but typically many different production environments (e.g., across various different geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or code can be deployed using deployment tools once the infrastructure has been provisioned.

[0239] Figure 16 FIG. 1600 is a block diagram illustrating an example schema of an IaaS architecture according to at least one embodiment. A service operator 1602 can be communicatively coupled to a secure host lease 1604, which can include a virtual cloud network (VCN) 1606 and a secure host subnet 1608. In some examples, the service operator 1602 can use one or more client computing devices, which can be portable handheld devices (e.g., cellular phones, a computing tablet, a personal digital assistant (PDA), or a wearable device (e.g., Google Head-Mounted Display), running software such as Microsoft Windows and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and enabling Internet, email, Short Message Service (SMS), or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, for example, including personal computers and / or laptop computers running various versions of Microsoft Apple and / or Linux operating systems. The client computing device can be a workstation computer running various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems such as, for example, Google Chrome OS). Alternatively, or additionally, the client computing device can be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a gesture input device) and / or a personal messaging device, which is capable of communicating via a network that can access VCN 1606 and / or the Internet.

[0240] VCN 1606 can include a Local Peer Gateway (LPG) 1610, which can be communicatively coupled to a Secure Shell (SSH) VCN 1612 via the LPG 1610 included in the SSH VCN 1612. The SSH VCN 1612 can include an SSH subnet 1614, and the SSH VCN 1612 can be communicatively coupled to a control plane VCN 1616 via the LPG 1610 included in the control plane VCN 1616. Moreover, the SSH VCN 1612 can be communicatively coupled to a data plane VCN 1618 via the LPG 1610. The control plane VCN 1616 and the data plane VCN 1618 can be included in a service lease 1619 that can be owned and / or operated by an IaaS provider.

[0241] The control plane VCN 1616 may include a control plane demilitarized zone (DMZ) layer 1620 that serves as a perimeter network (e.g., a portion of a corporate network between an intranet and an external network). Servers based on the DMZ can assume limited liability and help control vulnerabilities. Additionally, the DMZ layer 1620 may include one or more load balancer (LB) subnets 1622, a control plane application layer 1624 that may include one or more application subnets 1626, and a control plane data layer 1628 that may include one or more database (DB) subnets 1630 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). The one or more LB subnets 1622 included in the control plane DMZ layer 1620 may be communicatively coupled to the one or more application subnets 1626 included in the control plane application layer 1624 and an Internet gateway 1634 that may be included in the control plane VCN 1616, and the one or more application subnets 1626 may be communicatively coupled to the one or more DB subnets 1630 included in the control plane data layer 1628, as well as a service gateway 1636 and a network address translation (NAT) gateway 1638. The control plane VCN 1616 may include a service gateway 1636 and a NAT gateway 1638.

[0242] The control plane VCN 1616 may include a data plane mirror application layer 1640, which may include one or more application subnets 1626. The one or more application subnets 1626 included in the data plane mirror application layer 1640 may include virtual network interface controllers (VNICs) 1642 that may execute compute instances 1644. The compute instances 1644 may communicatively couple the one or more application subnets 1626 of the data plane mirror application layer 1640 to the one or more application subnets 1626 that may be included in the data plane application layer 1646.

[0243] The data plane VCN 1618 may include a data plane application layer 1646, a data plane DMZ layer 1648, and a data plane data layer 1650. The data plane DMZ layer 1648 may include one or more LB subnets 1622, which may be communicatively coupled to the one or more application subnets 1626 of the data plane application layer 1646 and an Internet gateway 1634 of the data plane VCN 1618. The one or more application subnets 1626 may be communicatively coupled to a service gateway 1636 of the data plane VCN 1618 and a NAT gateway 1638 of the data plane VCN 1618. The data plane data layer 1650 may also include one or more DB subnets 1630 that may be communicatively coupled to the one or more application subnets 1626 of the data plane application layer 1646.

[0244] The Internet gateway 1634 for the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to the metadata management service 1652, and the metadata management service 1652 can be communicatively coupled to the public Internet 1654. The public Internet 1654 can be communicatively coupled to the NAT gateway 1638 for the control plane VCN 1616 and the data plane VCN 1618. The service gateway 1636 for the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to the cloud service 1656.

[0245] In some examples, the service gateway 1636 for the control plane VCN 1616 or the data plane VCN 1618 can make application programming interface (API) calls to the cloud service 1656 without going through the public Internet 1654. The API calls from the service gateway 1636 to the cloud service 1656 can be one-way: the service gateway 1636 can make API calls to the cloud service 1656, and the cloud service 1656 can send the requested data to the service gateway 1636. However, the cloud service 1656 may not initiate API calls to the service gateway 1636.

[0246] In some examples, the secure host lease 1604 can be directly connected to the service lease 1619, which would otherwise be isolated. The secure host subnet 1608 can communicate with the SSH subnet 1614 via the LPG 1610, and the LPG 1610 can enable two-way communication on otherwise isolated systems. Connecting the secure host subnet 1608 to the SSH subnet 1614 can enable the secure host subnet 1608 to access other entities within the service lease 1619.

[0247] The control plane VCN 1616 can allow users of the service lease 1619 to set or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1616 can be deployed or otherwise used in the data plane VCN 1618. In some examples, the control plane VCN 1616 can be isolated from the data plane VCN 1618, and the data plane mirror application layer 1640 of the control plane VCN 1616 can communicate with the data plane application layer 1646 of the data plane VCN 1618 via the VNIC 1642, and the VNIC 1642 can be included in both the data plane mirror application layer 1640 and the data plane application layer 1646.

[0248] In some examples, a user or customer of the system can make requests, such as create, read, update, or delete (CRUD) operations, over a public Internet 1654 that can transmit requests to a metadata management service 1652. The metadata management service 1652 can transmit the requests over an Internet gateway 1634 to a control plane VCN 1616. The requests can be received by one or more LB subnets 1622 included in a control plane DMZ layer 1620. The one or more LB subnets 1622 can determine that the requests are valid, and in response to that determination, the one or more LB subnets 1622 can transmit the requests to one or more application subnets 1626 included in a control plane application layer 1624. If the requests are authenticated and a call to the public Internet 1654 is required, then the call to the public Internet 1654 can be transmitted to a NAT gateway 1638 that can make calls to the public Internet 1654. Metadata that the requests may expect to be stored can be stored in one or more DB subnets 1630.

[0249] In some examples, a data plane mirror application layer 1640 can facilitate direct communication between a control plane VCN 1616 and a data plane VCN 1618. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 1618. Via a VNIC 1642, the control plane VCN 1616 can communicate directly with resources included in the data plane VCN 1618 and thereby can perform configuration changes, updates, or other suitable modifications.

[0250] In some embodiments, a control plane VCN 1616 and a data plane VCN 1618 can be included in a service tenancy 1619. In this case, a user or customer of the system may not own or operate the control plane VCN 1616 or the data plane VCN 1618. Instead, an IaaS provider can own or operate the control plane VCN 1616 and the data plane VCN 1618, both of which can be included in the service tenancy 1619. This embodiment can enable isolation of networks that may prevent a user or customer from interacting with resources of other users or other customers. Moreover, this embodiment can allow a user or customer of the system to privately store databases without relying on the public Internet 1654, which may not have a desired threat prevention level, for storage.

[0251] In other embodiments, the (one or more) LB subnets 1622 included in the control plane VCN 1616 may be configured to receive signals from the service gateway 1636. In this embodiment, the control plane VCN 1616 and the data plane VCN 1618 may be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1654. A customer of the IaaS provider may desire this embodiment because the (one or more) databases used by the customer may be controlled by the IaaS provider and may be stored on the service lease 1619, which may be isolated from the public Internet 1654.

[0252] Figure 17 is a block diagram 1700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1702 (e.g., Figure 16 the service operator 1602) may be communicatively coupled to a secure host lease 1704 (e.g., Figure 16 the secure host lease 1604), which may include a virtual cloud network (VCN) 1706 (e.g., Figure 16 the VCN 1606) and a secure host subnet 1708 (e.g., Figure 16 the secure host subnet 1608). The VCN 1706 may include a local peering gateway (LPG) 1710 (e.g., Figure 16 the LPG 1610), which may be communicatively coupled to a Secure Shell (SSH) VCN 1712 (e.g., Figure 16 the SSH VCN 1612) via the LPG 1610 included in the SSH VCN 1712. The SSH VCN 1712 may include an SSH subnet 1714 (e.g., Figure 16 the SSH subnet 1614), and the SSH VCN 1712 may be communicatively coupled to a control plane VCN 1716 (e.g., Figure 16 the control plane VCN 1616) via the LPG 1710 included in the control plane VCN 1716. The control plane VCN 1716 may be included in a service lease 1719 (e.g., Figure 16 the service lease 1619), and a data plane VCN 1718 (e.g., Figure 16 the data plane VCN 1618) may be included in a customer lease 1721 that may be owned or operated by a user or customer of the system.

[0253] The control plane VCN 1716 may include a control plane DMZ layer 1720 (e.g., Figure 16 the control plane DMZ layer 1620), which may include the (one or more) LB subnets 1722 (e.g.,Figure 16 one or more LB subnets 1622), and may include one or more application subnets 1726 (e.g., Figure 16 one or more application subnets 1626) of the control plane application layer 1724 (e.g., Figure 16 the control plane application layer 1624), and may include one or more database (DB) subnets 1730 (e.g., similar to Figure 16 one or more DB subnets 1630) of the control plane data layer 1728 (e.g., Figure 16 the control plane data layer 1628). One or more LB subnets 1722 included in the control plane DMZ layer 1720 may be communicatively coupled to one or more application subnets 1726 included in the control plane application layer 1724 and an Internet gateway 1734 that may be included in the control plane VCN 1716 (e.g., Figure 16 the Internet gateway 1634), and one or more application subnets 1726 may be communicatively coupled to one or more DB subnets 1730 included in the control plane data layer 1728, as well as a service gateway 1736 (e.g., Figure 16 the service gateway 1636) and a network address translation (NAT) gateway 1738 (e.g., Figure 16 the NAT gateway 1638). The control plane VCN 1716 may include a service gateway 1736 and a NAT gateway 1738.

[0254] The control plane VCN 1716 may include a data plane mirror application layer 1740 that may include one or more application subnets 1726 (e.g., Figure 16 the data plane mirror application layer 1640). One or more application subnets 1726 included in the data plane mirror application layer 1740 may include a virtual network interface controller (VNIC) 1742 that may execute a compute instance 1744 (e.g., similar to Figure 16 the compute instance 1644) of 1642. The compute instance 1744 may facilitate communication between one or more application subnets 1726 of the data plane mirror application layer 1740 and one or more application subnets 1726 that may be included in the data plane application layer 1746 (e.g., Figure 16 the data plane application layer 1646) via the VNIC 1742 included in the data plane mirror application layer 1740 and the VNIC 1742 included in the data plane application layer 1746.

[0255] The Internet gateway 1734 included in the control plane VCN 1716 may be communicatively coupled to a metadata management service 1752 (e.g.,Figure 16 The metadata management service 1652), and the metadata management service 1752 can be communicatively coupled to a public Internet 1754 (e.g., Figure 16 the public Internet 1654). The public Internet 1754 can be communicatively coupled to a NAT gateway 1738 included in the control plane VCN 1716. A service gateway 1736 included in the control plane VCN 1716 can be communicatively coupled to a cloud service 1756 (e.g., Figure 16 the cloud service 1656).

[0256] In some examples, the data plane VCN 1718 can be included in a customer tenancy 1721. In this case, the IaaS provider can provide a control plane VCN 1716 for each customer, and the IaaS provider can set up a unique compute instance 1744 included in a service tenancy 1719 for each customer. Each compute instance 1744 can permit communication between the control plane VCN 1716 included in the service tenancy 1719 and the data plane VCN 1718 included in the customer tenancy 1721. The compute instance 1744 can permit resources provisioned in the control plane VCN 1716 included in the service tenancy 1719 to be deployed or otherwise used in the data plane VCN 1718 included in the customer tenancy 1721.

[0257] In other examples, a customer of the IaaS provider can have a database that exists in the customer tenancy 1721. In this example, the control plane VCN 1716 can include a data plane mirror application layer 1740, which can include one or more application subnets 1726. The data plane mirror application layer 1740 can reside in the data plane VCN 1718, but the data plane mirror application layer 1740 may not be in the data plane VCN 1718. That is, the data plane mirror application layer 1740 can access the customer tenancy 1721, but the data plane mirror application layer 1740 may not exist in the data plane VCN 1718 or be owned or operated by the customer of the IaaS provider. The data plane mirror application layer 1740 can be configured to make calls to the data plane VCN 1718, but may not be configured to make calls to any entity included in the control plane VCN 1716. The customer may desire to deploy or otherwise use resources provisioned in the control plane VCN 1716 in the data plane VCN 1718, and the data plane mirror application layer 1740 can facilitate the customer's desired deployment or other use of the resources.

[0258] In some embodiments, a customer of an IaaS provider can apply filters to a data plane VCN 1718. In this embodiment, the customer can determine what the data plane VCN 1718 can access, and the customer can restrict access from the data plane VCN 1718 to the public Internet 1754. The IaaS provider may not be able to apply filters or otherwise control the data plane VCN 1718's access to any external network or database. Applying filters and controls by the customer to the data plane VCN 1718 included in the customer's tenancy 1721 can help isolate the data plane VCN 1718 from other customers and the public Internet 1754.

[0259] In some embodiments, a cloud service 1756 can be invoked by a service gateway 1736 to access services that may not exist on the public Internet 1754, the control plane VCN 1716, or the data plane VCN 1718. The connection between the cloud service 1756 and the control plane VCN 1716 or the data plane VCN 1718 may not be real-time or continuous. The cloud service 1756 can exist on a different network owned or operated by the IaaS provider. The cloud service 1756 can be configured to receive calls from the service gateway 1736 and can be configured not to receive calls from the public Internet 1754. Some cloud services 1756 can be isolated from other cloud services 1756, and the control plane VCN 1716 can be isolated from cloud services 1756 that may not be in the same region as the control plane VCN 1716. For example, the control plane VCN 1716 can be located in "Region 1", and the cloud service "Deployment 15" can be located in Region 1 and "Region 2". If the service gateway 1736 included in the control plane VCN 1716 located in Region 1 makes a call to Deployment 15, then the call can be transmitted to Deployment 15 in Region 1. In this example, the control plane VCN 1716 or Deployment 15 in Region 1 may not be communicatively coupled or otherwise communicate with Deployment 15 in Region 2.

[0260] Figure 18 is a block diagram 1800 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1802 (e.g., Figure 16 the service operator 1602) can be communicatively coupled to a secure host tenancy 1804 (e.g., Figure 16 the secure host tenancy 1604), which can include a virtual cloud network (VCN) 1806 (e.g., Figure 16 the VCN 1606) and a secure host subnet 1808 (e.g., Figure 16 the secure host subnet 1608). The VCN 1806 can include an LPG 1810 (e.g.,Figure 16 1610), which may be communicatively coupled to the SSH VCN 1812 via the LPG 1810 contained in the SSH VCN 1812 (e.g., Figure 16 SSH VCN 1812). SSH VCN 1812 may include SSH subnet 1814 (e.g., Figure 16 SSH subnet 1614 of the control plane VCN 1816), and SSH VCN 1812 can be communicatively coupled to control plane VCN 1816 via LPG 1810 contained in control plane VCN 1816 (e.g., Figure 16 1616) and is coupled to the data plane VCN 1818 via the LPG 1810 included in the data plane VCN 1818 (e.g., Figure 16 The control plane VCN 1816 and the data plane VCN 1818 may be included in a service lease 1819 (e.g., Figure 16 Service lease 1619).

[0261] The control plane VCN 1816 may include a load balancer (LB) subnet 1822 (e.g., Figure 16 LB subnet(s) 1622) of the control plane DMZ layer 1820 (e.g., Figure 16 The control plane DMZ layer 1620 may include (one or more) application subnets 1826 (e.g., similar to Figure 16 (one or more) application subnets 1626) of the control plane application layer 1824 (e.g., Figure 16 ), and a control plane data layer 1828 (e.g., Figure 16 1828 of the control plane data layer 1820). The LB subnet(s) 1822 contained in the control plane DMZ layer 1820 may be communicatively coupled to the application subnet(s) 1826 contained in the control plane application layer 1824 and the Internet gateway 1834 (e.g., Figure 16 1634), and the application subnet(s) 1826 may be communicatively coupled to the DB subnet(s) 1830 and the service gateway 1836 (e.g., Figure 16 ) and a network address translation (NAT) gateway 1838 (e.g., Figure 16 The control plane VCN 1816 may include a service gateway 1836 and a NAT gateway 1838.

[0262] The data plane VCN 1818 may include a data plane application layer 1846 (e.g., Figure 16 the data plane application layer 1646 of Figure 16 ), a data plane DMZ layer 1848 (e.g., Figure 16 the data plane DMZ layer 1648 of

[0263] ), and a data plane data layer 1850 (e.g., Figure 16 the data plane data layer 1650 of

[0264] Figure 16 ). The data plane DMZ layer 1848 may include one or more trusted application subnets 1860 and one or more untrusted application subnets 1862 that may be communicatively coupled to the data plane application layer 1846 and one or more LB subnets 1822 of an Internet gateway 1834 included in the data plane VCN 1818. One or more trusted application subnets 1860 may be communicatively coupled to a service gateway 1836 included in the data plane VCN 1818, a NAT gateway 1838 included in the data plane VCN 1818, and one or more DB subnets 1830 included in the data plane data layer 1850. One or more untrusted application subnets 1862 may be communicatively coupled to a service gateway 1836 included in the data plane VCN 1818 and one or more DB subnets 1830 included in the data plane data layer 1850. The data plane data layer 1850 may include one or more DB subnets 1830 that may be communicatively coupled to a service gateway 1836 included in the data plane VCN 1818.

[0263] (One or more) untrusted application subnets 1862 may include one or more primary VNICs 1864(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1866(1)-(N). Each tenant VM 1866(1)-(N) may be communicatively coupled to a corresponding application subnet 1867(1)-(N) that may be included in a corresponding container egress VCN 1868(1)-(N), and the corresponding container egress VCNs 1868(1)-(N) may be included in corresponding customer tenancies 1870(1)-(N). Corresponding secondary VNICs 1872(1)-(N) may facilitate communication between one or more untrusted application subnets 1862 included in the data plane VCN 1818 and the application subnets included in the container egress VCNs 1868(1)-(N). Each container egress VCN 1868(1)-(N) may include a NAT gateway 1838 that may be communicatively coupled to a public Internet 1854 (e.g., Figure 16 the public Internet 1654 of

[0264] An Internet gateway 1834 included in the control plane VCN 1816 and included in the data plane VCN 1818 can be communicatively coupled to a metadata management service 1852 (e.g., Figure 16 the metadata management system 1652), and the metadata management service 1852 can be communicatively coupled to the public Internet 1854. The public Internet 1854 can be communicatively coupled to a NAT gateway 1838 included in the control plane VCN 1816 and included in the data plane VCN 1818. A service gateway 1836 included in the control plane VCN 1816 and included in the data plane VCN 1818 can be communicatively coupled to a cloud service 1856.

[0265] In some embodiments, the data plane VCN 1818 can be integrated with a customer lease 1870. In some cases, such as when it may be desirable to support during code execution, this integration may be useful or desirable for customers of the IaaS provider. A customer may provide code that may be disruptive, may communicate with other customer resources, or may otherwise cause undesired effects to run. In response to this, the IaaS provider can determine whether to run the code given to the IaaS provider by the customer.

[0266] In some examples, a customer of the IaaS provider can grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1846. The code running the functionality can be executed in the VMs 1866(1)-(N), and the code can not be configured to run anywhere else on the data plane VCN 1818. Each VM 1866(1)-(N) can be connected to a customer lease 1870. The corresponding containers 1871(1)-(N) included in the VMs 1866(1)-(N) can be configured to run the code. In this case, there can be double isolation (e.g., the containers 1871(1)-(N) run the code, where the containers 1871(1)-(N) may be at least included in the VMs 1866(1)-(N) included in one or more untrusted application subnets 1862), which can help prevent incorrect or otherwise undesired code from damaging the IaaS provider's network or damaging the networks of different customers. The containers 1871(1)-(N) can be communicatively coupled to the customer lease 1870 and can be configured to transmit or receive data from the customer lease 1870. The containers 1871(1)-(N) can not be configured to transmit or receive data from any other entity in the data plane VCN 1818. After the code execution is complete, the IaaS provider can terminate or otherwise dispose of the containers 1871(1)-(N).

[0267] In some embodiments, the (one or more) trusted application subnets 1860 may run code that may be owned or operated by an IaaS provider. In this embodiment, the (one or more) trusted application subnets 1860 may be communicatively coupled to the (one or more) DB subnets 1830 and configured to perform CRUD operations in the (one or more) DB subnets 1830. The (one or more) untrusted application subnets 1862 may be communicatively coupled to the (one or more) DB subnets 1830, but in this embodiment, the (one or more) untrusted application subnets may be configured to perform read operations in the (one or more) DB subnets 1830. Containers 1871(1)-(N) that may be included in each customer's VMs 1866(1)-(N) and may run code from the customer may not be communicatively coupled to the (one or more) DB subnets 1830.

[0268] In other embodiments, the control plane VCN 1816 and the data plane VCN 1818 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1816 and the data plane VCN 1818. However, communication may occur indirectly through at least one method. The LPG 1810 may be established by the IaaS provider, which may facilitate communication between the control plane VCN 1816 and the data plane VCN 1818. In another example, the control plane VCN 1816 or the data plane VCN 1818 may invoke cloud services 1856 via the service gateway 1836. For example, an invocation of cloud services 1856 from the control plane VCN 1816 may include a request for a service that may communicate with the data plane VCN 1818.

[0269] Figure 19 is a block diagram 1900 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1902 (e.g., Figure 16 the service operator 1602) may be communicatively coupled to a secure host lease 1904 (e.g., Figure 16 the secure host lease 1604), which may include a virtual cloud network (VCN) 1906 (e.g., Figure 16 the VCN 1606) and a secure host subnet 1908 (e.g., Figure 16 the secure host subnet 1608). The VCN 1906 may include an LPG 1910 (e.g., Figure 16 the LPG 1610), which may be via an SSH VCN 1912 (e.g., Figure 16LPG 1910 in SSH VCN 1612 of the present invention is communicatively coupled to SSH VCN 1912. SSH VCN 1912 may include SSH subnet 1914 (e.g., Figure 16 SSH subnet 1614 of the control plane VCN 1916), and SSH VCN 1912 can be communicatively coupled to control plane VCN 1916 via LPG 1910 contained in control plane VCN 1916 (e.g., Figure 16 The control plane VCN 1616 of FIG. 1914 is coupled to the data plane VCN 1918 via the LPG 1910 included in the data plane VCN 1918 (e.g., Figure 16 The control plane VCN 1916 and the data plane VCN 1918 may be included in a service lease 1919 (e.g., Figure 16 Service lease 1619).

[0270] The control plane VCN 1916 may include (one or more) LB subnets 1922 (e.g., Figure 16 (one or more) LB subnets 1622) of the control plane DMZ layer 1920 (e.g., Figure 16 The control plane DMZ layer 1620 may include (one or more) application subnets 1926 (e.g., Figure 16 (one or more) application subnets 1626) of the control plane application layer 1924 (e.g., Figure 16 control plane application layer 1624), may include (one or more) DB subnets 1930 (e.g., Figure 18 (one or more) DB subnets 1830) of the control plane data layer 1928 (e.g., Figure 16 The LB subnet(s) 1922 contained in the control plane DMZ layer 1920 may be communicatively coupled to the application subnet(s) 1926 contained in the control plane application layer 1924 and the internet gateway 1934 (e.g., Figure 16 1924), and the application subnet(s) 1926 may be communicatively coupled to the DB subnet(s) 1930 and the service gateway 1936 (e.g., Figure 16 ) and a network address translation (NAT) gateway 1938 (e.g., Figure 16 NAT gateway 1638). The control plane VCN 1916 may include a service gateway 1936 and a NAT gateway 1938.

[0271] The data plane VCN 1918 may include a data plane application layer 1946 (e.g., Figure 16 Data plane application layer 1646), data plane DMZ layer 1948 (e.g., Figure 16 Data plane DMZ layer 1648)), and data plane data layer 1950 (e.g., Figure 16 The data plane DMZ layer 1948 may include (one or more) trusted application subnets 1960 (e.g., Figure 18 (one or more) trusted application subnets 1860) and (one or more) untrusted application subnets 1962 (e.g., Figure 18 The trusted application subnet(s) 1960 may be communicatively coupled to the service gateway 1936 contained in the data plane VCN 1918, the NAT gateway 1938 contained in the data plane VCN 1918, and the DB subnet(s) 1930 contained in the data plane data layer 1950. The untrusted application subnet(s) 1962 may be communicatively coupled to the service gateway 1936 contained in the data plane VCN 1918 and the DB subnet(s) 1930 contained in the data plane data layer 1950. The data plane data layer 1950 may include the DB subnet(s) 1930 that may be communicatively coupled to the service gateway 1936 contained in the data plane VCN 1918.

[0272] The untrusted application subnet(s) 1962 may include primary VNICs 1964(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1966(1)-(N) residing within the untrusted application subnet(s) 1962. Each tenant VM 1966(1)-(N) may run code in a corresponding container 1967(1)-(N) and may be communicatively coupled to an application subnet 1926 that may be contained in a data plane application layer 1946 that may be contained in a container egress VCN 1968. Corresponding secondary VNICs 1972(1)-(N) may facilitate communications between the untrusted application subnet(s) 1962 contained in the data plane VCN 1918 and the application subnets contained in the container egress VCN 1968. The container egress VCN may include a primary VNIC 1964(1)-(N) that may be communicatively coupled to a public Internet 1954 (e.g., Figure 16 NAT gateway 1938 of the public Internet 1654).

[0273] The Internet gateway 1934 included in the control plane VCN 1916 and the Internet gateway 1934 included in the data plane VCN 1918 can be communicatively coupled to a metadata management service 1952 (e.g., Figure 16 the metadata management system 1652), and the metadata management service 1952 can be communicatively coupled to the public Internet 1954. The public Internet 1954 can be communicatively coupled to the NAT gateway 1938 included in the control plane VCN 1916 and included in the data plane VCN 1918. The service gateway 1936 included in the control plane VCN 1916 and included in the data plane VCN 1918 can be communicatively coupled to the cloud service 1956.

[0274] In some examples, Figure 19 the pattern shown in the architecture of the block diagram 1900 of Figure 18 can be considered an exception to the pattern shown in the architecture of the block diagram 1800 of

[0275] and may be desirable for customers of the IaaS provider if the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). The customer can access in real time the respective containers 1967(1)-(N) included in each customer's VMs 1966(1)-(N). The containers 1967(1)-(N) can be configured to make calls to the respective secondary VNICs 1972(1)-(N) included in the (one or more) application subnets 1926 of the data plane application layer 1946, and the data plane application layer 1946 can be included in the container egress VCN 1968. The secondary VNICs 1972(1)-(N) can transmit the calls to the NAT gateway 1938, and the NAT gateway 1938 can transmit the calls to the public Internet 1954. In this example, the containers 1967(1)-(N) that can be accessed by the customer in real time can be isolated from the control plane VCN 1916 and can be isolated from other entities included in the data plane VCN 1918. The containers 1967(1)-(N) can also be isolated from resources from other customers.In other examples, a customer can use containers 1967(1)-(N) to invoke cloud service 1956. In this example, the customer can run code in containers 1967(1)-(N) that requests services from cloud service 1956. Containers 1967(1)-(N) can transmit the request to secondary VNICs 1972(1)-(N), which can transmit the request to a NAT gateway that can transmit the request to public Internet 1954. Public Internet 1954 can transmit the request via Internet gateway 1934 to (one or more) LB subnets 1922 included in control plane VCN 1916. In response to determining that the request is valid, (one or more) LB subnets can transmit the request to (one or more) application subnets 1926, which can transmit the request to cloud service 1956 via service gateway 1936.

[0276] It should be recognized that the IaaS architectures 1600, 1700, 1800, 1900 depicted in the figures may have other components than those depicted. Additionally, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0277] In certain embodiments, the IaaS systems described herein may include application suite, middleware, and database service offerings that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) offered by the present assignee.

[0278] Figure 20 An example computer system 2000 in which various embodiments may be implemented is illustrated. System 2000 may be used to implement any of the computer systems described above. As shown, computer system 2000 includes a processing unit 2004 that communicates with a plurality of peripheral subsystems via a bus subsystem 2002. These peripheral subsystems may include a processing acceleration unit 2006, an I / O subsystem 2008, a storage subsystem 2018, and a communication subsystem 2024. Storage subsystem 2018 includes a tangible computer-readable storage medium 2022 and system memory 2010.

[0279] The bus subsystem 2002 provides a mechanism for enabling the various components and subsystems of the computer system 2000 to communicate with each other as intended. Although the bus subsystem 2002 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2002 can be any of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which can be implemented as Mezzanine buses manufactured to the IEEE P1386.1 standard.

[0280] The processing unit 2004, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 2000. One or more processors can be included in the processing unit 2004. These processors can include single-core or multi-core processors. In certain embodiments, the processing unit 2004 can be implemented as one or more independent processing units 2032 and / or 2034, where each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 2004 can also be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0281] In various embodiments, the processing unit 2004 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in the (one or more) processors 2004 and / or the storage subsystem 2018. Through appropriate programming, the (one or more) processors 2004 can provide the various functions described above. The computer system 2000 can additionally include a processing acceleration unit 2006, which can include a digital signal processor (DSP), a dedicated processor, and the like.

[0282] The I / O subsystem 2008 can include user interface input devices and user interface output devices. User interface input devices can include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keyboard, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can include, for example, motion sensing and / or gesture recognition devices, such as Microsoft A motion sensor that enables a user to control and interact with an input device such as a Microsoft 360 game controller through a natural user interface using gestures and voice commands. The user interface input device may also include an eye gesture recognition device, such as one that detects eye activity from a user (e.g., "blinking" when taking a photo and / or making a menu selection) and converts the eye gesture into an input to the input device (e.g., a Google Blink Detector). Additionally, the user interface input device may include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., a Navigator) through voice commands. The user interface input device may also include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, game pad, and graphics tablet, as well as audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye tracking devices. Additionally, the user interface input device may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. The user interface input device may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0283]

[0284] The user interface output device may include a display subsystem, indicator lights, or a non-visual display such as an audio output device, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one using a liquid crystal display (LCD) or a plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2000 to a user or other computer. For example, the user interface output device may include, but is not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

[0285] The computer system 2000 may include a storage subsystem 2018 that provides a tangible non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which provide the above functionality when executed by one or more cores or processors of the processing unit 2004. The storage subsystem 2018 may also provide a repository for storing data used in accordance with this disclosure.

[0286] As [[ID=2 depicted in the example of, the storage subsystem 2018 may include various components, including system memory 2010, computer-readable storage medium 2022, and computer-readable storage medium reader 2020. The system memory 2010 may store program instructions that can be loaded and executed by the processing unit 2004. The system memory 2010 may also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs may be loaded into the system memory 2010, including but not limited to client applications, web browsers, middle-tier applications, relational database management systems (RDBMSs), virtual machines, containers, etc.

[0287] The system memory 2010 may also store an operating system 2016. Examples of the operating system 2016 may include various versions of Microsoft Apple and / or Linux operating systems, various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems (such as iOS, Phone, OS, OS, and OS operating systems). In certain embodiments where the computer system 2000 executes one or more virtual machines, the virtual machines along with their guest operating systems (GOSs) may be loaded into the system memory 2010 and executed by one or more processors or cores of the processing unit 2004.

[0288] The system memory 2010 may have different configurations depending on the type of the computer system 2000. For example, the system memory 2010 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, the system memory 2010 may include a basic input / output system (BIOS) that contains basic routines that help transfer information between components within the computer system 2000, such as during startup.

[0289] The computer-readable storage medium 2022 can represent a remote, local, fixed, and / or removable storage device and a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 2004 of the computer system 2000) for use by the computer system 2000.

[0290] The computer-readable storage medium 2022 can include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for the storage and / or transmission of information. This can include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape cartridges, tapes, magnetic disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0291] For example, the computer-readable storage medium 2022 can include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a disk drive that reads from or writes to a removable non-volatile disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk (such as a CD ROM, DVD, and disk or other optical medium). The computer-readable storage medium 2022 can include, but is not limited to, drives, flash cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital audio tapes, and so on. The computer-readable storage medium 2022 can also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), SSDs based on volatile memory (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 2000.

[0292] Machine-readable instructions executable by one or more processors or cores of the processing unit 2004 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, which include volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0293] The communication subsystem 2024 provides an interface to other computer systems and networks. The communication subsystem 2024 serves as an interface for receiving data from other systems and sending data from the computer system 2000 to other systems. For example, the communication subsystem 2024 may enable the computer system 2000 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2024 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution) advanced data network technologies, WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, as an addition or alternative to the wireless interface, the communication subsystem 2024 may provide a wired network connection (e.g., Ethernet).

[0294] In some embodiments, the communication subsystem 2024 may also receive input communications in the form of structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc., on behalf of one or more users who may use the computer system 2000.

[0295] For example, the communication subsystem 2024 may be configured to receive data feeds 2026 from users of social networks and / or other communication services in real time, such as feeds, updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0296] In addition, the communication subsystem 2024 can also be configured to receive data in the form of a continuous data stream, which can include an event stream 2028 and / or event updates 2030 of real-time events that can be continuous or unbounded in nature and have no explicit termination. Examples of applications that produce continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and so on.

[0297] The communication subsystem 2024 can also be configured to output structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 2000.

[0298] The computer system 2000 can be one of various types, including handheld portable devices (e.g., cellular phones, computing tablets, PDAs), wearable devices (e.g., Glass head-mounted displays), PCs, workstations, mainframes, information kiosks, server racks, or any other data processing system.

[0299] Due to the ever-changing nature of computers and networks, the description of the computer system 2000 depicted in the figure is merely to serve as a specific example. Many other configurations with more or fewer components than the system depicted in the figure are possible. For example, custom hardware can also be used and / or specific elements can be implemented in hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices such as network input / output devices can also be employed. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will recognize other ways and / or methods of implementing the various embodiments.

[0300] Embodiments can be implemented by using a computer program product, including computer programs / instructions that, when executed by a processor, cause the processor to perform any method described in this disclosure.

[0301] Although specific embodiments have been described, various modifications, changes, alternative constructs, and equivalent forms are also included within the scope of this disclosure. Embodiments are not limited to operating within certain specific data processing environments but can operate freely within multiple data processing environments. Additionally, although embodiments have been described using a specific series of transactions and steps, those skilled in the art should appreciate that the scope of this disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above embodiments can be used alone or in combination.

[0302] In addition, although embodiments have been described using a specific combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the present disclosure. Embodiments can be implemented using only hardware, or only software, or using a combination thereof. The various processes described herein can be implemented in the same processor or in different processors in any combination. Accordingly, in cases where a component or service is described as being configured to perform certain operations, such configuration can be accomplished by, for example, designing electronic circuitry to perform the operations, programming a programmable electronic circuit (such as a microprocessor) to perform the operations, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes can use different techniques, or the same pair of processes can use different techniques at different times.

[0303] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is obvious that additions, subtractions, deletions, and other modifications and changes can be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalent forms are within the scope of the following claims.

[0304] In the context of describing the disclosed embodiments (especially in the context of the following claims), the terms "a", "an", "the", and similar references are to be construed to cover both the singular and the plural unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms "comprising", "having", "including", and "containing" are to be construed as open-ended terms (i.e., meaning "including but not limited to"). The term "connected" shall be construed to mean partly or wholly incorporated in, attached to, or joined together, even if there is something in between. Unless otherwise indicated herein, the recitation of a range of values herein is merely intended to be a shorthand method of individually referring to each separate value falling within the range, and each separate value is incorporated into the specification as if it were individually recited herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., "such as") provided herein is merely intended to better illuminate the embodiments and does not pose a limitation on the scope of the present disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the present disclosure.

[0305] Disjunctive language, such as the phrase "at least one of X, Y, or Z," unless otherwise expressly specified, is intended to be understood in the context of generally representing items, terms, etc., and can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language generally is not intended to and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0306] This disclosure describes preferred embodiments of the present disclosure, including the known best mode for carrying out the present disclosure. Variations of these preferred embodiments may become apparent to those of ordinary skill in the art after reading the foregoing description. Those of ordinary skill in the art should be able to appropriately adopt such variations and practice the present disclosure in a manner different from that specifically described herein. Accordingly, the present disclosure includes all modifications and equivalent forms of the subject matter recited in the appended claims as permitted by applicable law. Moreover, unless otherwise indicated herein, the present disclosure includes any combination of the above elements in all possible variations thereof.

[0307] Example embodiments of the present disclosure may be described in accordance with the following clauses.

[0308] Clause 1. A method is disclosed. The method may include a computer system identifying multiple corresponding sets of workloads executed on each of a plurality of hosts. The method may include a computer system identifying a plurality of response levels that specify the applicability of a corresponding set of reduction actions to the plurality of hosts. In some embodiments, a first response level of the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of resources. The method may include determining a first estimate of power reduction caused by the first response level based at least on (a) the multiple corresponding sets of workloads executed on each of the plurality of hosts and (b) the applicability of the first set of reduction actions to the plurality of hosts according to the first response level. The method may include selecting the first response level from the plurality of response levels based at least on the first estimate of power reduction caused by the first response level. The method may include causing the first set of reduction actions to be applied to the plurality of hosts according to the selected first response level.

[0309] Clause 2. The method of Clause 1, further including determining a second estimate of power reduction caused by a second response level of the plurality of response levels based at least on (a) the multiple corresponding sets of workloads executed on each of the plurality of hosts and (b) the applicability of a second set of reduction actions to the plurality of hosts according to the second response level. In some embodiments, selecting the first response level from the plurality of response levels is based at least on the first estimate of power reduction caused by the first response level.

[0310] Clause 3. The method of Clause 1 or 2, wherein determining a first estimate of the power reduction caused by the first response level includes: 1) determining that a first set of reduction actions is applicable to a first subset of the plurality of hosts, 2) identifying multiple corresponding sets of workloads executed on each host in the first subset of hosts, and 3) determining the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts.

[0311] Clause 4. The method of Clause 3, wherein determining the first estimate of the power reduction caused by the first response level further includes determining the sum of the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts as the first estimate of the power reduction caused by the first response level.

[0312] Clause 5. The method of Clause 3, wherein determining the first estimate of the power reduction caused by the first response level further includes: 1) determining the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts after applying the first set of reduction actions to the first subset of hosts, and 2) determining the first estimate of the power reduction caused by the first response level based on (a) the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts and (b) the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts after applying the first set of reduction actions to the first subset of hosts.

[0313] Clause 6. The method of any one of Clauses 1-5, further includes: 1) determining the difference between the current value of the total power consumption of the plurality of hosts and the current value of the total power threshold of the plurality of hosts, and 2) determining that the first value of the power reduction caused by the first response level is greater than the difference. In some embodiments, the first response level is selected at least based on determining that the first value of the power reduction caused by the first response level is greater than the difference.

[0314] Clause 7. The method of any one of Clauses 1-6, wherein the multiple corresponding sets of workloads executed on each host in the plurality of hosts and the multiple response levels specifying the applicability of the corresponding set of reduction actions to the plurality of hosts are at least partially identified based on at least one of the following: 1) identifying degradation or failure of a temperature control system of a physical environment including the plurality of hosts, or 2) identifying a government reduction in power supply, or 3) identifying an increase in external temperature.

[0315] Clause 8. Discloses a method. The method may include a computer system identifying multiple corresponding sets of workloads executed on each of multiple hosts. The method may include a computer system identifying multiple response levels, the multiple response levels specifying the applicability of a corresponding set of reduction actions to the multiple hosts. In some embodiments, a first response level of the multiple response levels specifies the applicability of a first set of reduction actions to the multiple resources such that a first reduction action of the first set of reduction actions is applicable to a first resource of the multiple resources. The method may include determining a first estimate of a predicted impact caused by the first response level based at least on (a) the multiple corresponding sets of workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level. In some embodiments, the workload impact is determined based on at least one of the priority of the host, the number of affected hosts, the number of affected workloads, the priority level of the affected workloads, the number of affected customers, or the priority level of the affected customers. The method may include selecting the first response level from the multiple response levels based at least on the first estimate of the workload impact caused by the first response level. The method may include causing the first set of reduction actions to be applied to the multiple hosts according to the selected first response level.

[0316] Clause 9. The method of Clause 8, further including determining a second estimate of a workload impact caused by a second response level of the multiple response levels based at least on (a) the multiple corresponding sets of workloads executed on each of the multiple hosts and (b) the applicability of a second set of reduction actions to the multiple hosts according to the second response level. In some embodiments, selecting the first response level from the multiple response levels is based at least on the first estimate of the workload impact caused by the first response level and the second estimate of the workload impact caused by the second response level.

[0317] Clause 10. The method of Clause 9, wherein the first set of reduction actions is more stringent than the second set of reduction actions, and the first estimate of the workload impact caused by the first response level is less than the second estimate of the workload impact caused by the second response level.

[0318] Clause 11. The method of any one of Clauses 8 - 10, wherein determining the first estimate of the workload impact caused by the first response level includes: 1) determining that the first set of reduction actions is applicable to a first subset of the multiple hosts, 2) identifying the multiple corresponding sets of workloads executed on each host in the first subset of hosts as the affected workloads caused by the first response level, and 3) determining at least one of the following: (a) the number of affected workloads caused by the first response level, or (b) the type of priority of the affected workloads caused by the first response level.

[0319] Clause 12. The method of any one of Clauses 8 - 11, wherein determining a first estimate of the workload impact caused by a first response level includes: 1) determining that a first set of reduction actions is applicable to a first subset of the plurality of hosts, 2) identifying multiple corresponding sets of workloads executed on each host in the first subset of hosts, 3) identifying the corresponding customers of the multiple corresponding sets of workloads executed on each host in the first subset of hosts as the affected customers caused by the first response level, 4) determining at least one of the following: (a) the number of affected customers caused by the first response level, or (b) the type of priority of the affected customers caused by the first response level.

[0320] Clause 13. The method of any one of Clauses 8 - 12, further comprising determining a second estimate of the power reduction caused by the first response level based at least on (a) the multiple corresponding sets of workloads executed on each host in the plurality of hosts and (b) the applicability of the first set of reduction actions to the plurality of hosts according to the first response level. In some embodiments, the first response level is selected from the plurality of response levels based at least on the first estimate of the workload impact caused by the first response level and the second estimate of the power reduction caused by the first response level.

[0321] Clause 14. The method of Clause 13, wherein the selected first response level is the response level among the plurality of response levels associated with the minimum workload impact, and the power reduction achieved by the minimum workload impact is greater than or equal to the difference between the current value of the total power consumption of the plurality of hosts and the current value of the total power threshold of the plurality of hosts.

[0322] Clause 15. A method is disclosed. The method may include a computer system determining multiple corresponding predicted workloads to be executed on each of multiple hosts during a future time period. The method may include a computer system identifying multiple response levels that specify the applicability of a corresponding set of reduction actions to the multiple hosts. In some embodiments, a first response level of the multiple response levels specifies the applicability of a first set of reduction actions to the multiple resources. The method may include determining a first estimate of power reduction caused by the first response level based at least on (a) the multiple corresponding predicted workloads to be executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level. The method may include selecting the first response level from the multiple response levels based at least on the first estimate of power reduction caused by the first response level. The method may include identifying one or more workloads that (a) are currently being executed on the multiple hosts and (b) will be affected by applying the first set of reduction actions to the multiple hosts according to the selected first response level. The method may include migrating the affected workloads from the multiple hosts to one or more other hosts prior to the future time period.

[0323] Clause 16. The method of clause 15, wherein determining the multiple corresponding predicted workloads to be executed on each of the multiple hosts during the future time period is based on historical patterns of workloads executed on the multiple hosts.

[0324] Clause 17. The method of clause 15 or 16, wherein determining the multiple corresponding predicted workloads to be executed on each of the multiple hosts during the future time period is performed using a machine learning model that has been previously trained using a supervised learning algorithm to predict a set of workloads based at least in part on historical workload data provided as training data.

[0325] Clause 18. The method of any of clauses 15 - 17, further including a computer system obtaining a predicted value of the total power consumption of the multiple hosts during the future time period. The method may further include a computer system obtaining a predicted value of the total power threshold of the multiple hosts during the future time period. The method may further include, in response to determining that the predicted value of the total power consumption exceeds the predicted value of the total power threshold, making a selection from the multiple response levels.

[0326] Clause 19. The method of any one of Clauses 15 - 18 further includes identifying, by a computer system, a predicted failure of a temperature control system associated with the plurality of hosts in a future time period. The method may further include obtaining, by the computer system, a predicted value of a total power threshold for the plurality of hosts in the future time period, at least in part based on identifying the predicted failure. The method may further include selecting, in response to identifying the predicted failure, from the plurality of response levels.

[0327] Clause 20. The method of Clause 19, wherein identifying the predicted failure of the temperature control system utilizes a machine learning model that has been previously trained using training data including historical data associated with the plurality of hosts, and the machine learning model is trained using a supervised learning algorithm to identify the predicted failure from input data.

[0328] Clause 21. The method of any one of Clauses 15 - 20, wherein the historical data of the training data includes temperature control system data, historical power consumption data corresponding to a set of hosts, and historical total power thresholds.

[0329] Clause 22. A system is disclosed. The system may include a memory configured to store instructions and one or more processors configured to execute the instructions to perform the method of any one of Clauses 1 - 21.

[0330] Clause 23. A non - transitory computer - readable medium is disclosed. Instructions may be stored on the non - transitory computer - readable medium, and when executed by a processor, cause the processor to perform the method of any one of Clauses 1 - 21.

[0331] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in full herein.

[0332] In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects disclosed above may be used singly or in combination. Additionally, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method, comprising: identifying, by a computer system, multiple corresponding sets of workloads executed on each of multiple hosts; identifying, by the computer system, multiple response levels, the multiple response levels specifying the applicability of a corresponding set of reduction actions to the multiple hosts; wherein a first response level among the multiple response levels specifies the applicability of a first set of reduction actions to the multiple resources; determining a first estimate of power reduction caused by the first response level based at least on (a) the multiple corresponding sets of workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level; selecting the first response level from the multiple response levels based at least on the first estimate of power reduction caused by the first response level; and applying the first set of reduction actions to the multiple hosts according to the selected first response level.

2. The method according to claim 1, further comprising: determining a second estimate of power reduction caused by a second response level among the multiple response levels based at least on (a) the multiple corresponding sets of workloads executed on each of the multiple hosts and (b) the applicability of a second set of reduction actions to the multiple hosts according to the second response level; wherein selecting the first response level from the multiple response levels is based at least on the first estimate of power reduction caused by the first response level.

3. The method according to claim 1 or 2, wherein determining the first estimate of power reduction caused by the first response level comprises: determining a first subset of the multiple hosts to which the first set of reduction actions is applicable; identifying the multiple corresponding sets of workloads executed on each host in the first subset of hosts; and determining the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts.

4. The method according to claim 3, wherein determining the first estimate of power reduction caused by the first response level further comprises: determining the sum of the corresponding power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts as the first estimate of power reduction caused by the first response level.

5. The method according to claim 3, wherein determining the first estimate of power reduction caused by the first response level further comprises: after applying the first set of reduction actions to the first subset of hosts, determining the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts; and determining the first estimate of power reduction caused by the first response level based on (a) the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts and (b) the corresponding estimated power consumption of the multiple corresponding sets of workloads executed on each host in the first subset of hosts after applying the first set of reduction actions to the first subset of hosts.

6. The method according to any one of claims 1-5, further comprising: determining the difference between the current value of the total power consumption of the multiple hosts and the current value of the total power threshold of the multiple hosts; and Determine that a first value of the power reduction caused by the first response level is greater than the difference; Wherein the first response level is selected at least based on determining that a first value of the power reduction caused by the first response level is greater than the difference.

7. The method according to any one of claims 1-6, wherein the multiple sets of corresponding workloads executed on each of the multiple hosts and the multiple response levels specifying the applicability of a corresponding set of reduction actions to the multiple hosts are at least partially identified based on at least one of the following: Identify degradation or failure of a temperature control system of a physical environment including the multiple hosts; Identify a government's reduction of power supply; or Identify an increase in external temperature.

8. A method, Comprising: Identifying, by a computer system, multiple sets of corresponding workloads executed on each of multiple hosts; Identifying, by a computer system, multiple response levels that specify the applicability of a corresponding set of reduction actions to the multiple hosts; Wherein a first response level among the multiple response levels specifies the applicability of a first set of reduction actions to the multiple resources, such that a first reduction action in the first set of reduction actions is applicable to a first resource among the multiple resources; Determining a first estimate of a predicted impact caused by the first response level at least based on (a) multiple sets of corresponding workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level; Wherein the workload impact is determined based on at least one of the priority of the host, the number of affected hosts, the number of affected workloads, the priority level of the affected workloads, the number of affected customers, or the priority level of the affected customers; Selecting the first response level from the multiple response levels at least based on a first estimate of the workload impact caused by the first response level; And Applying the first set of reduction actions to the multiple hosts according to the selected first response level.

9. The method according to claim 8, further Comprising: Determining a second estimate of the workload impact caused by a second response level among the multiple response levels at least based on (a) multiple sets of corresponding workloads executed on each of the multiple hosts and (b) the applicability of a second set of reduction actions to the multiple hosts according to the second response level; Wherein the first response level is selected from the multiple response levels at least based on a first estimate of the workload impact caused by the first response level and a second estimate of the workload impact caused by the second response level.

10. The method according to claim 9, wherein the first set of reduction actions is more stringent than the second set of reduction actions, and a first estimate of the workload impact caused by the first response level is less than a second estimate of the workload impact caused by the second response level.

11. The method according to any one of claims 8-10, wherein determining a first estimate of the workload impact caused by the first response level Comprises: Determining that the first set of reduction actions is applicable to a first subset of the multiple hosts; Identify multiple corresponding workloads executed on each host in a first subset of the hosts as affected workloads caused by a first response level; and Determine at least one of the following: (a) the number of affected workloads caused by the first response level, or (b) the priority type of the affected workloads caused by the first response level.

12. The method according to any one of claims 8-11, wherein determining a first estimate of the workload impact caused by the first response level comprises: Determine that a first set of reduction actions is applicable to the first subset of the multiple hosts; Identify multiple corresponding workloads executed on each host in the first subset of the hosts; Identify the corresponding customers of the multiple corresponding workloads executed on each host in the first subset of the hosts as affected customers caused by the first response level; and Determine at least one of the following (a) the number of affected customers caused by the first response level, or (b) the priority type of the affected customers caused by the first response level.

13. The method according to any one of claims 8-12, further comprises: Determine a second estimate of the power reduction caused by the first response level based at least on (a) the multiple corresponding workloads executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level; wherein the first response level is selected from the multiple response levels based at least on the first estimate of the workload impact caused by the first response level and the second estimate of the power reduction caused by the first response level.

14. The method according to claim 13, wherein the selected first response level is the response level among the multiple response levels associated with the minimum workload impact, and the power reduction achieved by the minimum workload impact is greater than or equal to the difference between the current value of the total power consumption of the multiple hosts and the current value of the total power threshold of the multiple hosts.

15. A method, comprises: Determine, by a computer system, multiple corresponding predicted workloads to be executed on each of multiple hosts during a future time period; Identify, by the computer system, multiple response levels that specify the applicability of a corresponding set of reduction actions to the multiple hosts; wherein a first response level among the multiple response levels specifies the applicability of a first set of reduction actions to the multiple resources; Determine a first estimate of the power reduction caused by the first response level based at least on (a) the multiple corresponding predicted workloads to be executed on each of the multiple hosts and (b) the applicability of the first set of reduction actions to the multiple hosts according to the first response level; Select the first response level from the multiple response levels based at least on the first estimate of the power reduction caused by the first response level; Identify one or more workloads that (a) are currently being executed on the multiple hosts and (b) will be affected by applying the first set of reduction actions to the multiple hosts according to the selected first response level; and Prior to a future time period, migrate affected workloads from the plurality of hosts to one or more other hosts in advance.

16. The method according to claim 15, wherein determining the respective sets of predicted workloads to be executed on each of the plurality of hosts during a future time period is based on historical patterns of workloads executed on the plurality of hosts.

17. The method according to claim 15 or 16, wherein determining the respective sets of predicted workloads to be executed on each of the plurality of hosts during a future time period is performed using a machine learning model that has been previously trained using a supervised learning algorithm to predict a set of workloads based at least in part on historical workload data provided as training data.

18. The method according to any one of claims 15-17, further comprising: obtaining, by a computer system, a predicted value of the total power consumption of the plurality of hosts during a future time period; obtaining, by a computer system, a predicted value of the total power threshold of the plurality of hosts during a future time period; and selecting, in response to determining that the predicted value of the total power consumption exceeds the predicted value of the total power threshold, from the plurality of response levels.

19. The method according to any one of claims 15-18, further comprising: identifying, by a computer system, a predicted failure of a temperature control system associated with the plurality of hosts during a future time period; obtaining, by a computer system, a predicted value of the total power threshold of the plurality of hosts during a future time period based at least in part on identifying the predicted failure; and selecting, in response to identifying the predicted failure, from the plurality of response levels.

20. The method according to claim 19, wherein identifying the predicted failure of the temperature control system utilizes a machine learning model that has been previously trained using training data including historical data associated with the plurality of hosts, and the machine learning model is trained using a supervised learning algorithm to identify the predicted failure from input data.

21. The method according to any one of claims 15-20, wherein the historical data of the training data includes temperature control system data, historical power consumption data corresponding to a set of hosts, and historical total power thresholds.

22. A system, comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform the method according to any one of claims 1-21.

23. A non-transitory computer-readable medium having instructions stored thereon, which when executed by a processor cause the processor to perform the method according to any one of claims 1-21.

Images