Techniques for orchestrating load shedding

Abstract

The disclosed technology relates to orchestrating power consumption reduction across multiple hosts. Power consumption of a power consuming device (e.g., a host, server, etc.) may be monitored relative to a power threshold. When the current power consumption corresponding to the devices breaks through a power threshold, or at any appropriate time, the system may identify a set of reduction actions configured to reduce the total power consumption. The power threshold may be dynamically updated based on operating conditions and environmental factors of the related system. A plurality of response levels may be used, each response level associated with a corresponding set of reduction actions. An impact on the customer, the host, and / or the workload may be calculated at runtime based on the current condition and the workload, and a particular level of response may be selected based on the calculated impact. These techniques enable sufficient but minimal impact responses to be employed.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 423,762, titled "Orchestrated DC - Scale Load Shedding", filed on November 8, 2022, U.S. Provisional Application No. 63 / 439,576, titled "Orchestrated DC - Scale Load Shedding", filed on January 18, 2023, and U.S. Non - Provisional Application No. 18 / 338,695, titled "Techniques for Orchestrated Load Shedding", filed on June 21, 2023. The contents of these applications are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to techniques for orchestrating the reduction of power consumption in a data center. The disclosed systems, methods, devices, and services implement dynamic orchestration methods to limit power consumption within a data center, including but not limited to leveraging power caps, suspending hosts, migrating instances and / or hosts, and / or shutting down hosts to achieve a desired power reduction. Background Art

[0004] Data centers are configured with a power infrastructure that provides various security features in accordance with a hierarchical power distribution. Power is provided by a local power utility and is distributed to various components of the data center according to this power distribution hierarchy, including power distribution units (PDUs) (e.g., transformers, switchboards, bus ducts, rack PDUs, etc.) and power - consuming devices (e.g., servers, network devices, etc.). This ensures that the power consumed by all downstream devices complies with the power limits of each upstream device. When demand peaks and / or when components fail, the data center may no longer be able to effectively handle the demand, which may lead to widespread power failures, causing at least severe damage to downstream devices, resulting in a reduction in processing capacity within the data center and / or a data center power failure, which in turn leads to a poor user experience. In the worst - case scenario, a power failure in one data center may trigger cascading power failures of other devices within the same data center or other data centers as workloads are redistributed in an attempt to recover from the initial power failure. Additionally, in some embodiments, external factors (e.g., government regulations) may require reducing the power consumption of a particular data center within a specific time.

[0005] Conventional methods of managing load shedding include manual tasks where an operator shuts down host racks or individual devices one by one to reduce power consumption. Alternatively, conventional techniques include pressing an emergency shutdown switch or button that is configured to turn off the power to an entire data center or a majority of the data center. The operator determining which components to shut down typically does not have knowledge of the workloads or customers affected by these actions or the extent of the impact of taking these actions. Conventional methods like those mentioned here result in suboptimal approaches to affected customers and workloads. More sophisticated methods may have a lesser impact on customers and / or equipment / workloads while still providing the desired power reduction for the situation. Accordingly, it is desirable to improve power management techniques, particularly those related to load shedding, to reduce power consumption to a sufficient impact and amount and avoid the drawbacks of conventional methods. SUMMARY OF THE INVENTION

[0006] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is apparent that the various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

[0007] Some embodiments may include a method. The method may include obtaining, by a computer system, configuration data that includes a plurality of response levels that specify the applicability of a corresponding set of reduction actions to a plurality of hosts. In some embodiments, a first response level of the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts. The method may include obtaining, by the computer system, a current value of a total power consumption of the plurality of hosts. The method may further include obtaining, by the computer system, a current value of a total power threshold of the plurality of hosts. The method may further include selecting, at least based on a difference between the current value of the total power consumption and the current value of the total power threshold, the first response level from the plurality of response levels. The method may further include applying, to at least one host of the plurality of hosts, the first set of reduction actions according to the selected first response level. In some embodiments, applying the first set of reduction actions is effected by performing operations or instructing components of the computer system to perform operations for applying the first set of reduction actions to the plurality of hosts.

[0008] In some embodiments, the method includes performing, by the computer system, the first set of reduction actions for the plurality of hosts according to the selected first response level.

[0009] In some embodiments, the method includes determining whether the current value of the total power consumption exceeds the current value of the total power threshold and, in response to determining that the current value of the total power consumption exceeds the current value of the total power threshold, making a selection from the plurality of response levels. In some embodiments, the selected first response level is applicable to a subset of the plurality of hosts.

[0010] In some embodiments, the first response level specifies the applicability of a first set of reduction actions to multiple hosts based at least on host attributes. In some embodiments, host attributes may include any suitable combination of priority levels, types, or categories.

[0011] In some embodiments, each successive response level among multiple response levels specifies multiple successive sets of stringent reduction actions to be applied to multiple hosts.

[0012] In some embodiments, a first power ceiling value applied to a first host by a first reduction action according to a first response level is lower than a second power ceiling value applied to the first host by a different reduction action according to a second response level.

[0013] In some embodiments, the first response level specifies applying a first reduction action to a subset of multiple hosts associated with a priority level lower than a first level, and the second response level specifies applying a second reduction action to a second subset of multiple hosts associated with a priority level lower than a second level. In some embodiments, the second level is higher than the first level. In some embodiments, the second subset of multiple hosts does not include the first subset of multiple hosts.

[0014] In some embodiments, the selected first response level is the response level among multiple response levels associated with the least stringent set of reduction actions that achieves a new value of total power consumption that is less than or equal to the current value of a total power threshold.

[0015] In some embodiments, selecting the first response level from multiple response levels based at least on a difference between the current value of total power consumption and the current value of the total power threshold may include at least one of the following: 1) determining that a first reduction in total power consumption resulting from the first response level is greater than the difference; 2) determining that a second reduction in total power consumption resulting from a second response level is less than the difference; and / or 3) selecting the first response level.

[0016] In some embodiments, due to a decrease in the total power threshold, the current value of total power consumption has breached the current value of the total power threshold. In some embodiments, the current value of total power consumption has exceeded the current value of the total power threshold for a period of time.

[0017] In some embodiments, the decrease in the total power threshold is caused by at least one of the following: 1) degradation or failure of a temperature control system of a physical environment including multiple hosts, 2) government requirements to reduce power consumption, and / or 3) an increase in external temperature.

[0018] In some embodiments, the method may include determining that a first value of a total power threshold exceeds a current value of total power consumption during a first time period. The method may include determining that the current value of total power consumption exceeds the current value of the total power threshold during a current time period after the first time period. In some embodiments, the current value of the total power threshold during the current time period may be less than the first value of the total power threshold during the first time period.

[0019] In some embodiments, the first reduction action includes at least one of the following: 1) enforcing a power ceiling on the host; 2) migrating a workload from the host; or 3) shutting down or pausing the host.

[0020] In some embodiments, the method may include obtaining, by a computer system, a new value of a total power threshold. The method may include performing a recovery from a first response level based at least on the new value of the total power threshold.

[0021] In some embodiments, performing a recovery from a second response level includes at least one of the following: 1) stopping power capping on the host, 2) migrating the workload back to the host, or 3) restarting a host that was previously shut down or paused.

[0022] In some embodiments, in response to receiving user input at a user interface, a first response level among a plurality of response levels is performed. In some embodiments, the selected first response level is associated with a reduction amount that exceeds a difference between a current value of total power consumption and a current value of a total power threshold.

[0023] In some embodiments, another method is disclosed. The method may include obtaining, by a computer system, configuration data that includes a plurality of response levels that specify applicability of a corresponding set of reduction actions to a plurality of hosts. In some embodiments, a first response level among the plurality of response levels specifies applicability of a first set of reduction actions to the plurality of hosts. The method may further include obtaining, by the computer system, a predicted value of total power consumption of the plurality of hosts during a future time period, and the method may further include obtaining, by the computer system, a predicted value of a total power threshold of the plurality of hosts during the future time period. The method may further include selecting a first response level from the plurality of response levels based at least on a predicted difference between the predicted value of total power consumption and the predicted value of the total power threshold. The method may further include identifying, according to the selected first response level, one or more workloads that (a) are currently executing on the plurality of hosts and (b) will be affected by applying the first set of reduction actions to the plurality of hosts. The method may further include migrating, prior to the future time period, the affected workloads from the plurality of hosts to one or more other hosts.

[0024] In some embodiments, the method may further include any suitable combination of the following: 1) determining a predicted value of the total power threshold based on at least one of: the health of a temperature control system of a physical environment including a plurality of hosts, or 2) an announced or foreseen government power supply reduction, or 3) a predicted increase in external temperature.

[0025] In some embodiments, the method may further include determining a predicted value of the total power consumption based on historical patterns of the total power consumption.

[0026] In some embodiments, identifying one or more workloads that (a) are currently executing on a plurality of hosts and (b) are affected by applying a first set of reduction actions to the plurality of hosts according to a selected first response level includes: 1) determining that a specific action will be applied to a first host according to a selected second response level; 2) determining that a first workload is currently executing on the first host; and / or 3) determining that the first workload will be affected by applying the first set of reduction actions to the plurality of hosts according to the selected first response level.

[0027] Systems, devices, and computer media are disclosed, where each of the systems, devices, and computer media may include one or more memories, and instructions corresponding to the methods disclosed herein may be stored therein. These instructions may be executed by one or more processors of the disclosed systems and devices to perform the methods disclosed herein. One or more computer programs may be configured to perform specific operations or actions corresponding to the described methods by including instructions that, when executed by a data processing device, cause the device to perform these actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An example physical environment (e.g., a data center or a portion thereof) including various components is depicted in accordance with at least one embodiment.

[0029] Figure 2 A simplified diagram of an exemplary power distribution infrastructure including various components of a data center is shown in accordance with at least one embodiment.

[0030] Figure 3 An example architecture of an exemplary power orchestration system is illustrated in accordance with at least one embodiment, the power orchestration system being configured to orchestrate power consumption reduction of various resources applicable to a physical environment (e.g., a data center, a room of a data center, etc.).

[0031] Figure 4 An example architecture of a power management service for detecting and constraining excessive power consumption is illustrated in accordance with at least one embodiment.

[0032] Figure 5Illustrates an example power distribution hierarchy corresponding to the arrangement of components of Figure 2 in accordance with at least one embodiment.

[0033] Figure 6 Is a flowchart illustrating an example method for managing excessive power consumption in accordance with at least one embodiment.

[0034] Figure 7 Illustrates an example architecture of a VEPO orchestration service for orchestrating power consumption constraints and / or power-off tasks in accordance with at least one embodiment.

[0035] Figure 8 Is a table illustrating a set of example response levels and multiple sets of corresponding reduction actions for each response level in accordance with at least one embodiment.

[0036] Figure 9 Is a diagram illustrating an example scope of components affected by one or more response levels used by a VEPO orchestration service for orchestrating power consumption constraints and / or power-off tasks in accordance with at least one embodiment.

[0037] Figure 10 Is a block diagram illustrating an example use case of applying multiple response levels based on the current conditions of a set of hosts in accordance with at least one embodiment.

[0038] Figure 11 Is a schematic diagram of an example user interface in accordance with at least one embodiment.

[0039] Figure 12 Illustrates the flow of an example method for training one or more machine learning models in accordance with at least one embodiment;

[0040] Figure 13 Is a block diagram illustrating an example method for causing the application of a corresponding set of reduction actions associated with a response level in accordance with at least one embodiment;

[0041] Figure 14 Is a block diagram illustrating an example method for pre-migrating workloads affected by a selected response level in accordance with at least one embodiment;

[0042] Figure 15 Is a block diagram illustrating a mode for implementing a cloud infrastructure as a service system in accordance with at least one embodiment.

[0043] Figure 16 Is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system in accordance with at least one embodiment.

[0044] Figure 17is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0045] Figure 18 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0046] Figure 19 is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Description

[0047] In the following description, various embodiments will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, those skilled in the art will also understand that the embodiments may be practiced without specific details. Additionally, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0048] The present disclosure relates to managing power consumption and orchestrating power consumption reduction within an environment (e.g., a host or a power distribution unit of a data center, or portions thereof). More particularly, techniques for enabling orchestrated load shedding to achieve power consumption reduction due to current conditions and according to a total power threshold are described.

[0049] Balancing power supply and consumption within a data center is desirable. When the consumed power exceeds the available supply, the balance can be restored by increasing the power supply or reducing the power consumption rate. If this balance is not maintained and components attempt to consume more power than is available, then circuit breakers may trip, disconnecting the components from the power source.

[0050] The maximum total power threshold for a particular environment may depend on many factors, including but not limited to the current power consumption value at a host within the environment, the operating conditions of associated components (such as temperature control systems (e.g., HVAC, coolers, etc.)), environmental conditions (e.g., ambient temperature, external temperature outside the environment), etc. Conventional systems are not configured to manage power consumption based on these factors. When power consumption peaks and power outages are impending, conventional techniques mainly utilize manual efforts to shut down or suspend hosts. This results in suboptimal load shedding strategies because the operators implementing the load shedding typically do not understand the impact of their actions. For example, when choosing which devices to shut down or suspend, the operator rarely (if ever) knows the customers, workloads, instances, or hosts or their respective priorities. A power outage may cause interruptions to the operations performed by components in a data center. For example, if a circuit breaker trips and disconnects a server from the power supply of the data center, the websites hosted on the data center server will crash. Suboptimal load shedding strategies may be inefficient and may potentially exceed the amount of power reduction sufficient to reduce the total power consumption to the desired amount. Additionally or alternatively, suboptimal load shedding strategies may have a broader / greater impact on the underlying hosts, instances, workloads, and corresponding customers than might be expected.

[0051] The balance between power supply and power consumption in a data center can be managed by maintaining a balance between available supply and demand and / or consumption. Sometimes, it may not be feasible to expand the power supply in a data center because power is typically statically allocated in long-term contracts with power utilities. Although the supply may be statically allocated, the power consumption in a data center may vary, sometimes significantly. As a simple example, when the number of threads executed by a server's processor increases, the power consumption of the server may increase. The ambient temperature within a data center affects the power consumed by the data center's cooling system. When the cooling system works harder, it consumes more power as it tries to reduce the ambient temperature experienced in the data center. When the cooling system fails, the data center may no longer be able to tolerate the heat generated by the current power consumption level. The demand caused by some components in a data center (such as uninterruptible power supplies, power distribution units, cooling systems, or bus ducts) may be difficult to adjust. However, some components (such as servers, virtual machines, and / or bare-metal instances) may be more easily constrained. Additionally, workloads and / or instances may be migrated to other hosts and / or instances to concentrate resources on a smaller subset of hosts, resulting in more idle and / or vacant hosts. Subsequently applying power reduction to the idle / vacant hosts can ensure minimizing the impact (e.g., the number of affected hosts, customers, instances, workloads).

[0052] Many conventional power management methods utilize power capping to constrain the operation of power-consuming devices (e.g., servers, network devices, etc.) within a data center. When power capping is used, a power limit can be used to constrain the power consumed by the devices. The power limit is used to constrain (e.g., throttle) the operation at the server to ensure that the power consumption of the server does not exceed the power limit. Using power capping ensures that the allocated power limits of upstream devices are not breached, resulting in each upstream device being provisioned based on a worst-case scenario where it is assumed that each downstream device consumes its corresponding maximum allocated power. However, these downstream devices may typically consume less power than their maximum allocated power, leaving at least a portion of the power provisioned to the upstream devices unutilized. These methods waste valuable power and limit the density of power-consuming devices that can be used within the data center.

[0053] An efficient power infrastructure within a data center is necessary for increasing a provider's profit margin, managing scarce power resources, and making the services provided by the data center more environmentally friendly. A data center that includes components of a hosted multi-tenant environment (e.g., a public cloud) may experience above-average consumption because not all cloud tenants use it simultaneously. To push power consumption closer to the statically allocated capacity to increase the efficiency and resource utilization of the data center, a data center provider can increase servers and / or leases such that the power consumption across all power-consuming devices is closer to the allocated power capacity of the data center. However, in some cases, narrowing the gap between the allocated power capacity and power consumption increases the risk of circuit breaker tripping and the loss of the ability to utilize computing resources. The techniques described herein minimize the frequency at which the operation at downstream devices is constrained, enabling these devices to utilize previously unutilized power while maintaining a high level of safety in avoiding power failures.

[0054] Technical Effects

[0055] The disclosed systems and methods provide an automated and dynamic orchestration approach to constrain power consumption (e.g., via power capping, migrating workloads, etc.), suspend hosts, migrate instances and / or hosts, and / or power off hosts to achieve a desired power reduction. In some embodiments, at least a portion of the functions performed for these actions are user-selectable and / or based on user input. The described techniques provide multiple response levels. Each response level can be associated with a set of reduction actions (referred to herein as "actions" for brevity). Each response level (referred to herein as a "level" for brevity) can provide multiple sets of increasingly stringent reduction actions to be applied. When a total power threshold is breached, a response level can be selected based on the estimated power reduction corresponding to each response level (e.g., by the system based on user input, etc.). The least stringent level and / or the level with the least impact can be selected (e.g., by the system based on user input, etc.) that, if implemented, is sufficient to cause the total power consumption of the data center to drop below the total power threshold.

[0056] The estimated impact of applying the reduction actions associated with a level can be determined at runtime based on the current attributes of the workloads implemented by virtual machine and / or bare-metal instances, the customers associated with the workloads and / or affected hosts, and the priorities corresponding to these workloads, hosts, and / or customers. The requirements (e.g., corresponding to the total power threshold) can be dynamically modified based on the operational status data and environmental data corresponding to the various components of the data center. These factors can be used to identify changes in the total power threshold (e.g., the threshold amount of total power consumption / heat / demand that the components of the data center can collectively manage) as the power management capabilities within the data center occur. Response level selection can be triggered by real-time demand changes, via a request (e.g., a user request, a request submitted by a government entity to enforce power consumption reduction, etc.), or any suitable trigger, and can be based on the impact of implementing the corresponding reduction actions.

[0057] Using the techniques described herein, the power constraints, migration tasks, and / or pause or shutdown tasks employed can be adjusted based on the current conditions and resources in a data center (e.g., hosts, instances, workloads), thereby mitigating or completely avoiding over-reduction. These techniques provide real-time capabilities for reducing the impact of power management response actions on customers, hosts, instances, and / or workloads, while ensuring effective avoidance of the risk of power failures. The systems and methods described herein provide a more efficient and effective power orchestration method than conventional systems, which can lead to a more satisfactory user experience. The specific actions to be implemented can be selected based on at least any suitable combination (e.g., automatically by the system, through user selection, etc.): 1) maximizing the overall power consumption reduction, 2) identifying / implementing the least stringent response level, 3) identifying / implementing the least impactful response level, and / or 4) identifying a response level whose estimated power reduction is sufficient to bring the current total power consumption value below the current or predicted total power threshold, while also providing the least amount of over-power reduction (e.g., more reduction than required to bring the current power consumption below the current total power threshold). Thus, these techniques provide a more flexible and intelligent method for shedding load in various contexts and / or due to various trigger events.

[0058] Figure 1 An example environment (e.g., environment 100) including various components is depicted in accordance with at least one embodiment. Environment 100 can be a physical environment, such as a data center (e.g., data center 102) or a portion thereof (e.g., a room of a data center, such as room 110A). Environment 100 can include a dedicated space for hosting any suitable number of servers, such as servers 104A-P (collectively referred to as "servers 104"), and the infrastructure for hosting these servers, such as network hardware, a cooling system (also referred to as a "temperature control system"), and storage devices. Servers 104A-P can also be referred to as "hosts". The networking hardware of data center 100 (not depicted herein) can enable remote users to interact with the servers via a network (e.g., the Internet). Any suitable number of servers 104 (e.g., 10, 14, 21, 42, etc.) can be maintained in various racks, such as racks 106A-H (collectively referred to as "racks 106"). Racks 106 can include a framework or enclosure in which a corresponding set of servers is placed and / or installed.

[0059] Various subsets of the rack 106 can be organized into groups referred to as "rows" (e.g., rows 108A-D, collectively referred to as "row 108"). In some embodiments, row 108 can include any suitable number of racks (e.g., 5, 8, 10, up to 10, etc.) that are co-located (e.g., within a threshold distance of each other). In other embodiments, a row can be an organizational unit, and the racks having a given row can be placed at different locations (not necessarily within a threshold distance of each other). As an example, row 108 can be located in a room (e.g., room 110A, room 110N, etc.). A room (e.g., room 110A) can be a partition or physical enclosure or physical environment of a building in which any suitable number of racks 106 are placed. In other embodiments, a room can be an organizational unit, and the rooms can be located at different physical locations, or multiple rooms can be located in a single partition of a building.

[0060] Various temperature control systems (e.g., (one or more) temperature control systems 112A-N, (one or more) temperature control systems 114A-N, etc.) can be configured to manage the ambient temperature in a data center or a portion thereof. As a non-limiting example, (one or more) temperature control systems 112A-N can be associated with room 110A, while (one or more) temperature control systems 114A-N can be associated with room 110N. Any suitable number of temperature control systems can be associated with the data center and / or a portion thereof. In some embodiments, these temperature control systems can be any suitable heating, ventilation, and air conditioning (HVAC) equipment (e.g., air conditioning units), coolers (e.g., chilled water circulation equipment that controls temperature by circulating a liquid such as water), etc. In some embodiments, each temperature control system can be associated with the amount of corresponding heat that it is capable of and / or configured to manage (e.g., the amount of heat generated by the corresponding power consumption of the servers 104).

[0061] Figure 2 illustrates, according to at least one embodiment, including various components (e.g., Figure 1Simplified diagram of an exemplary power distribution infrastructure 200 of components of data center 102). The power distribution infrastructure 200 can be connected to a utility power source (not shown), and power can be initially received at one or more uninterruptible power supplies (uninterruptible power supply((one or more)UPS) 202). In some embodiments, power can be received from the utility at the (one or more) UPS via a substation on site (not shown), which is configured to establish an appropriate voltage level for distributing power through the data center. The (one or more) UPS 202 can individually include dedicated batteries or generators that provide emergency power in the event of a failure of the input power source. The (one or more) UPS 202 can monitor the input power and provide backup power when a drop in the input power is detected.

[0062] The power distribution infrastructure 200 can include any appropriate number of intermediate power distribution units ((one or more)PDU) (e.g., (one or more) intermediate PDU 204), which are connected to the (one or more) UPS 202 and receive power / electricity from it. Any appropriate number of the (one or more) intermediate PDU 204 can be deployed between the UPS in the ((one or more) UPS 202) and any appropriate number of row PDUs (e.g., row PDU 206). The power distribution units (e.g., (one or more) intermediate PDU 204, row PDU 206, (one or more) rack PDUs 208, etc.) can be any appropriate devices configured to control and distribute power / electricity. Exemplary power distribution units can include, but are not limited to, main distribution boards, distribution boards, remote distribution boards, busbars, power strips, transformers, etc. Power can be provided from the (one or more) UPS 202 to the (one or more) intermediate PDU 204. The (one or more) intermediate PDU 204 can distribute power to downstream components of the power distribution infrastructure 200 (e.g., row PDU 206).

[0063] The power distribution infrastructure 200 can include any appropriate number of row power distribution units (including row PDU 206). The row PDU can include any appropriate PDU (e.g., remote distribution board, busbar / channel, etc.), which is deployed between an intermediate PDU (e.g., the PDU in the (one or more) intermediate PDU 204) and one or more rack PDUs (e.g., rack PDU 208A, rack PDU 208N, collectively referred to as "(one or more) rack PDUs 208"). A "row PDU" refers to a PDU configured to distribute power to one or more rows of devices (e.g., row 210, including servers 212A-D, collectively referred to as "servers 212"). As described above, a row (e.g., row 210) can include any appropriate number of racks (e.g., racks 214A-N, collectively referred to as "racks 214") in which the servers 212 are located.

[0064] The power distribution infrastructure 200 may include any suitable number of rack power distribution units (including (one or more) rack PDUs 208). A rack PDU may include any suitable PDU that is deployed between row PDUs (e.g., row PDU 206) and one or more servers (e.g., servers 212A, 212B, etc.) corresponding to a rack (e.g., rack 214A, Figure 1 an example of rack 106). A "rack PDU" refers to any suitable PDU that is configured to distribute power to one or more servers within a rack. A rack (e.g., rack 214A) may include any suitable number of servers 212. In some embodiments, (one or more) rack PDUs 208 may include intelligent PDUs that are also configured to monitor, manage, and control power consumption at multiple devices (e.g., rack PDU 208A, server 212A, server 212B, etc.).

[0065] Servers 212 (each an example of Figure 1 server 104) may each individually include a power controller ((one or more) power controllers 216A-D, collectively referred to as "power controller 216"). A power controller refers to any suitable hardware or software component that operates at a device (e.g., a server) and is configured to monitor and / or manage power consumption at the device. Power controllers 216 may individually monitor the power consumption of the respective servers on which they operate. Power controllers 216 may each be configured to enforce a power capping limit to constrain power consumption at the respective servers. Enforcing a power capping limit may include any suitable combination of the following: monitoring power consumption at the server, determining whether to constrain (e.g., limit, restrict, etc.) power consumption at the server (e.g., at least in part based on a comparison of the server's current power consumption with a stored power capping limit), and limiting / restricting power consumption at the server (e.g., using processor and memory dynamic voltage and frequency scaling to throttle server power consumption). Enforcing a power capping limit may be referred to as "power capping".

[0066] Figure 1 The data center 102 in may include various components depicted in the power distribution infrastructure 200. For example, room 110A may include one or more busways (each busway an example of row PDU 206). A busbar (also referred to as a "busway") refers to a conduit of conductive material that can distribute power (e.g., within room 110A). A busway may receive power from the power distribution unit of (one or more) intermediate PDUs 204 and supply power to one or more racks (e.g., Figure 1 row 108A) associated with a row (e.g.,Figure 1 rack 106A, Figure 1 rack 106B, etc.) with power. Each power infrastructure component that distributes / provides power to other components also consumes a portion of the power passing through it. This loss may be due to heat loss generated when power flows through the component or power directly consumed by the component (e.g., power consumed by a processor in a rack PDU).

[0067] Figure 3 FIG. illustrates an example architecture of an example power orchestration system 300 according to at least one embodiment, which is configured to orchestrate power consumption reduction of various resources applicable to a physical environment (e.g., a data center, a room in a data center, etc.). The term "resource" may be considered to include hosts, workloads (e.g., virtual machines and / or bare-metal instances running these workloads), and / or customers associated with these hosts and / or workloads / instances. The power orchestration system 300 may be configured to monitor power consumption levels (e.g., current individual and / or total power consumption values corresponding to various hosts such as (one or more) hosts 324, each host being an example of Figure 1 one or more) servers 104). The power orchestration system 300 may manage power consumption within the physical environment at least in part based on this monitoring such that a total power threshold is enforced. As used herein, the total power threshold represents the maximum power consumption amount that the components of the system 300 can manage under current conditions. The total power threshold may be dynamically adjusted as conditions change (e.g., based on an order from a government entity to reduce power consumption, based on environmental conditions (actual or predicted), based on current power consumption values (actual or predicted), based on the operating conditions (actual or predicted) of various components such as the temperature control system of the physical environment, etc.). The specific impact (e.g., scope, applicability) of the changes made to enforce the current total power threshold may vary depending on real-time conditions. Some example actions that may be employed to enforce the current total power threshold may be power capping (e.g., enforcing a maximum power consumption ceiling at a specific device such as one or more hosts 324) or otherwise allocating a budgeted amount of power to any appropriate device (e.g., Figure 2 one or more) hosts 324, (one or more) PDUs 202, 204, 206, 208, etc.) in, suspending or shutting down hosts and / or instances (e.g., VMs or BMs) and / or workloads running at the instances, migrating a workload from one instance to another, migrating a workload from one host to another, etc.

[0068] The power orchestration system 300 may include various components such as Figure 3The components depicted therein. For example, the power orchestration system 300 may include a VEPO orchestration service 302. The VEPO orchestration service 302 may be configured to obtain various data based on which impact and mitigation actions can be determined. By way of example, the VEPO orchestration service 302 may be configured to obtain any suitable combination of: power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318.

[0069] The power data 310 may include any suitable budget / assignment value, power ceiling value, and / or current power consumption value of one or more hosts 324 or one or more PDUs (including the rack PDU 320 ( Figure 2 an example of the rack PDU 208 of)), which indicates the amount of power currently consumed by the device. The power data 310 may be stored at a location accessible to the VEPO orchestration service 302, and / or the power data 310 may be obtained from a source and / or service configured to manage and / or obtain such data. In some embodiments, the power management service 304 may be configured to obtain the power data 310 and store such data at a location from which the VEPO orchestration service 302 can retrieve the data. In some embodiments, the power management service 304 or another component configured to obtain such data may provide the power data 310 to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or directly via a request.

[0070] The account data 312 may include any suitable attributes of the customer, any suitable attributes of one or more hosts and / or one or more instances associated with the customer, etc. Some example attributes may include the category associated with the customer account (e.g., a free tier category, which indicates that the customer is using the service in a non-paying manner that may be provided with limited functionality or resources; a free trial category, which indicates that the customer is using the service in a non-paying manner for a limited period of time). The category may correspond to the priority associated with the customer, or the priority may be assigned to the customer, host, or workload in some other way. The priority may indicate the degree of relationship importance, such as low priority, medium priority, and high priority, but other priority schemes may also be considered. Another example attribute may include an identifier associated with the customer, host, instance, and / or workload. The account data 312 may be stored at a location accessible to the VEPO orchestration service 302, and the account data 312 may be retrieved from that location, or the account data 312 may be obtained from a service and / or source configured to manage and / or obtain such data (e.g., an account service of a cloud computing environment not depicted). In some embodiments, the account data 310 is provided directly from the source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0071] Environmental data 314 may include any suitable data associated with the physical environment or environmental data indicating external conditions. By way of example, environmental data 314 may include environmental temperature readings of the physical environment, external temperature outside the physical environment, design points indicating the external temperature that the physical environment is designed to withstand, and the like. At least a portion of environmental data 314 may be collected from one or more sensors (not shown) configured to measure specific conditions such as environmental temperature within the physical environment, external temperature outside the physical environment, and the like. In some embodiments, at least a portion of environmental data 314 may be provided by a weather source (such as a weather forecast stored in a storage device of the power orchestration system 300) or obtained from an external source (such as a weather forecast website). In some embodiments, environmental data 314 may include tables and / or protocols from which a reduced capacity / capability can be determined / identified. By way of example, a table or mapping may be included in environmental data 314 indicating that a particular difference between the environmental temperature and the temperature outside the physical environment (referred to as the "external" temperature) is associated with a particular reduced capacity of power consumption that the components of the system or individual components can manage. Environmental data 314 may be stored in a location accessible to the VEPO orchestration service 302, and environmental data 314 may be retrieved from that location, or environmental data 314 may be obtained from a service and / or source configured to manage and / or obtain such data (e.g., a metrics service in a cloud computing environment, the power management service 304, etc.). In some embodiments, environmental data 314 is provided directly from the source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0072] Host / instance data 316 may include any suitable data associated with the host and / or instance. Host / instance data 316 may include workload metadata that identifies the workloads being executed at the host and / or via a particular instance. In some embodiments, host / instance data 316 may identify the corresponding customer, host, and / or instance and / or the corresponding priority of the workload, and the like. At least a portion of host / instance data 316 may initially be obtained and / or maintained by a separate service (e.g., the compute service 306, an example of a compute service control plane in a cloud computing environment). Host / instance data 316 may be stored in a location accessible to the VEPO orchestration service 302, and host / instance 316 may be retrieved from that location, or host / instance 316 may be obtained from a service and / or source configured to manage and / or obtain such data (e.g., the compute service 306, etc.). In some embodiments, host / instance data 316 may be provided directly from the source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0073] The operational data 318 can include any suitable data associated with the operating conditions or status of one or more devices or components corresponding to a physical environment. As a non-limiting example, the operational data 318 can include the operating status of one or more temperature control systems (e.g., Figure 1 the (one or more) temperature control systems 112). In some embodiments, the operational data 318 can indicate which temperature control systems are operable and / or the operable capacity of these systems. In some embodiments, the environmental data 314 can include tables and / or protocols from which a reduced capacity / capability can be determined / identified. By way of example, a table or mapping can be included in the operational data 318 indicating that a failure (e.g., complete or partial) of a particular component (e.g., a particular temperature control system) is associated with a particular reduced capacity of power consumption that can be managed by the components of the overall system or a single component of the system. At least some portion of the operational data 318 can be initially obtained and / or maintained by a separate service (e.g., the power management service 304, etc.). The operational data 318 can be stored in a location accessible to the VEPO orchestration service 302 and the operational data 318 can be retrieved from that location, or the operational data 318 can be obtained from a service and / or source (e.g., the compute service 306, etc.) configured to manage and / or obtain such data. In some embodiments, the operational data 318 can be provided directly to the VEPO orchestration service 302 from a source or a component configured to obtain such data according to a predefined schedule, frequency, or periodicity or via a request.

[0074] It should be appreciated that the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 can include current data indicating current values and / or historical data indicating corresponding historical values. In some embodiments, future attributes of the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 can be predicted at least in part based on these historical values. In some embodiments, machine learning can be utilized to predict any suitable portion of these future attributes. Example methods for predicting future values of the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 are discussed in more detail Figure 12 below. In some embodiments, the VEPO orchestration service 302 can be configured to aggregate data of any suitable combination of the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 into a table, mapping, or database from which the data can be filtered or sorted to identify subsets of resources (e.g., hosts, instances, workloads, customers) for a given set of constraints (e.g., low-priority workloads, free-tier customers, etc.).

[0075] The VEPO orchestration service 302 can be configured to identify and / or modify the total power threshold associated with the physical environment based at least in part on any suitable combination of the following: current / total power consumption values (actual or predicted) associated with any suitable combination of the host(s) 324; a received request associated with a government entity (e.g., a local power authority) that orders / requests a specific power reduction or an overall power reduction, possibly for a specific time period (e.g., the next 24 hours); current or predicted environmental conditions (e.g., current or predicted ambient temperature, current or future external temperature, etc.); current or predicted operating conditions corresponding to the temperature control system (e.g., current or predicted total / partial failure), etc.

[0076] The VEPO orchestration service 302 can be configured to manage the actual power consumption values corresponding to the hosts / instances / workloads of the physical environment through associated functions or functions provided by other systems and / or services. Managing the actual power consumption values can include power capping, pausing workloads, instances, hosts, shutting down workloads / instances / hosts, migrating an instance from one host to another, migrating a workload from one instance and / or host to another instance and / or host, etc.

[0077] The VEPO orchestration service 302 can obtain configuration data (e.g., mappings, protocol sets, rules, etc.) corresponding to multiple response levels. Each response level can be associated with a corresponding set of reduction actions (e.g., power capping, migration, pause, shutdown, etc.) that can be performed on the components of the physical environment (e.g., the server(s) 104, the PDU(s) 202, 204, 206, 208, etc.). In some embodiments, each set of reduction actions corresponding to a particular level is associated with a potentially different reduction in the total power consumption of the data center. In some embodiments, the multiple response levels indicate an increase in the severity of performing reduction actions to reduce the total power consumption of the data center. "Severity" is intended to refer to the relative degree of destructiveness of the action. For example, the action of power capping a server may be considered less severe than the action of completely shutting down the server, because in the former, the server can still provide some processing capacity, although the capacity is reduced, while in the latter, the server provides no processing capacity. A set of example response levels is discussed in more detail. Figure 9 A set of example response levels is discussed in more detail.

[0078] The VEPO orchestration service 302 can be configured to determine the impact of applying a reduction action of a given level to hosts / instances / workloads (collectively referred to as "resources") of a physical environment using configuration data corresponding to the specification of (one or more) response levels and their corresponding reduction actions. The impact of applying a given action may depend on the current conditions of the runtime resources. In some embodiments, identifying the impact of a given reduction action may include identifying a set of resources and / or customers for which the action is applicable if implemented. Thus, the estimated impact is intended to refer to the scope or applicability of a given action or level. Identifying the scope and / or applicability of a given action or level may include identifying the specific resources and / or customers that will be affected by implementing the action or level and / or identifying any appropriate attributes associated with the applicable resources and / or customers. As a simple example, the VEPO orchestration service 302 may identify that if the action is taken to shut down all idle hosts, then X idle hosts among the (one or more) hosts 324 will be affected, and these (one or more) hosts are associated with a particular customer or a particular number of customers, and / or the (one or more) hosts and / or customers are associated with other attributes such as category (e.g., free tier), priority (e.g., high priority), etc.

[0079] The VEPO orchestration service 302 can be configured to estimate the impact and / or actual power reduction that one or more levels may experience based at least in part on any suitable combination of configuration data specifying levels and corresponding actions, and power data 310, account data 312, host / instance data, etc. The estimated impact can be identified for one or more response levels, one or more reduction actions corresponding to a given level, etc. (e.g., which (one or more) hosts, (one or more) instances, (one or more) workloads, (one or more) customers may be affected, what attributes are associated with the affected (one or more) hosts, (one or more) instances, (one or more) workloads, and / or customers, how many of the affected (one or more) hosts / (one or more) instances / (one or more) workloads / (one or more) customers are there, etc.). In some embodiments, the estimated impact of a given action can be aggregated with the estimated impact of all actions of a given level to identify the estimated impact of the given level.

[0080] The VEPO orchestration service 302 can be configured to determine the estimated impact and / or actual power reduction that one or more levels may experience based at least in part on current data, historical data, or predictive data (e.g., current values, historical values, or predictive values corresponding to power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318).

[0081] In some embodiments, the VEPO orchestration service 302 may be configured to present, recommend, or automatically select a given response level from a set of possible response levels, based at least in part on any suitable combination of the estimated impact and / or estimated power reduction that may be experienced if a reduction action corresponding to a given level is implemented (e.g., carried out). Thus, in some embodiments, the power orchestration system 300 may recommend and / or select a particular level based on any suitable combination of: 1) determining a level that, if implemented, would likely result in a sufficient but not excessive reduction in power consumption at the resources of the physical environment (e.g., the minimum power consumption reduction amount sufficient to bring the current power consumption value below the current total power threshold), 2) determining a level that includes a set of the least stringent actions, or 3) determining a level or action with the least impact (e.g., a set of affected resources and / or customers with the fewest resources / customers likely to be affected, having a priority or category indicating the lowest degree of collective importance, etc.). Determining the least stringent level or action, or the level or action with the least impact, may be determined independently of the estimated power consumption reduction that may be experienced by implementing that level and / or action, or determining the least stringent level / action and / or the level / action with the least impact may be determined from one or more levels that, if implemented under the given current conditions, are estimated to result in a sufficient reduction in power consumption such that the power consumption value drops to an aggregate value below the current total power threshold).

[0082] The power management service 304 may be configured to provide functionality for identifying power ceiling values for one or more hosts and / or instances, based at least in part on the total power threshold, the budget / assigned power threshold associated with the (one or more) PDUs, and / or the current individual and / or total power consumption values. In some embodiments, the power management service 304 may be configured to identify the resources to which power ceilings are to be applied and / or the specific values of these power ceilings, based at least in part on any suitable combination of current, historical, or predicted values of the power data 310, the account data 312, the environmental data 314, and the host / instance data 316. In some embodiments, the VEPO orchestration service 302 may trigger and utilize the functionality provided by the power management service 304 for identifying / modifying the budget / assigned power corresponding to one or more resources (e.g., hosts, instances, PDUs, etc.), identifying which or how many resources are applicable, and any suitable combination of the estimated power reduction values that may be experienced if a given level of (one or more) actions is implemented. In some embodiments, the power management service 304 may perform some or all of these functions.

[0083] The VEPO orchestration service 302 and / or the power management service 304 may host the user interface 308. The user interface 308 may be configured to provide any suitable application metadata corresponding to the following combinations: current individual and / or power consumption values corresponding to any suitable number or type of resources (e.g., hosts, instances, workloads, etc.) or customers, any suitable attributes associated with these resources or customers (e.g., priority, category, etc.), prediction conditions (e.g., changes in predicted total power thresholds), current conditions (e.g., current total power thresholds), estimated impacts (e.g., the number, identifier, or other attributes associated with affected host / instance / workload customers), and / or estimated power reduction (e.g., the estimated reduction expected to be experienced if an action of that level is applied), and / or the identification of the action(s) corresponding to a given level. The user interface 308 may present any suitable combination of this application metadata. The application metadata may correspond to any suitable combination of available response levels. In some embodiments, the application metadata corresponding to current / predicted power consumption values and / or current / predicted total power thresholds may be displayed. In some embodiments, the application metadata presented at the user interface 308 may correspond to any suitable number of levels of estimated power consumption reduction and / or estimated impacts. In some embodiments, the VEPO orchestration service 302 may provide a particular level of recommendation (e.g., via the provided application metadata corresponding to that level or the reduction actions associated with that level), and the confirmation / or rejection of the recommended level may be input via an optional option at the user interface 308. If confirmed, the VEPO orchestration service 302 may be configured to perform operations to effectuate the implementation of the confirmed level and its corresponding actions. In some embodiments, the user interface 308 may present the application metadata for any suitable number of response levels and / or corresponding reduction actions, and one or more options may be provided at the user interface 308 to enable the user to select a particular level and / or action.

[0084] Receiving user input that indicates selection of a particular level and / or action or that confirms selection of a particular level and / or action can cause the VEPO orchestration service 302 to perform operations to effectuate the implementation of the confirmed level and its corresponding action. Effectuating or implementing the confirmed level and / or its corresponding action can include instructing any suitable components of the power orchestration system (e.g., power management service 304, compute service 306) to effectuate any suitable portion of the level and / or the corresponding (one or more) actions. Instructing one or more components can include providing any suitable data, such as any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, operation data 318, to the instructed (one or more) components. In some embodiments, the instructed components can be configured to effectuate the (one or more) actions or level based on the data provided by the VEPO orchestration service 302, or the instructed components can obtain any suitable portion of the data from a location storing power data 310, account data 312, environmental data 314, host / instance data 316, or operation data 318.

[0085] As a non-limiting example, if the (one or more) actions to estimate the level of reduction or impact include power capping, then the power management service 304 can be configured to estimate a power ceiling, estimate individual or total power consumption reduction, and / or effectuate the estimated power ceiling to achieve the estimated individual / total power consumption reduction. In some embodiments, the power ceiling can be effectuated directly by the power management service 304, or by instructing the compute service 306. Other actions, such as migrating instances or workloads, pausing instances, workloads, or hosts, shutting down hosts, or preventing (at least for a period of time) future assignment or startup of instances and / or workloads, can be identified by the VEPO orchestration service 302 and / or the compute service 306, and effectuated by the VEPO orchestration service 302 through the functionality provided by the compute service 306. In some embodiments, the compute service 306 can be configured to identify hosts / instances / workloads / customers that may be affected and provide such information to the VEPO orchestration service 302 and / or the power management service 304.

[0086] The compute service 306 can be configured to communicate with the baseboard management controller (BMC) / integrated lights-out manager (ILOM) 322 of the rack PDU 320. ILOM is an example of a BMC for illustrative purposes. In some embodiments, the BMC / ILOM 322 can be a dedicated service processor that monitors the physical state of the host machine, which in this case is the rack PDU 320. Similarly, the (one or more) hosts 324 (e.g., the (one or more) hosts of the rack 319, to which the rack PDU 320 is coupled in accordance with Figure 5The power distribution hierarchy under discussion that manages its power can include a BMC / ILOM 328, which can be a dedicated service processor, that monitors the physical state of the host, which in this case is one of the host(s) 324. The BMC / ILOM 322 and the BMC / ILOM 328 can be configured to manage and / or enforce, etc., the budgeted power and / or power capping on the device on which the BMC / ILOM 328 operates or on downstream devices. For example, the BMC / ILOM 322 can receive a power capping value for use by the BMC / ILOM 328 to constrain the power consumption of a given host. The BMC / ILOM 322 can distribute the power capping to the corresponding BMC / ILOM 328 associated with that power capping. In some embodiments, the BMC / ILOM 322 can distribute the (one or more) power cappings stored at the applicable host(s), but the enforcement of these power cappings does not occur until the BMC / ILOM 322 provides further instructions to the BMC / ILOM 328. The BMC / ILOM 328 can enforce the power capping at the host level and / or instance level and / or at the workload level of one or more workloads (not shown) executed at the instance.

[0087] The BMC / ILOM 322 to the BMC / ILOM 328 can be used to instruct a device (e.g., the rack PDU 320 and / or the host(s) 324) to pause operation or shut down the host / instance / workload. In some embodiments, the BMC / ILOM 322 to the BMC / ILOM 328 can be instructed to resume operation of the host / instance / workload. If shut down, then the device can be instructed (e.g., by the computing service 306 and / or by an upstream device such as its PDU) to start from the shut-down state (e.g., via its BMC / ILOM). This can include the computing service 306 sending the (one or more) instructions to the BMC / ILOM 322 to the BMC / ILOM 328 to pause and / or shut down the host / instance / workload.

[0088] In some embodiments, the computing service 306 can perform any appropriate operations to migrate a workload from one instance to another (e.g., migrate to an instance on the same host, migrate to a different instance operating on a different host, etc.), migrate an instance from one host to another (e.g., hosts in the same rack, hosts in different racks, etc.), migrate the instance / workload back to the host that initially hosted the previously migrated instance / workload, etc. In some embodiments, the computing service 306 can be configured to ensure that a host / instance / workload is not assigned to a host and / or instance to which a level and / or action applicable to the implementation or ongoing implementation pertains.

[0089] Any suitable combination of (one or more) devices (e.g., (one or more) hosts 324, rack PDU 320) may include one or more power controllers (e.g., BMC / ILOM 328, BMC / ILOM 322, respectively). A power controller may be any suitable hardware or software component operating at a device (e.g., a server) that is configured to monitor and / or manage power consumption at the device. A power controller may individually monitor the power consumption of the corresponding device on which it operates. Each power controller may be configured to enforce a power capping limit to constrain the power consumption at the corresponding device. Enforcing a power capping limit may include monitoring the power consumption at the device, determining whether to constrain (e.g., limit, cap, etc.) the power consumption at the device (e.g., at least in part based on a comparison between the current power consumption of the device and a stored power capping limit), and limiting / capping the power consumption at the device (e.g., using dynamic voltage and / or frequency scaling of the (one or more) processors and / or memories of the device to throttle the power consumption at the device). Any suitable operations associated with power capping may be implemented by the power controller.

[0090] In some embodiments, a power controller (e.g., BMC / ILOM 322) may communicate with another power controller (e.g., BMC / ILOM 328) corresponding to one of the (one or more) hosts 324 via a direct connection (e.g., via a cable) and / or via a (one or more) network. A power controller (e.g., BMC / ILOM 328) may provide power consumption data indicative of the current power consumption of the device (e.g., cumulative power consumption over a period of time, current power consumption rate, etc.). The power consumption data may be provided to a power controller (e.g., BMC / ILOM 322) at any suitable frequency, periodically, or according to a predefined schedule or event (e.g., upon breaching a predefined consumption threshold, upon a threshold amount change in the consumption rate, upon determining that one or more predefined conditions are met, upon determining the thermal properties of the device, etc.).

[0091] In some embodiments, a power controller (e.g., BMC / ILOM 328) may receive a power capping value (also referred to as a "power cap") from a power controller (e.g., BMC / ILOM 322). In some embodiments, additional data may be provided with the power capping value. By way of example, an indicator may be included in the power capping value that indicates whether the power capping value is to be applied immediately. In some embodiments, it may be default to immediately apply / enforce the received power cap. In other embodiments, it may be default not to immediately apply / enforce the received power cap.

[0092] When applying / enforcing a power limit, a power controller (e.g., BMC / ILOM 328) can monitor the power consumption at the device. This can include using metering devices or software that are configured to identify / calculate the power consumption data of the device (e.g., cumulative power consumption over a period of time, current power consumption rate, change in the current power consumption rate within a time window, etc.). As part of applying / enforcing a power limit (also referred to as "power capping"), the power controller (e.g., BMC / ILOM 328) can determine whether to constrain (e.g., limit, cap, etc.) the power consumption at the device (e.g., at least in part based on comparing the current consumption rate of the device and a stored power limit value). When constraining the power consumption at the device, the power controller (e.g., BMC / ILOM 328) can limit / cap the power consumption at the device (e.g., using dynamic voltage and frequency scaling of the (one or more) processors and / or memory to inhibit the power consumption at the device). In some embodiments, when the current consumption data of the device (e.g., cumulative consumption, consumption rate within a time window, etc.) approaches the enforced power limit value (e.g., breaches a threshold less than the power limit value), the power controller (e.g., BMC / ILOM 328) can execute instructions to limit / cap the power consumption of the device. When constraining / capping the power consumption at the device (also referred to as "throttling"), the power controller (e.g., BMC / ILOM 328) ensures that the power consumption of the device remains below the power consumption indicated by the power limit value. The power controller (e.g., BMC / ILOM 328) can be configured to generally constrain the power at the host, or to constrain the power at the host according to the instances and / or workloads to which the power limit may apply.

[0093] In some embodiments, the power controller of the (one or more) hosts 324 can be configured to allow the (one or more) hosts 324 to run unconstrained (e.g., not capped based on power consumption and power limit) until the power controller (e.g., BMC / ILOM 328) is instructed to apply / enforce a power limit. In some embodiments, the power controller (e.g., BMC / ILOM 328) can receive a power limit value that may or may not later be instructed to be enforced, but when received, the power limit value can be stored in memory and not used for power management at the device. Thus, in some embodiments, the power controller (e.g., BMC / ILOM 328) can avoid starting a power capping operation (e.g., the above comparison and determination) until it is instructed to do so (e.g., via an indicator provided by the power controller 448). The indication to start enforcing the power limit can be received together with the power limit value, or can be received as a separate communication from the power controller 448.

[0094] According to the power distribution hierarchy 500, each power controller of one or more PDUs (e.g., BMC / ILOM 322) can be configured to distribute, manage, and monitor power for any suitable number of (one or more) devices (e.g., (one or more) hosts 324 of the rack 319). In some embodiments, the power controller (e.g., BMC / ILOM 322) can be a computing agent or program installed at a given PDU (e.g., Figure 3 the rack PDU 320). The power controller (e.g., BMC / ILOM 322) can be configured to obtain power consumption data from (one or more) hosts 324 of the rack 319 corresponding to the devices with which it is associated (e.g., the devices for which the given PDU is configured to distribute, manage, or monitor power). The power controller (e.g., BMC / ILOM 322) can receive the consumption data according to a predefined frequency, periodicity, or schedule implemented by the power controller (e.g., BMC / ILOM 328), and / or the power controller (e.g., BMC / ILOM 322) can receive the consumption data in response to a request for the consumption data to the power controller (e.g., BMC / ILOM 328). The power controller (e.g., BMC / ILOM 322) can be configured to request the consumption data from the power controller (e.g., BMC / ILOM 328) according to a predefined frequency, periodicity, or schedule implemented by the power controller (e.g., BMC / ILOM 322).

[0095] The power controller (e.g., BMC / ILOM 322) can transmit the received consumption data (directly or through the computing service 306) to the power management service 402 at any suitable time according to any suitable frequency, periodicity, or schedule, or due to the satisfaction of one or more predefined conditions (e.g., the individual / cumulative / collective consumption rate of (one or more) hosts 324 changes by more than a threshold). In some embodiments, the power controller (e.g., BMC / ILOM 322) can aggregate the power consumption data received from (one or more) hosts 324 (e.g., via the power controller (e.g., BMC / ILOM328) of each device) before transmitting the aggregated consumption data to the power management service 402. In some embodiments, the consumption data can be aggregated by host, instance type / category / priority, workload type / category / priority, or customer.

[0096] A power controller (e.g., BMC / ILOM 322) can receive one or more power ceiling values corresponding to one or more hosts 324 at any appropriate time (e.g., from the execution manager 410 of the power management service 402). These one or more power ceiling values can be calculated by the power management service 402 (e.g., via the constraint identification manager 408). In some embodiments, the power controller (e.g., BMC / ILOM 322) can receive a timing value, where the power ceiling value indicates the duration to be used for a timer. The power controller (e.g., BMC / ILOM 322) can be configured to generate and initiate a timer having an associated duration / time period corresponding to the timing value. When the timer expires, indicating that the time period corresponding to the time period has passed, the power controller (e.g., BMC / ILOM 322) can transmit data to one or more hosts, including an indicator or other appropriate data indicating that one or more hosts continue power capping using the power ceiling values previously stored at each device. In some embodiments, the power ceiling value can be provided with an indicator. In other embodiments, the power ceiling value can be provided by the power controller (e.g., BMC / ILOM 322) to the power controller of the device (e.g., BMC / ILOM 328) immediately after receiving the one or more power ceiling values from the power management service 402. Thus, in some embodiments, the power ceiling value can be transmitted by the power controller (e.g., BMC / ILOM 322) to the power controller (e.g., BMC / ILOM 328), stored in memory, but not enforced by the power controller (e.g., BMC / ILOM 322) until the power controller (e.g., BMC / ILOM 328) receives subsequent data from the power controller (e.g., BMC / ILOM 322) indicating that the power controller (e.g., BMC / ILOM 328) should start power capping.

[0097] The VEPO orchestration service 302, the power management service 304, the computing service 306, the rack PDU 320, one or more hosts 324 (collectively referred to as " Figure 3The "(one or more) devices" can communicate via one or more wired or wireless networks (e.g., (one or more) networks 808). The storage device storing power data 310, account data 312, environmental data 314, host / instance data 316, and operation data 318 can also communicate with the VEPO orchestration service 302, power management service 304, computing service 306, rack PDU 320, (one or more) hosts 324 via one or more wired or wireless connections (e.g., via one or more networks). In some embodiments, (one or more) networks can include any one or a combination of various different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks.

[0098] Figure 3 The (one or more) devices can be any suitable type of computing device, such as, but not limited to, server devices, networking devices, or any suitable device within a data center. In some embodiments, some devices (e.g., rack PDU 320 and (one or more) hosts 324) are arranged in a power distribution hierarchy (such as the power distribution hierarchy 500 discussed in conjunction with Figure 5 In some embodiments, (one or more) hosts 324 can correspond to and be represented by level 1 nodes of the power distribution hierarchy 500. The rack PDU 320 ("rack leader") can correspond to and be represented by a higher-level node of the power distribution hierarchy (e.g., a level 2 node).

[0099] Figure 3 Each of the (one or more) devices can include at least one memory. The (one or more) processors can each be suitably implemented in hardware, computer-executable instructions, firmware, or a combination thereof. Figure 3 The computer-executable instruction or firmware implementation of the (one or more) processors of the (one or more) devices can include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0100] Figure 3 The memory of the (one or more) devices can store program instructions that can be loaded and executed on the (one or more) corresponding processors of a given device, as well as data generated during the execution of these programs. Depending on the configuration and type of the user computing device, the memory can be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). Figure 3The (one or more) devices may also include additional removable and / or non-removable storage devices including, but not limited to, magnetic storage devices, optical disks, and / or tape storage devices. A disk drive and its associated computer-readable medium can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for a computing device. In some embodiments, the memory may separately include multiple different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM.

[0101] Turning to the contents of the memory in more detail, the memory may include an operating system, one or more data repositories, and one or more applications, modules, instances, workloads, or services such as the VEPO orchestration service 302, the power management service 304, the computing service 306, etc.

[0102] Figure 3 The (one or more) devices may include (one or more) communication connections that allow the (one or more) devices to communicate with each other via (one or more) networks (not shown). Figure 3 The (one or more) devices may also include I / O devices such as a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc.

[0103] Figure 4 Illustrated is an example architecture 400 of a power management service 402 ( Figure 3 an example of the power management service 304) according to at least one embodiment. The power management service may include an input / output processing manager 404 that may be configured to receive and / or transmit any appropriate data between the power management service 402 and Figure 3 any other appropriate device. As depicted, the power management service 402 may include a consumption monitoring manager 406, a constraint identification manager 408, and an enforcement manager 410, but more or fewer computing components or subroutines may be similarly utilized.

[0104] The consumption monitoring manager 406 may be configured to determine current individual and / or total power consumption values (e.g., corresponding to Figure 3 the (one or more) hosts 324, each host being Figure 1(example(s) of one or more of the servers 104). The constraint recognition manager 408 may be configured to perform any suitable functions corresponding to identifying one or more power ceiling values for a host, instance, or workload (e.g., (one or more of) the hosts 324). According to at least one embodiment, the enforcement manager 410 may be configured to enforce and / or trigger the enforcement of a power ceiling at any suitable device. Allocating power may refer to the process of assigning a budgeted amount of electrical power (e.g., expected load / power draw) to any suitable component. A power ceiling may be an example of a budgeted amount of electrical power.

[0105] In some embodiments, the functions described in connection with the power management service 402 may be performed by one or more virtual machines implemented in a hosted computing environment (e.g., on (one or more of) the hosts 324, corresponding to Figure 1 any suitable number of the servers 104). The hosted computing environment may include one or more rapidly provisioned and released computing resources, which may include computing, networking, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. Multiple example cloud computing environments are provided and discussed in more detail below. Figures 15 - 18 provide and discuss multiple example cloud computing environments in more detail.

[0106] The power management service 450 (e.g., the consumption monitoring manager 406) may be configured to receive / obtain consumption data from (one or more of) the PDUs (e.g., from a rack PDU 320 of a power controller (such as Figure 3 the BMC / ILOM 322). As described above, the consumption data may be aggregated or accumulated for one or more devices. As a non-limiting example, an instance of the consumption data received by the power management service 402 may include the cumulative / aggregated consumption data for each device associated with a given PDU (e.g., managed by a given PDU). For example, the PDU providing the consumption data may be a rack PDU (e.g., the rack PDU 320), and the consumption data provided by the PDU may include the cumulative / aggregated and / or individual consumption data values for each device in the rack (e.g., the hosts in (one or more of) the hosts 324 associated with a given rack 319). In some embodiments, the cumulative / aggregated consumption data values may additionally include the power consumption of the PDU. The power management service 402 may access the individual and / or aggregated or cumulative power consumption data for individual devices or all devices associated with this instance of the consumption data from the consumption data. The power management service 402 may be configured to be at least partially based on representing Figure 5Perform any suitable operations (e.g., aggregating consumption data, calculating a power ceiling value, calculating a timing value, determining a budget power value, identifying whether power capping is required (e.g., when the aggregated consumption breaches / may breach the budget power, etc.)) on the data of the power distribution hierarchy 500 and / or any suitable configuration data indicating the arrangement of components within the data center (e.g., indicating which devices distribute power to which devices). For example, the configuration data may include data representing the power distribution hierarchy 500.

[0107] Figure 5 Illustrates an example of a power distribution hierarchy 500 corresponding to the arrangement of components of Figure 2 The power distribution hierarchy 500 may represent the arrangement of any suitable number of components of a power system (such as the power distribution infrastructure components discussed in connection with Figure 2 the power distribution infrastructure 200). The power distribution hierarchy 500 may include any suitable number of nodes organized according to any suitable number of levels. Each level may include one or more nodes. The root level (e.g., level 5) may include a single root node of the power distribution hierarchy 500. Each node of the power distribution hierarchy 500 may represent a corresponding component of the power distribution infrastructure 200. A set of one or more nodes at a given level may descend from a particular node corresponding to a higher level of the power distribution hierarchy 500. A set of lower-level components represented by nodes at a lower level (e.g., level 1) may receive power distributed by a higher-level component represented by a node at level 2 (e.g., a component upstream of the lower-level components), which in turn receives power from a higher-level component represented by a node at level 3, which receives power from a higher-level component represented by a node at level 4. In some embodiments, each component of the power system receives power initially distributed by a component corresponding to a node at level 5 (e.g., the highest level of the power distribution hierarchy 500) (e.g., node 502, the root node). Node 502 may receive power from a utility power source (e.g., a local power utility system).

[0108] As depicted, the power distribution hierarchy 500 includes node 502 at level 5. In some embodiments, node 502 may represent Figure 2The uninterruptible power supply of one or more UPSs 202. The components corresponding to node 502 can distribute / supply power to the components corresponding to node 504 at level 4. Components (e.g., lower-level components) that receive power from higher-level components (components represented by nodes in the power distribution hierarchy 500 that are at a higher level than the level of the nodes representing the lower-level components) can be considered subordinate to the higher-level components. In some embodiments, node 304 can represent a component subordinate to the UPS represented by node 502, such as Figure 2 One of the one or more intermediate PDUs 204 (e.g., distribution panel). The components represented by node 504 can be configured to distribute / supply power to the components represented by nodes 506 and 508 respectively.

[0109] Nodes 506 and 508 at level 3 can each represent Figure 2 The corresponding components (e.g., the corresponding one or more row PDUs). The components corresponding to node 506 (e.g., row PDU 206, busbar) can distribute / supply power to the components corresponding to node 510 (e.g., Figure 2 The rack PDU 208A) and the components corresponding to node 512 (e.g., Figure 2 The rack PDU 208N). The components corresponding to node 510 (e.g., rack PDU 208A) can distribute / supply power to the components corresponding to nodes 514 and 516 at level 1 (representing Figure 2 The servers 212A and 212B respectively), and these components can be monitored / managed by the components of servers 212A and 212B (such as power controllers 216A and 216B). The components corresponding to nodes 514 and 516 can be arranged in the same rack.

[0110] The components corresponding to node 512 (e.g., rack PDU 208A) can distribute / supply power to the components corresponding to node 518 at level 1 (e.g., Figure 2 The server 214C including the power controller 216C). Nodes 514, 516, and 518 at level 1 can be arranged in the same row (e.g., Figure 2 Row 210) and / or associated with the same row.

[0111] Returning to node 508 at level 3, node 508 (e.g., a different row PDU, such as a remote power board) can distribute / supply power to the components corresponding to node 510 (e.g., Figure 2 The rack PDU 208A) and the components corresponding to node 512 (e.g., Figure 2The rack PDU 208 distributes / supplies power. Components corresponding to node 520 (e.g., the rack PDU) can distribute / supply power to components corresponding to nodes 522 and 526 at level 1 (e.g., corresponding servers including respective power controllers). Components corresponding to nodes 524 and 526 can be arranged in the same rack. Components corresponding to node 522 (e.g., another rack PDU) can distribute / supply power to components corresponding to nodes 528 and 530 at level 1 (e.g., corresponding servers including respective power controllers). Components corresponding to nodes 528 and 530 can be arranged in the same rack.

[0112] The specific number of components (e.g., corresponding to level 1 nodes) that receive power distributed from a higher-level component (e.g., corresponding to a node at level 2) may be different from Figure 5 the number shown. The specific number of levels within the power distribution hierarchy may vary depending on the specific arrangement of components used in a given data center. It is contemplated that each non-root level (e.g., in the Figure 5 example, levels 1 - 4) can include a different number of nodes, which represent a different number of components than the number of nodes depicted in each non-root level of Figure 5 . The nodes can be arranged in a configuration different from but similar to the configuration depicted in Figure 5 .

[0113] Return Figure 4 , the power management service 402 (e.g., the constraint identification manager 408) can calculate aggregated / cumulative consumption data to identify aggregated / cumulative consumption data for one or more devices at a higher level of the power distribution hierarchy 500. For example, the power management service 402 (e.g., the constraint identification manager 408) can utilize the consumption data provided by one or more rack PDUs and calculate the aggregated / cumulative consumption data for row devices. For example, the consumption data corresponds to the devices represented by the level 1 nodes of Figure 5 , and these devices share a common set of nodes up to level 3 of the hierarchy. In this case, the common level 3 nodes represent row PDUs, such as busbars. In some embodiments, the consumption data indicating the power consumption of one or more rack PDUs (e.g., an example of a component corresponding to level 2 of the power distribution hierarchy 500) can be included in the consumption data provided to the power management service 402. The consumption data provided by the rack PDUs can be provided according to any suitable frequency, periodicity, or schedule, or in response to a request transmitted by the power management service 402. The power management service 402 can obtain data related to Figure 3Consumption data associated with any suitable number of (one or more) devices (such as (one or more) hosts 324 and / or racks of (one or more) devices corresponding to one or more racks (e.g., rack 319)) can be aggregated or calculated for any suitable number of components represented by any suitable nodes and / or levels of the power distribution hierarchy 500.

[0114] The power management service 402 (e.g., the constraint identification manager 408) can be configured to calculate one or more power caps (e.g., power cap values for one or more servers to which power is distributed by the higher-level device (e.g., the row-level device in this example)) at least in part based on the power values (e.g., budgeted power amounts) assigned to the higher-level components. The higher-level components can correspond to any suitable level (e.g., levels 2 - 5) of the power distribution hierarchy 500 other than the lowest level (e.g., level 1). For example, the power management service 402 can calculate the power cap value for a server to which power is distributed by a given row device (e.g., a busbar). These calculations can be based on consumption data provided by the rack PDU to which power is distributed by the row device. In some embodiments, the power management service 402 can store the consumption data for subsequent use. The power management service 402 can utilize historical consumption data when calculating these power cap values. In some embodiments, the power management service 402 can obtain, utilize, and / or train one or more machine learning models to identify specific power cap values for one or more hosts 324 (e.g., components corresponding to level 1 of the power distribution hierarchy) from historical consumption data. These techniques will be discussed in more detail Figure 12 below.

[0115] The power management service 402 (e.g., the enforcement manager 410) can calculate the timing value of a timer (e.g., a timer that can be initiated and managed by one or more PDUs (such as rack PDU 320)). The timing value can be calculated based on at least any suitable combination of the following: the change in the power consumption rate of the higher-level component, the direction of change (increase / decrease) in the power consumption of the higher-level component, the power spike tolerance of the higher-level component and / or the entire system 300, or the enforcement time associated with the lower-level device (e.g., the time between when each of one or more devices (e.g., one or more hosts 324) is instructed to enforce the power cap value and the time when each of one or more devices (e.g., one or more hosts 324) will actively enforce the cap (e.g., the time to first constrain / throttle the power consumption at that device, or at least determine whether to constrain / throttle)).

[0116] The power management service 402 (e.g., the enforcement manager 410) can transmit the calculated timing values to the PDU(s) 402 at any appropriate time. In some embodiments, the power management service 450 can initially determine the power ceiling for a given higher-level component (e.g., a row component) without considering the consumption that occurs relative to other components at the same level (e.g., other row components). In some embodiments, when the timer is initialized or elapses, or at any appropriate time, the power management service 402 can process the consumption data corresponding to other components at the same level to determine whether it is more desirable to perform power capping on one or more of the downstream components of these same-level components. In some embodiments, the power management service 402 can utilize the priorities associated with the device(s) and / or the workloads running on the device(s) to determine the power ceiling value (e.g., at least partially based on Figure 3 the priority identified by the host / instance data 316). The power management service 402 can be configured to preferentially perform power capping on the low-priority device(s) / workload(s), while leaving the device(s) / workload(s) with higher priority values unconstrained. In some embodiments, the power management service 402 can be configured to preferentially perform power capping on a group of the highest-consuming devices (e.g., across one row, multiple rows, etc.), while leaving the lower-consuming devices to operate unconstrained. In some embodiments, if the priority associated with a device and / or workload is high (or higher than the priorities associated with other devices and / or workloads), then even if the device is included in the group of the highest-consuming devices, the specific consumption of the device can be left unconstrained. Thus, the power management service 402 can be configured to prioritize the priority of the device / workload over the power consumption at that specific device.

[0117] In some embodiments, the power ceiling values can be initially determined by the power management service 402 (e.g., by the constraint identification manager) for a given row. These power ceiling values can be provided to the power controller of the rack PDU (e.g., BMC / ILOM 320, or another agent or component of the rack PDU 320), which in turn can distribute the power ceiling values to the power controllers (e.g., BMC / ILOM 328) at the host(s) 324 for storage. The power management service 402 (e.g., the constraint identification manager 408) can process the consumption data of other row devices on the same row to determine the power ceiling values for the devices corresponding to different rows. This can be advantageous because the devices managed by another row may not consume their budgeted power, leaving some unutilized power at the device level of that row. In some embodiments, the power management service 402 (e.g., the constraint identification manager 408) can be configured to determine whether it is more advantageous to cap the power of the devices on one row while allowing at least some of the devices on another row to operate without constraints. The benefits of determining a set of power ceiling values can be at least partially based on minimizing the estimated impact (e.g., the number of devices to be power-capped), minimizing the priority values associated with the devices to be power-capped, maximizing the number of devices associated with a particular high-priority value that will not be power-capped, etc. The power management service 402 can utilize a predefined protocol (e.g., a set of rules) to determine whether enforcing the power ceiling values it has sent to a PDU is more or less advantageous / beneficial than the different power ceiling values it has identified based on processing the consumption data of multiple PDUs associated with one or more other rows.

[0118] The power management service 402 (e.g., the enforcement manager 410) can be configured to formulate the enforcement of a set of power ceiling values determined to be the most advantageous (based on a predefined set of rules) to ensure that one or more circuit breakers within the data center do not trip and / or the total power threshold is enforced. If the current / total power consumption level exceeds the current total power threshold, then the power management service 402 can be configured to perform or trigger an operation to bring the current / total power consumption level to a value below the current total power threshold.

[0119] In some embodiments, the monitoring, identification, and / or constraint based on the power ceiling values can be determined directly by the power management service 402 based on power data 310, account data 312, environmental data 314, host / instance data 316, operation data 318, etc.

[0120] In some embodiments, the power management service 402 may be configured to identify and / or enforce a power ceiling value based at least in part on instructions and / or constraints provided by the VEPO orchestration service 302 and / or other applicable data. As a non-limiting example, the VEPO orchestration service 302 may filter aggregated data of any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operational data 318 based on a scaling action associated with a level (e.g., "reduce the power ceiling of all low-priority hosts by 10%") to determine the estimated impact of implementing the action, including the number and / or subset of resources (e.g., hosts, instances, workloads, customers) to which the action would be applied if the action were implemented. For example, the VEPO orchestration service 302 may identify the number and / or identifiers corresponding to all low-priority hosts from aggregated data of any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operational data 318. In some embodiments, the identifier and / or instruction (10% power ceiling reduction) may be provided to the power management service 402, which receives it through the input / output processing module 404 and is implemented by the enforcement manager 410. As another non-limiting example, any suitable portion of the metadata associated with a scaling action (e.g., "10% power ceiling reduction for all low-priority hosts") may be provided to the power management service 402, and the constraint identification manager 408 may be configured to identify specific hosts / instances / workloads from the metadata associated with the scaling action. In other words, any suitable combination of the VEPO orchestration service 302 and / or the power management service 402 (e.g., power management service 304) may be configured to identify the estimated impact of a given action. In some embodiments, the power management service 402 (e.g., the constraint identification manager 408) may be configured to identify an estimated power reduction value corresponding to the individual or total power consumption of one or more devices (e.g., (one or more) hosts 324), assuming hosts / instances / workloads / customers that are expected to be affected if the action (or level) is implemented.

[0121] If it is determined that a previously provided set of power caps to the power controller (e.g., BMC / ILOM 322 and / or BMC / ILOM 322) is most beneficial, then the power management service 402 can transmit data to the power controller to cause the power controller to transmit an indicator to the (one or more) devices to start power capping based on the previously distributed power caps. Alternatively, if the power management service 402 determines that a new set of power caps is more (or most) beneficial from a power management perspective, then it can transmit data to the power controller (e.g., BMC / ILOM 322) to cancel the timer. In some embodiments, canceling the timer may cause the previously distributed power caps to be deleted by an instruction from the power controller (e.g., transmitted in response to canceling the timer), or these power caps may default to timing out (e.g., according to a predefined time period). The power management service 402 can transmit the new power caps to the appropriate PDU (which may include or exclude the same PDU to which the previous set of power caps was transmitted), and indicate that the corresponding devices should immediately enforce these power caps. These PDUs can transmit these power caps along with an instruction to the receiving devices to immediately start the power capping operation (e.g., including monitoring the consumption related to the power cap value, determining whether the cap is based on the power cap value and the current consumption of the device, and capping or unconstraining the operation of the device based on the determination).

[0122] In some embodiments, if the timer expires at the original PDU (e.g., the time period corresponding to the timing value has passed) and no cancellation has been received from the power management system 402, then the power controller (e.g., BMC / ILOM 322) can automatically (e.g., via BMC / ILOM 328) instruct the (one or more) devices to start the power capping operation using the stored power cap values. This technique ensures fail-safe when the power management service 402 fails to instruct the PDU or cancel the timer for any reason.

[0123] The above technique enables more lower-level devices to operate unconstrained by minimizing the number and / or frequency of capping the power consumption at these devices. Additionally, the peaks of downstream devices in a given device hierarchy can be allowed to exceed the budgeted power of the device while still ensuring that the maximum power capacity of the device (a value higher than the budgeted value) is not exceeded. This allows the consumption of lower-level devices to cause the consumption levels at higher-level devices to operate within a power buffer that is not used in a conventional system. The described system and technique allow for more efficient utilization of the power distribution components of system 400 and reduce waste while ensuring avoidance of power failures.

[0124] In some embodiments, the power management service 402 may select a set of power capping values determined to be most advantageous (e.g., based at least in part on any suitable combination such as maximum expected reduction and / or minimum estimated impact, etc.), or may receive from the VEPO orchestration service 302 and / or implement a specific set of power capping values provided by the power management service 402 (previously identified by the power management service 402 and provided to the VEPO orchestration service 302). In some embodiments, the identified specific power capping values may be identified at least in part based on maximizing power consumption reduction, identifying a reduction associated with a sufficient power cap, which reduction associated with the sufficient power cap is identified alone or in conjunction with other operations corresponding to reduction actions or levels associated with reducing the total power consumption value of the (one or more) devices below the current total power threshold.

[0125] Figure 6 is a flowchart illustrating an example method 600 for managing excessive power consumption according to at least one embodiment. Method 600 may include more or fewer operations than those depicted or described with respect to Figure 6 These operations may be performed in any suitable order. Any suitable portion of the operations described in connection with the power management service 606 may be additionally or alternatively performed by Figure 3 the VEPO orchestration service 302 and / or the compute service 306.

[0126] Method 600 may begin at 610, where the PDU 604 may receive and / or obtain consumption data from the (one or more) devices 602 (e.g., via a request). The (one or more) devices 602 may be Figure 3 examples of the (one or more) hosts 324 (e.g., multiple servers) and may be arranged within a common server rack (e.g., rack 319). The PDU 604 is Figure 2 an example of the rack PDU 208A and / or Figure 3 the rack PDU 320. Each of the (one or more) devices 602 may be a device to which the PDU 604 distributes power. The (one or more) PDUs 608 may be examples of other rack PDUs associated with the same row as the PDU 604 and thus receive power from the same row PDU as the PDU 604 (e.g., Figure 2 the row PDU 206, corresponding to Figure 5The node 506) receives power. The consumption received from the PDU(s) 608 can correspond to the power consumption of the device(s) 612. In some embodiments, at least some portions of the PDU(s) 608 and the device(s) 612 correspond to a different row than the row corresponding to the device(s) 602. In some embodiments, any suitable number of consumption data instances received by the PDU 604 (referred to as "device consumption data") can be associated with a single device in the device(s) 602. The same may be true for the device consumption data received by the PDU(s) 608. In some embodiments, the PDU 604 and the PDU(s) 608 can aggregate and / or perform calculations based on the received device consumption data instances to generate rack consumption data corresponding to each respective PDU. For example, the PDU 605 can generate rack consumption data from the device consumption data provided by the device(s) 602. The rack consumption data can include the device consumption data instances received by the PDU 604 at 610. Similarly, the rack consumption data provided by the PDU(s) 608 can include the device consumption instances received from the device(s) 612. The rack consumption data generated by the PDU 604 and / or the PDU(s) 608 can be generated at least in part based on the power consumption of the PDU 604 and / or the PDU(s) 608, respectively.

[0127] At 614, the power management service 606 ( Figure 4 the power management service 402 of Figure 3 an example of the power management service 304 of Figure 2 an example of the PDU(s) 208 of Figure 3 an example of the BMC / ILOM 328 of

[0128] At 616, the power management service 606 can be at least in part based on another PDU (e.g., Figure 5Line-level devices not depicted therein, such as bus bars) to determine whether to calculate the power ceiling value of (one or more) devices 602 based on the maximum and / or budgeted power amounts associated therewith and the consumption data received from PDU 604 and / or (one or more) PDUs 608. In some embodiments, the power management service 606 may access from the stored data the maximum and / or budgeted power amounts associated with, for example, line-level PDUs (e.g., Figure 2 the line PDU 206 not depicted herein), or the power management service 606 may receive the maximum and / or budgeted amounts from a PDU (line-level PDU) associated with the maximum and / or budgeted power amounts at any appropriate time. The power management service 606 may aggregate the rack consumption data from PDU 604 and / or (one or more) PDUs 608 to determine the cumulative power consumption value for the line-level devices (e.g., the total power consumption of the devices downstream from the line-level device within the previous time window, the change in the consumption rate from the perspective of the line-level device, the direction of the change in the consumption rate from the perspective of the line-level device, the time period required to initiate power capping at one or more devices 602 and / or 612, etc.). In some embodiments, the power management service 606 may generate additional cumulative power consumption data from at least some portion of the cumulative power consumption data values and / or using historical device consumption data (e.g., cumulative or individual consumption data). For example, the power management service 606 may calculate any appropriate combination of the following at least in part based on historical consumption data (e.g., historical device consumption data corresponding to (one or more) devices 602 and / or 512 and / or historical rack consumption data): the change in the consumption rate from the perspective of the line-level device, the direction of the change in the consumption rate from the perspective of the line-level device, etc.

[0129] Alternatively, in some embodiments, the power management service 606 may receive at 616 any appropriate data and / or instructions from the VEPO orchestration service 402 for identifying any appropriate portion of the reduction actions (e.g., power capping only for low-priority hosts) and / or any appropriate devices for any appropriate constraints and / or applicable constraints. The power management service 606 may identify a subset of (one or more) hosts for which to calculate the power ceiling (e.g., in this example, only from the low-priority host candidates). For example, the operation of power capping only for low-priority hosts may cause the power management service 606 to identify multiple power ceilings for the subset of these low-priority hosts from all low-priority hosts (e.g., a subset of (one or more) hosts 324).

[0130] If the cumulative power consumption (e.g., the power consumption corresponding to the device(s) 602 and 612) exceeds the budgeted power amount associated with the row-level PDU, then the power management service 606 may determine that a power ceiling value should be calculated. If the cumulative power consumption does not exceed the budgeted amount associated with the row-level PDU, then the power management service 606 may determine that a power ceiling should not be calculated, and method 600 may end. Alternatively, since the cumulative power consumption of the downstream devices (e.g., the device(s) 602 and 612) exceeds the budgeted amount associated with the row-level PDU, the power management service may determine that a power ceiling should be calculated and may proceed to 618.

[0131] In some embodiments, the power management service 606 may additionally or alternatively determine that the power ceiling value should be calculated at 616 based at least in part on providing historical consumption data (e.g., device consumption data and / or rack consumption data corresponding to the device(s) 602 and / or 612) as input data to one or more machine learning models. Any suitable supervised or unsupervised machine learning algorithm may be used to train the machine learning model(s) to identify, from the historical consumption data provided as input, the likelihood that the consumption corresponding to the device associated with the historical consumption data will exceed the budgeted amount (e.g., the budgeted power amount allocated to the row-level device). The machine learning model(s) may be trained using a corresponding training data set that includes instances of historical consumption data. These machine learning model(s) training is discussed in more detail. Using the above techniques, the power management service 506 may identify that the power ceiling value should be calculated based on the cumulative device / rack consumption level exceeding the budgeted power associated with the row-level device and / or based on determining from the output of the machine learning model(s) that the cumulative device / rack consumption level may exceed the budgeted power associated with the row-level device. Determining that the device / rack consumption level may exceed the budgeted power of the row-level device may be determined based on receiving an output from the machine learning model(s) indicating the likelihood (e.g., likely / unlikely), or by comparing an output value (e.g., percentage, confidence value, etc.) indicating the likelihood of exceeding the budgeted power of the row-level device with a predefined threshold. An output value indicating a likelihood exceeding the predefined threshold may cause the power management service 606 to determine that power capping is reasonable and that the power ceiling value should be calculated. Figure 12 The training of these machine learning model(s) is discussed in more detail. Using the above techniques, the power management service 506 may identify that the power ceiling value should be calculated based on the cumulative device / rack consumption level exceeding the budgeted power associated with the row-level device and / or based on determining from the output of the machine learning model(s) that the cumulative device / rack consumption level may exceed the budgeted power associated with the row-level device. Determining that the device / rack consumption level may exceed the budgeted power of the row-level device may be determined based on receiving an output from the machine learning model(s) indicating the likelihood (e.g., likely / unlikely), or by comparing an output value (e.g., percentage, confidence value, etc.) indicating the likelihood of exceeding the budgeted power of the row-level device with a predefined threshold. An output value indicating a likelihood exceeding the predefined threshold may cause the power management service 606 to determine that power capping is reasonable and that the power ceiling value should be calculated.

[0132] At 618, power management service 606 may calculate power ceiling values for any suitable number of devices 602 and / or devices 612 (one or more). For example, power management service 606 may utilize device consumption data and / or rack consumption data corresponding to each of devices 602 and 612 (one or more). In some embodiments, power management service 606 may determine the difference between the cumulative power consumption of a row of devices (e.g., devices 602 (one or more) and any devices 612 (one or more) in the same row) (calculated by aggregating device and / or rack consumption data corresponding to devices 602 (one or more) and devices 612 (one or more)) and the budget power associated with the row-level devices. This difference may be used to identify the power consumption amount to be constrained at the devices associated with the row. Power management service 606 may determine power ceiling values for any suitable combination of devices 602 and / or devices 612 in the same row based on identifying a power ceiling value that, if enforced by the devices, would result in power consumption being reduced to a value less than the budget power associated with the row-level devices.

[0133] In some embodiments, power management service 606 may determine specific power ceiling values for any suitable combination of devices 602 and 612 corresponding to the same row based at least in part on consumption data and / or priority values associated with the devices and / or the workload associated with the power consumption of the devices. In some embodiments, these priority values may be provided as part of the device consumption data provided by the devices associated with the consumption data, or the priority values may be determined at least in part based on device type, workload type, etc. obtained from the consumption data or any other suitable source (e.g., separate data accessible to power management service 606). Using priorities associated with the devices or workloads respectively, power management service 606 may calculate power ceiling values in a manner that favors power-capped devices associated with lower-priority workloads over power-capped devices associated with higher-priority workloads. In some embodiments, power management service 606 may calculate power ceiling values based at least in part on favoring power-capped devices that consume at a higher rate over other power-capped devices that consume at a lower rate (e.g., a group of highest-consuming devices). In some embodiments, power management service 606 may calculate power ceiling values based at least in part on a combination of multiple factors, including the consumption value of each device and the priority associated with the device or the workload being executed by the device. Power management service 606 may identify power ceiling values for the devices that avoid fully capping high-consuming devices and / or high-priority devices, or that cap these devices to a lesser extent compared to devices that consume less power and / or are associated with lower priorities.

[0134] At 620, the power management service 620 may calculate a timing value corresponding to the duration of a timer initialized and managed by a PDU (e.g., PDU 604). The timing value may be calculated based at least in part on any suitable combination of the rate of change of power consumption from the perspective of the row-level device, the direction of change of power consumption from the perspective of the row-level device, or the time at which power capping is performed at each device related to the power ceiling calculated at 618. For example, the power management service 620 may determine the timing value based on determining a relatively large increase in power consumption from the perspective of the row-level device, where the timing value is greater than the timing value for a relatively small increase in power consumption from the perspective of the row-level device. Thus, the more the power consumption rate increases, the smaller the timing value (corresponding to a shorter timer), while the less the power consumption rate increases, may result in a larger timing value (corresponding to a longer timer). Similarly, when the time at which power capping is performed at each device related to the power ceiling is relatively short with respect to the calculation of a first timing value, the calculated first timing value may be less than the calculated second timing value. Thus, the faster the device expecting power capping can enforce the ceiling, the shorter the timing value may be. In some embodiments, no timer is calculated for a power ceiling determined for data provided or indicated via the VEPO orchestration service 302.

[0135] At 622, the calculated power ceiling value calculated at 618 and / or the timing value calculated at 620 may be transmitted to the PDU 604. Although not shown, the corresponding power ceiling value and / or timing value of any one or more devices 612 in the same row may be transmitted at 620. Although not shown, in some embodiments, the transmission at 622 may be based at least in part on an instruction received from the VEPO orchestration service 302 to enforce / implement the power ceiling value calculated at 618.

[0136] At 624, when the timing value is provided at 622, the PDU 604 may use the timing value to initialize a timer having a duration corresponding to the timing value. As described above, in the case where the power management service 606 is instructed by the VEPO orchestration service 302, the timing value may not be provided at 622 and may not be stored at 624.

[0137] At 626, the PDU 604 may transmit the power ceiling value (if received at 622) to one or more devices 602. At 628, one or more devices 602 may store the received power ceiling value. In some embodiments, the power ceiling value provided at 626 may include an indicator indicating that enforcement does not start, or the indicator may not be provided, and one or more devices 602 may default to avoid starting power capping.

[0138] At 630, the power management service 606 can perform a higher level of analysis on the device / rack consumption data received from any one or more devices 612 that correspond to a row different from the row corresponding to the one or more devices 602. As part of this process, the power management service 606 can identify unused power associated with other row-level devices. If there is unused power, then the power management service 606 can calculate a new set of power ceiling values at least in part based on the consumption data corresponding to at least one other row of devices. In some embodiments, any suitable power ceiling values identified by the power management service 606 can be provided to the VEPO orchestration service 302 at any suitable time. As described above, this new set of power ceiling values can be advantageous because the devices managed by another row-level device may not consume their budgeted power, leaving unused power at that row-level device. In some embodiments, the power management service 606 can be configured to determine whether it may be more advantageous to cap the devices of another row while allowing at least some of the one or more devices 602 (and possibly some of the one or more devices 612 corresponding to the same row as the one or more devices 602) to operate without constraint. The benefits of each power capping method can be calculated at least in part based on minimizing the number of devices to be power capped, minimizing the priority values associated with the devices to be power capped, maximizing the number of devices associated with a particular high-priority value that will not be power capped, and so on. The power management service 606 can utilize a predefined scheme or set of rules to determine whether enforcing the power ceiling value it has determined for a row (e.g., corresponding to the power ceiling value transmitted at 622) is more or less advantageous than a different power capping value identified based on processing consumption data from multiple rows. In some embodiments, the benefits of each set of power ceiling values identified by the power management service 606 can be accessed, and the most advantageous set of power ceilings can be selected by the power management service 606 (e.g., according to a set of rules, according to user input selecting a particular response level, etc.).

[0139] The power management service 606 can be configured to enforce a set of power ceiling values determined to be more (or most) favorable (or the set of power ceiling values indicated by the VEPO orchestration service 302) to ensure that one or more circuit breakers within the data center do not trip and / or the total power threshold is enforced. If it is determined that a previously provided set of power ceilings to the PDU 604 is less favorable than the set of power ceiling values calculated at 630, then the power management service 606 can immediately transmit data to the PDU 604 to cause the PDU 604 to transmit an indicator to the device(s) 602 to immediately begin power capping based on the previously distributed power ceilings. Alternatively, the power management service 606 can take no further action, which can allow the timer at the PDU 604 to elapse. Any appropriate action performed by the power management service 606 based on the determination or identification made by the power management service 606 can alternatively be triggered by an instruction from the VEPO orchestration service 302.

[0140] If the power management service 606 determines that a new set of power ceilings is more (or most) favorable from a power management perspective, then it can transmit data to the power controller (e.g., Figure 3 the BMC / ILOM 322) to cancel the timer. The previously distributed power ceilings can be deleted by an instruction from the power controller (e.g., the BMC / ILOM 322) (e.g., transmitted in response to canceling the timer), or those power ceilings can default timeout based on a predefined time period. The power management service 606 can transmit the new power ceilings to a suitable PDU (which can include or exclude the same PDU to which the previous set of power ceilings was transmitted), indicating that the power ceilings will be immediately enforced by the corresponding device. These PDUs can transmit these power ceilings along with an instruction to the receiving device to immediately begin the power capping operation (e.g., including monitoring the consumption with respect to the power ceiling value, determining whether to cap based on the power ceiling value and the current consumption of the device, and operating or leaving the operation of the device unconstrained based on the determination to cap the device).

[0141] If, from a power management perspective, the power ceiling value calculated at 630 is determined to be more favorable than the power ceiling value calculated at 618 and is ultimately stored at (one or more) devices 602 at 628, then method 600 can proceed to 632, where the power ceiling value calculated at 630 can be transmitted to (one or more) PDUs 608 (e.g., any (one or more) PDUs 608 that manage devices associated with the power ceiling value). In some embodiments, the power management service 606 can provide an indication that the power ceiling value transmitted at 632 will be immediately enforced. The operation described at 632 can be triggered by the power management service 606 based on a determination / identification made by the power management service 606 or in response to receiving an instruction from the VEPO orchestration service 302.

[0142] In response to receiving the power ceiling value and the indication at 632, the (one or more) PDUs 608 that distribute power to the devices associated with these power ceiling values can transmit the power ceiling value and the indication to the devices associated with these power ceiling values. At 636, the receiving devices of the (one or more) devices 612 can perform a power capping operation without delay based on receiving the indication. This includes 1) determining whether to limit / qualify the operation based on the current consumption of the device compared to the power ceiling value provided to the given device, and 2) limiting / qualifying or avoiding limiting / qualifying the power consumption of the device.

[0143] In some embodiments, at least in part based on identifying that the power ceiling value calculated at 630 is more (or most) favorable than the power ceiling value calculated at 618, the power management service 606 can transmit data to cancel the timer and / or the power ceiling value transmitted at 622. In some embodiments, the PDU 604 can be configured to cancel the timer at 640. In some embodiments, the PDU 604 can transmit at 642 such that the (one or more) devices 602 delete the data of the power ceiling value from the memory at 644.

[0144] At 646, in the case where a more favorable set of power ceiling values (as described above) is not found or the power management service 606 does not transmit a cancellation (as described in connection with 638 above), the PDU 604 may identify that the timer initiated at 624 has expired (e.g., the duration corresponding to the timing value provided at 622 has passed). Accordingly, the PDU 604 may transmit an indication to the device(s) 602 at 648, thereby instructing the device(s) 602 to start power capping based on the power ceiling value stored at the device(s) 602 at 628. Receiving the indication at 648 may cause the device(s) 602 to start power capping at 650 based on the power ceiling value stored at 628. In some embodiments, the VEPO orchestration service 302 may be configured to control the operations of 646 - 650, and the power management service 606 cannot perform the operations of 646 - 650 unless the VEPO orchestration service 302 instructs it to do so.

[0145] In some embodiments, if the PDU 604 identifies that the power consumption of the device(s) 602 has decreased, then the PDU 604 may be configured to cancel the timer at any appropriate time after the timer is initiated at 624. In some embodiments, this may cause the PDU 604 to transmit data to the device(s) 602 to cause the device(s) 602 to discard the previously stored power ceiling value from local memory.

[0146] In some embodiments, the power ceiling value enforced at any appropriate device may time out or may be replaced by a power ceiling value calculated by the power management service 606 at any appropriate time. In some embodiments, the power management service 606 may transmit a cancellation or replacement of the power ceiling value to any appropriate device via its corresponding rack PDU. If a cancellation of the power ceiling value is received, then the device may delete the previously stored power ceiling value, thereby allowing the device to resume operation without constraint. If a replacement power ceiling value is received, then the device may store the new power ceiling value and enforce the new power ceiling value immediately or when instructed by its rack PDU. In some embodiments, being instructed by its PDU to enforce a new power ceiling value may cause the device's device (e.g., the power controller 446) to replace the previously used power ceiling value for power capping with the new power ceiling value that the device is instructed to enforce.

[0147] Any suitable operations of method 600 can be continuously performed at any suitable time to manage overconsumption (e.g., a situation where the consumption of a server corresponding to a row-level device exceeds the budgeted power of the row-level device), so as to enable more efficient use of previously unused power while avoiding power outages due to circuit breaker tripping. It should be recognized that the power management service 606 can perform similar operations for any suitable level of the power distribution hierarchy 500. Although an example of monitoring power consumption corresponding to a row-level device (e.g., a device represented by a level 3 node of the power distribution hierarchy 500) has been provided, similar operations can be performed for higher-level devices at any suitable level (e.g., components corresponding to any level in levels 2-5 of the power distribution hierarchy 500). As described above, any suitable functions / operations of the power management service 606 can be triggered at least in part based on the determination / identification performed by the power management service 606, or any suitable functions / operations of the power management service 606 can be triggered via instructions from the VEPO orchestration service 302.

[0148] Figure 7 Illustrates an example architecture of a VEPO orchestration service 702 ( Figure 3 an example of the VEPO orchestration service 302) according to at least one embodiment, which is configured to orchestrate the management of the power consumption of a host according to a total power threshold. The VEPO orchestration service 702 can include an input / output processing manager 704, which can be configured to receive and / or transmit any suitable data between the VEPO orchestration service 702 and Figure 3 any other suitable device. As shown, the power management service 402 can include a demand manager 710, an impact identification manager 706, and an enforcement manager 708, but more or fewer computing components or subroutines can be similarly used.

[0149] The demand manager 710 can be configured to monitor and / or modify the total power threshold associated with a physical environment (e.g., Figure 1 a data center 102, a room 110A, a row 108A, etc.). Monitoring can include monitoring Figure 3Any suitable combination of power data 310, environmental data 314, host / instance data 316, and / or operational data 318. Modifying the total power threshold can be performed, and the amount by which the total power threshold is modified can be determined at least in part based on a plurality of trigger events. These trigger events can include, but are not limited to: 1) receiving a request from a government entity (e.g., a local power authority) to curtail power either indefinitely or for a period of time; 2) determining current and / or predicted environmental conditions (e.g., current environmental or external temperature, predicted environmental or external temperature); 3) determining the current or predicted total power consumption corresponding to the devices in the physical environment; 4) determining the current and / or predicted operating conditions of one or more temperature control units in the physical environment. The amount by which the total power threshold can or will be modified can be determined at least in part based on a plurality of factors, including but not limited to: the difference between the environmental temperature and the external temperature, the power consumption attributed to or associated with a particular component (e.g., a particular temperature control unit), the amount or percentage specified in the request received from the government entity, or the difference between the current or predicted total power consumption value and the current or predicted total power threshold.

[0150] The impact recognition manager 706 can be configured to generate (either directly or by invoking the functionality of another service or process such as the compute service 306 and / or the power management service 304) one or more instances of an estimated impact corresponding to a response level and / or a reduction action corresponding to a response level. The VEPO orchestration service 702 can obtain configuration data 712 that specifies a plurality of VEPO levels and their corresponding set of one or more reduction actions.

[0151] Figure 8 is Table 800 according to at least one embodiment, which illustrates a set of example response levels and the corresponding set of reduction actions for each response level. The data in Table 800 can be provided differently, such as by being mapped or embedded within the program code of the VEPO orchestration service 702. Each response level in the set of response levels can be associated with a corresponding set of reduction actions that can be performed on a plurality of resources (e.g., hosts, instances, workloads, etc.) in the data center, and each corresponding set of reduction actions is associated with a different reduction in the total power consumption of the data center. In some embodiments, the plurality of response levels indicate an increasing severity and / or which reduction actions to perform to reduce the total power consumption of the data center.

[0152] For example, Table 800 includes four VEPO response levels, but any suitable number of response levels can be used. The VEPO response level 1 can be associated with one reduction action. Specifically, the reduction action specifies that all vacant and / or idle hosts and hypervisors that can be evacuated (e.g., via migration) will be shut down (e.g., powered off).

[0153] In some embodiments, the VEPO response level 1 may include stopping or prohibiting the start or assignment of instances and / or workloads to the affected hosts. In some embodiments, the VEPO response level 1 may include reduction actions, which include migrating instances and / or workloads from one host to another host, or from one instance to another instance (in the case of workload migration).

[0154] The VEPO response level 2 may be associated with multiple actions (such as any / all reduction actions discussed above in connection with the VEPO response level 1) and one or more additional reduction actions. For example, an additional reduction action corresponding to shutting down the hypervisor hosting the instances and / or workloads associated with customers of category 1 (e.g., free-tier customers, free-trial customers, etc.).

[0155] The VEPO response level 3 may be associated with multiple actions (such as any / all reduction actions associated with the VEPO response level 1 and / or 2) and one or more additional reduction actions. For example, the VEPO response level 3 may include additional reduction actions corresponding to shutting down the hypervisors of category 2 bare-metal instances (e.g., pay-as-you-go and / or enterprise BM) and hosting category 3 virtual machines (e.g., VMs associated with pay-as-you-go and enterprise categories) (e.g., after evacuating higher-priority VMs). In some embodiments, the additional reduction actions may specify that BMs and / or VMs associated with customers and / or cloud services of priority level 1 (e.g., critical customers and cloud provider services) should be excluded from consideration.

[0156] The VEPO response level 4 may be associated with multiple actions (such as any / all reduction actions associated with the VEPO response levels 1, 2, and / or 3) and one or more additional reduction actions. For example, the VEPO response level 4 may include additional reduction actions corresponding to shutting down all compute hosts in the customer enclave, except those that remain in a critical state and / or are considered critical for recovery.

[0157] Figure 8 The number of response levels and / or specific reduction actions depicted is not intended to limit the scope of the present disclosure. Any suitable number of response levels may be employed, each corresponding to any suitable number of reduction actions. The reduction actions may include or exclude hosts, instances, workloads, and / or customers based on any suitable attributes associated therewith. For example, based on priority, category, status, and / or based on future recovery requirements. In some embodiments, the hosts, instances, and workloads considered candidates for applying these reduction actions may be based on a limited scope described in more detail Figure 9 in the following.

[0158] AlthoughFigure 8 The response levels depicted in Figure 8 include some overlap reduction actions, but this is not a requirement. In some embodiments, the reduction actions for each level may include a lesser or greater degree of overlap, including no overlap, resulting in completely unique actions associated with each level.

[0159] Figure 9 FIG. 900 is a schematic illustration of an example scope of components affected by one or more response levels of a VEPO orchestration service (e.g., Figure 7 the VEPO orchestration service 702 of Figure 9 ) for orchestrating power consumption constraints, migrations, pauses, and / or power-off tasks. The schematic illustration depicts a user realm 902 and a cloud service realm 904. The user realm 902 may include computing instances operating within a customer footprint (e.g., footprint 908) that run workloads (or any suitable workloads) corresponding to one or more services 906. The instances and / or workloads corresponding to one or more services 906 may be associated with one or more users 910.

[0160] In contrast, the cloud service realm 904 may include computing instances that run workloads corresponding to one or more services 912 (e.g., control and / or data plane services, such as those discussed in connection with Figures 15 - 18 service leasing) and / or resources (such as object storage devices, virtual cloud networking, etc.).

[0161] In some embodiments, although one or more hosts 324 may host any suitable combination of instances and / or workloads corresponding to one or more services 906 associated with a user (e.g., a customer) and / or one or more services 912 and / or resources associated with a cloud provider, only the hosts, instances, and / or workloads (e.g., computing resources) corresponding to the customer realm may be considered candidates to which reduction actions may be applied. In some embodiments, reduction actions may be completely avoided (at least for some period of time) for services and / or resources associated with the cloud provider because these services and / or resources are necessary for later recovery.

[0162] Now returning Figure 7 , in some embodiments, the functionality described in connection with the power management service 402 may be performed by one or more virtual machines implemented in a hosted computing environment (e.g., on one or more hosts 324, corresponding to Figure 1 any suitable number of servers 104 of Figure 9 ). The hosted computing environment may include one or more rapidly provisioned and released computing resources, which may include computing, networking, and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. Below, with respect to Figures 15 - 18Provide and discuss in more detail multiple example cloud computing environments.

[0163] The VEPO orchestration service 702 can be configured to manage power consumption corresponding to hosts / instances / workloads of the physical environment through functions provided by associated functions or other systems and / or services. Managing power consumption values can include power capping, pausing workloads, instances, hosts, shutting down workloads / instances / hosts, migrating an instance from one host to another, migrating a workload from one instance and / or host to another instance and / or host, etc.

[0164] The impact identification manager 706 can be configured to utilize configuration data 712 corresponding to specifications of multiple response levels and their corresponding mitigation actions (e.g., Figure 8 table 800) to determine the estimated impact of applying a mitigation action of a given level to hosts / instances / workloads (collectively referred to as "resources") of the physical environment. The estimated impact of applying a given action and / or level can depend on the current conditions of the runtime resources. In some embodiments, identifying the estimated impact of a given mitigation action can include identifying a set of resources and / or customers to which the action would apply if implemented. Thus, the estimated impact is intended to refer to the scope or applicability of a given action or level. Identifying the scope and / or applicability of a given action or level can include identifying the specific resources and / or customers that would be affected by implementing the action or level and / or identifying any appropriate attributes associated with the applicable resources and / or customers. As a simple example, the impact identification manager 706 can identify that if the action of shutting down all idle hosts is taken, then X idle hosts among the (one or more) hosts 324 would be affected, and these (one or more) hosts are associated with a particular customer or a particular number of customers, and / or the (one or more) hosts and / or customers are associated with other attributes (such as category (e.g., free tier), priority (e.g., high priority), etc.). Mitigation actions can provide progressively more severe actions and / or progressively broader impacts. As an example of a progressively severe action, a lower level can specify an action of power capping a particular host associated with a particular attribute, while a higher level can specify completely shutting down the host. Examples of progressively broader impacts can include a lower level that affects a small number of hosts (e.g., 2, 5, 10) corresponding to a single low-priority customer, while a higher level may affect a larger number of hosts (e.g., 100) corresponding to the same or more customers, and these hosts are associated with a wider range of priorities, etc. Thus, a broader impact refers to the quantity, scope, or range of applicability of a given action to hosts, instances, workloads, or customers.

[0165] The impact recognition manager 706 can be configured to estimate the impact that one or more levels may experience and / or estimate a power reduction, at least in part, based on configuration data specifying levels and corresponding actions and any suitable combination of power data 310, account data 312, host / instance data, etc. The estimated impact can be identified for one or more response levels, one or more reduction actions corresponding to a given level, etc. (e.g., which (one or more) hosts, (one or more) instances, (one or more) workloads, (one or more) customers may be affected, which attributes are associated with the affected (one or more) hosts, (one or more) instances, (one or more) workloads, and / or customers, how many (one or more) hosts / (one or more) instances / (one or more) workloads / (one or more) customers are affected, etc.). In some embodiments, the estimated impact of a given action can be aggregated with the estimated impacts of all actions for a given level to identify the estimated impact of the given level.

[0166] In some embodiments, the impact recognition manager 706 can be used to estimate the impact of each level and / or estimate a power reduction. In some embodiments, determining the estimated power reduction can be an incremental process, including identifying applicable resources (e.g., hosts, instances, and / or workloads), determining the current power consumption of the applicable resources, estimating the reduction for each response if the action is implemented / realized, and aggregating the individual estimated reductions of the individual responses to identify the total estimated power reduction corresponding to a given action. This process can be repeated for each action for a given level, and the final estimated power consumption reductions for each action can be aggregated to determine the total estimated power consumption reduction for the given level. The functionality of the impact recognition manager 706 can be invoked by the enforcement manager 708.

[0167] In some embodiments, the demand manager 710 can invoke the functionality of the enforcement manager 708, at least in part, based on monitoring or determining a need to change the total power threshold. Monitoring can include monitoring Figure 3Any suitable combination of power data 310, environmental data 314, host / instance data 316, and / or operational data 318. Modifying the total power threshold can be performed, and the amount by which the total power threshold is modified can be determined at least in part based on a plurality of trigger events. These trigger events can include, but are not limited to: 1) receiving a request from a government entity (e.g., a local power authority) to reduce power indefinitely or for a period of time; 2) determining current and / or predicted environmental conditions (e.g., current environmental or external temperature, predicted environmental or external temperature); 3) determining the current or predicted total power consumption corresponding to the devices in the physical environment; 4) determining the current and / or predicted operating status of one or more temperature control units in the physical environment. The amount by which the total power threshold can or will be modified can be determined at least in part based on a plurality of factors, including but not limited to: the difference between the environmental temperature and the external temperature, the power consumption attributed to or associated with a particular component (e.g., a particular temperature control unit), the amount or percentage specified in the request received from the government entity, or the difference between the current or predicted total power consumption value and the current or predicted total power threshold.

[0168] In some embodiments, after determining the amount of change in the total power threshold, the demand manager 710 can invoke the functionality of the enforcement manager 708 to determine which response level is appropriate for a given change. As a non-limiting example, the enforcement manager 708 can incrementally invoke the functionality of the impact identification manager 706 (e.g., via a function call or other means) to determine the impact associated with a given level (e.g., Figure 8The VEPO response level 1) corresponding estimated impact and / or estimated power reduction. The result estimated impact and / or estimated power reduction calculated by the impact identification manager can be returned to the enforcement manager 708. In some embodiments, the enforcement manager 708 can obtain and / or implement a predefined scenario to determine the appropriateness of selecting one response level over another. For example, the environment manager 708 can be configured to compare the estimated power reduction of a given level with the current total power threshold identified by the demand manager 710 to determine whether the estimated power consumption of the given level (e.g., VEPO response level 1) is sufficient (e.g., exceeds the change required to reduce the current total power consumption (e.g., identified and provided by the power management service 402) to below the total power threshold identified by the demand manager 710). If sufficient, then in some embodiments, the enforcement manager 708 can evaluate the estimated impact according to the predefined scenario to determine whether the estimated impact is sufficient and / or acceptable. As a non-limiting example, even if the estimated power reduction is sufficient to reduce the current total power consumption below the current total power threshold, if the estimated impact exceeds the conditions of the predefined scenario (e.g., the number of affected hosts exceeds the threshold identified in the predefined scenario), then the enforcement manager 708 can reject the response level. If the estimated power consumption and / or estimated impact of the response level fails to meet the requirements / conditions of the predefined scenario, then the enforcement manager 708 can be configured to call the impact identification manager 706 to determine the estimated impact and / or estimated power reduction corresponding to a higher VEPO response level (e.g., VEPO response level 2).

[0169] The estimation of impact and / or power reduction can be invoked by the enforcement manager 708 and determined by the impact identification manager 706 one level at a time until the enforcement manager 708 finds the first level that meets the requirements / conditions of the predefined scenario regarding the estimated impact and / or estimated power reduction. The level of the lowest level determined to meet all the impact conditions of the predefined scenario (e.g., the level that appears higher in Table 800) can be referred to as the "least impactful resource level". The level of the lowest level determined to meet all the power reduction conditions of the predefined scenario and / or to comply with (e.g., be below) the current total power threshold (e.g., the level that appears higher in Table 800) can be referred to as the "least reduced resource level". The level of the lowest level among the response levels determined to meet all the conditions regarding the estimated impact and estimated power reduction can be referred to as the "first sufficient response level". The enforcement manager 708 can be configured to select the response level, depending on the predefined scenario, that is the first to be identified as the least impactful, least reduced, or the first sufficient response level.

[0170] In some embodiments, enforcement manager 708 may cause impact identification manager 706 to determine an estimated impact and / or estimated power reduction for each level (or some subset, such as the first five, the next five, etc.). Enforcement manager 708 may be configured to select a particular response level based at least in part on comparing the estimated impact and / or estimated power reduction for each level to determine the response level with the least impact (e.g., having a relatively minimal impact), the least reduction (e.g., resulting in a minimal power reduction sufficient to reduce the current total power consumption below the current total power threshold), or the least reduction and the least impact (e.g., based at least in part on a weighted algorithm).

[0171] Impact Identification Manager 706 (perhaps by calling Figure 3 The estimated impacts and / or estimated reductions calculated by the computer service 306 and / or the functionality of the power management service 304, and / or any appropriate data (such as current values ​​and / or forecast values) on which these estimates rely can be presented via interface 11 discussed in more detail below.

[0172] As discussed herein, any suitable determination and / or identification by demand manager 710, impact identification manager 706, and / or enforcement manager 708 (or any suitable service or subroutine called therefrom) may rely on current data, historical data, and / or forecasted data. As a non-limiting example, demand manager 710 may be based on Figure 1 The amount by which the change needs to be made and / or the current total power threshold needs to be modified may be determined based at least in part on historical data (e.g., indicating one or more partial or complete failures or shutdowns of the temperature control unit(s) 112 or predicted operating conditions of the units over a future period of time. The amount by which the change needs to be made and / or the current total power threshold needs to be modified may be determined based at least in part on historical data (e.g., indicating one or more partial or complete failures or shutdowns of the temperature control unit(s) 112) and / or based on a machine learning model (e.g., in combination with Figure 12 The predicted condition is identified by the output provided by the model trained in the manner described above). In some embodiments, the amount of change in the total power threshold determined by the demand manager 710 can be based on a predefined formula and / or table. For example, a formula and / or table can be obtained from which the capacity associated with the temperature control unit can be determined and / or calculated. The table (if used) can indicate the amount of power consumption attributed to or associated with the temperature control unit. Therefore, a failure (e.g., a partial and / or complete failure) of the temperature control unit may be associated with all or a proportional share of the power consumption attributed to or associated with the temperature control unit (e.g., the amount of power consumption in the form of heat that the temperature control unit is configured to handle). Therefore, the demand manager 710 can determine that a complete failure of the temperature control unit requires reducing the total power threshold by an amount equal to the total power consumption attributed to or associated with the temperature control unit specified in the table.

[0173] In some embodiments, the enforcement manager 708 may be configured to present, recommend, or automatically select a given response level from a possible set of response levels based at least in part on any suitable combination of the estimated impact that may be experienced and / or the estimated power reduction if a reduction action corresponding to a given level is implemented (e.g., effected). Thus, in some embodiments, the VEPO orchestration service 702 may recommend and / or select a particular level based on any suitable combination of: 1) determining a level that, if implemented, would likely result in a sufficient but not excessive reduction in power consumption at the resources of the physical environment (e.g., the minimum power consumption reduction sufficient to bring the current power consumption value below the current total power threshold), 2) determining a level that includes a least restrictive set of actions, or 3) determining a level or action with the least impact (e.g., a set of affected resources and / or customers with the fewest resources / customers likely to be affected, a priority or category indicating the lowest degree of collective importance). Determining the least restrictive level or action, or the level or action with the least impact may be determined without regard to the estimated power consumption reduction that may be experienced by implementing that level and / or action, or determining the least restrictive level / action and / or the level / action with the least impact may be determined from one or more levels that, if implemented under given current conditions, are estimated to result in a sufficient reduction in power consumption to bring the power consumption value below an aggregated value of the current total power threshold. In some embodiments, the enforcement manager 708 may cause any suitable data corresponding to the current, predicted, or estimated power consumption and / or impact to be presented on Figure 11 the interface. In some embodiments, the selection of a particular level to be implemented may be driven by user input provided on a user interface. Thus, in some embodiments, the enforcement manager 708 may refrain from performing an operation to effect a reduction action for an implemented level until the user identifies and / or confirms the level to be implemented via user input provided on the user interface.

[0174] In some embodiments, performing these operations can include instructing the power management service 402 and / or the compute service 306 to perform operations. For example, the enforcement manager 708 can provide the power management service 402 with identifiers of affected resources (e.g., hosts, instances, workloads, etc.) and estimated power reduction and / or reduction action data specifying the actions to be taken (e.g., resources specific to power capping, power capping all low-priority hosts / instances / workloads, power capping specific or all resources meeting a set of specific attributes by a specific amount (e.g., 10%, at least x amount, etc.)). As described above, this can be configured to enact / implement corresponding power caps as directed by the enforcement manager 708. Similarly, the compute service 306 can be provided with identifiers of affected resources (e.g., hosts, instances, workloads, etc.) and reduction action data specifying the actions for the compute service 306 to take (e.g., shutting down all idle hosts, shutting down these specific hosts, pausing all low-priority workloads, migrating and / or compressing all low-priority workloads to maximize the number of idle / vacant hosts, etc.). As described above, this can be configured to enact / implement corresponding power caps as directed by the enforcement manager 708.

[0175] The demand manager 710 can also be configured to identify that a previous trigger that led to the achievement of a response level no longer occurs and can modify the total power threshold based on the identification that the trigger condition no longer occurs. For example, the demand manager 710 (e.g., via monitoring the operational data 318) can identify that a temperature control unit that previously failed is no longer operating. Accordingly, the demand manager 710 can modify (e.g., increase) the total power threshold by the amount attributed to the temperature control unit when fully operational. In accordance with this change, the enforcement manager 706 can be configured to perform any appropriate operations to reverse, as much as possible, the reduction actions taken previously in accordance with any response level selected in response to the original trigger condition. If power capping was used, then the enforcement manager 706 can instruct the power management service 402 to remove those previously applied power caps. If migrations of workloads and / or instances were performed, then the enforcement manager 706 can instruct the compute service 306 to attempt to migrate those resources back to the original hosts and / or instances hosting the instance and / or workload, respectively. If resources were paused and / or shut down, then the enforcement manager 706 can instruct the compute service 306 to perform operations to resume the instance and / or workload and / or send a signal to the shut-down host to start. In this way, the enforcement manager 706 can perform any appropriate operations to recover from the reduction actions taken based on a previously selected given response level.

[0176] Figure 10 is a block diagram illustrating an example use case of applying multiple response levels based on the current conditions of a set of hosts according to at least one embodiment.Figure 10 The examples provided in Figure 8 discussion of the VEPO response levels and corresponding actions are Figure 7 the current configuration data used by the VEPO orchestration service 702 of

[0177] · Servers 1004A and 1004C are vacant.

[0178] · Server 1004D is hosting resources

[0179] (e.g., instances) associated with a customer at category level 1.

[0180] · Server 1004I is hosting BM instances associated with a customer associated with priority level 2.

[0181] · Server 1004J is hosting the BM of a customer associated with priority level 3.

[0182] · Server 1004K is hosting the VM of a customer associated with priority level 1.

[0183] · Server 1004B is hosting one or more instances that are considered critical for recovery. In some embodiments, these instances may be associated with the service(s) 912 provided by the cloud provider.

[0184] As described above, the VEPO orchestration service 702 can identify the estimated impact and / or estimated power reduction corresponding to each response level either step - by - step or in parallel. The VEPO orchestration service 702 can identify a response applicable to a given level (e.g., a reduction action corresponding to a given response level) (e.g., through functionality provided by the power management service 304 and / or the compute service 306). For example, in the ongoing example, the VEPO orchestration service 702 (through the power management service 304) can identify that servers 1004A and 1004C are a set of resources for which the redundant action application of VEPO response level 1 is based on (e.g., from non - cloud - provider - based resources such as resources associated with a user's domain such as user domain 902), and thus identify that VEPO response level 1 is estimated to potentially impact 2 servers, namely servers 1004A and 1004C. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which the estimated power reduction for a given level (if enacted / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 10% reduction will have no impact on the customer. The idle BM capacity is reduced to 0%, the idle CM capacity is severely reduced, etc.). Any appropriate portion of this information can be presented on the user interface 1100, with similar data associated with other VEPO response levels either presented or not presented.

[0185] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources on which server 1004D is hosting redundant action applications at the VEPO response level 3 based on (e.g., from non-cloud-provider-based resources such as resources associated with a user's domain (such as user domain 902)), also identify that the VEPO response level 2, including a reduced action at the VEPO response level 1, is estimated to potentially affect servers 1004A and 1004C. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at the VEPO response level 2 (if formulated / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 20% reduction has a minimal impact on the customer. At most, 400 low-priority VMs are affected, etc.). Any appropriate portion of this information can be presented on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0186] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying that the BM hosted by server 1004I and the VM hosted by server 1004J are resources for redundant action applications at the VEPO response level 3 based on (e.g., from non-cloud-provider-based resources such as resources associated with a user's domain (such as user domain 902)), also identify that the VEPO response level 3, including reduced actions at the VEPO response levels 1 and 2, is estimated to potentially affect servers 1004A, 1004C, and 1004D. Resources hosted by server 1004K and / or server 1004K can be excluded based on meeting an exclusion associated with the evaluated response level. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at the VEPO response level 3 (if formulated / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 40% reduction has a moderate impact. At most, 200 category 1 or 2 BMs / VMs are affected, etc.). Any appropriate portion of this information can be displayed on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0187] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying that the BM hosted by the server 1004I and the VM hosted by the server 1004J are resources for redundant action applications at the VEPO response level 4 based on resources identified (e.g., from non-cloud-provider-based resources such as resources associated with a user premise (such as the user premise 902)), also identify that the VEPO response 4 estimate may affect all servers other than the server 1004B (e.g., a set of servers including the servers 1004A and 1004C - 1004L) based on the scaled actions including VEPO response levels 1 - 3. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which the estimated power reduction at the VEPO response level 4 (if formulated / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a reduction of 80% has a serious impact, regional downtime, etc.). Any appropriate portion of this information can be displayed on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0188] Figure 11 is a schematic diagram of an example user interface 1100 according to at least one embodiment. The user interface 1100 can include any appropriate number and type of graphical interface elements (e.g., dropdown boxes 1102 - 1108) to select and / or view VEPO data corresponding to a given region, availability domain (AD), building, and / or room, respectively. Other interface elements such as radio buttons and / or edit boxes can be used. The VEPO data presented on the user interface 1100 can correspond to the region, AD, building, and room selected via the graphical interface elements 1102 - 1108 (also referred to as "options" 1102 - 1108). As depicted, the VEPO data 1109 is displayed according to the values selected via the options 1102 - 1108.

[0189] As discussed above, any appropriate information used by the VEPO orchestration service 702, the power management service 304, and / or the computing service 306 can be presented via the user interface 1100. As a non-limiting example, any appropriate combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318 can be presented in the user interface 1100.

[0190] The specific content of the VEPO data 1109 displayed on the user interface 1100 may vary, but as depicted, the VEPO data 1109 includes VEPO level data (e.g., presented in column 1110 related to Figure 8the identifier corresponding to the VEPO response level), the current power consumption value (e.g., the current power consumption value presented in column 1112), the estimated reduction associated with each VEPO level (e.g., the estimated VEPO reduction presented in column 1114), and the estimated impact associated with each VEPO level (e.g., the estimated VEPO impact presented in column 1116).

[0191] The current total power threshold can be displayed at 1118 (e.g., the current total power threshold determined by Figure 7 the demand engine 710). The current power reduction required to meet the current total power threshold can be presented at 1120. This value (e.g., indicating a need to reduce by 5 kW) can be calculated based at least in part on identifying the difference between the current total power threshold (e.g., 11 kW) and the current power consumption presented at 1112 (e.g., 16 kW) (e.g., by the demand engine 710).

[0192] The user interface 1100 can be configured to present estimated impact data corresponding to each VEPO response level (referred to herein as "VEPO level" for brevity) within column 1116. In some embodiments, a region corresponding to a row of column 1116 can be selected to present an expanded view of the estimated impact data corresponding to a particular VEPO level. Similarly, a region corresponding to a row of column 1114 can be selected to present an expanded view of the estimated VEPO reduction corresponding to a particular VEPO level.

[0193] The user interface 1100 can include an indicator 1122 that indicates the recommended VEPO response level (as depicted, the system recommends VEPO level 2). As discussed above, the recommended VEPO level can be identified by the power orchestration service 702 based on the above factors. In some embodiments, the recommended VEPO level (e.g., VEPO level 2) can be selected based on identifying that VEPO level 2 is associated with the minimum estimated VEPO reduction amount to meet the required reduction indicated at 1120. In some embodiments, the VEPO level 2 can be recommended additionally or alternatively based at least in part on identifying that the estimated impact of applying VEPO level 2 provides a minimum impact relative to the estimated impact of VEPO level 104.

[0194] In some embodiments, a region corresponding to each row of VEPO data 1109 can be selected to select a given VEPO response level. Once selected, an additional menu or options can be provided to the user for indicating that the system continue with the selected VEPO response according to the reduction actions associated with that level.

[0195] Figure 12The flow diagram depicts an example method 1200 for training one or more machine learning models according to at least one embodiment. In some embodiments, any suitable machine learning algorithm (e.g., supervised, unsupervised, etc.) and any suitable number of training data sets (e.g., training data 1208) can be used to train the (one or more) models 1202 by (e.g., the Figure 3 power management service 304 of Figure 3 the power orchestration service 302 of Figure 3 the computing service 306, or different devices or systems). A supervised machine learning algorithm refers to a machine learning task that includes learning an inference function that maps inputs to outputs based on a labeled training data set of known example input / output pairs. An unsupervised machine learning algorithm refers to a set of algorithms for analyzing and clustering unlabeled data sets (e.g., unlabeled data 1210). These algorithms are configured to identify patterns or data groupings without human intervention. In some embodiments, any suitable number of the (one or more) models 1202 can be trained during the training phase 1204.

[0196] According to at least one embodiment, at least one of the (one or more) models can be trained to identify the likelihood or confidence that the total power consumption of a downstream device (e.g., the Figure 3 (one or more) hosts 324 of Figure 2 ) will exceed a budget threshold corresponding to an upstream device (e.g., a row-level device such as the Figure 3 (one or more) hosts 324 of Figure 1 (one or more) servers 104, etc.) that distributes power to the (one or more) hosts 324 and / or any suitable combination of the (one or more) servers 104. The training data 1208 used to train one or more of the (one or more) models 1202 can include any suitable combination of individual and / or total power consumption values (current or historical) of power data 310 corresponding to one or more hosts (e.g., the Figure 3 (one or more) hosts 324 of Figure 1 (one or more) servers 104, etc.). In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the (one or more) models can be trained to identify a prediction corresponding to the total power consumption of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 can include the current or historical power caps applied to the corresponding hosts, and the output provided by such (one or more) models 1208 can identify the predicted power cap values for any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104.

[0197] According to at least one embodiment, at least one of the one or more models can be trained to identify a predicted workload for any suitable combination of one or more hosts 324 and / or one or more servers 104. Training data 1208 for training one or more of the one or more models 1202 to identify a predicted workload can include any suitable combination of current and / or historical data of host / instance data 316 corresponding to one or more hosts (e.g., Figure 3 one or more hosts 324 of, Figure 1 one or more servers 104 of, etc.). In some embodiments, the training data 1208 for training these one or more specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more models can be trained to identify the likelihood or confidence that a predicted set of workloads will be assigned to any suitable combination of one or more hosts 324 and / or one or more servers 104.

[0198] According to at least one embodiment, at least one of the one or more models can be trained to identify a predicted partial or complete failure of one or more temperature control units. Training data 1208 for training one or more of the one or more models 1202 to identify such failures can include current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where one or more hosts 324 and / or one or more servers 104 are located, current or past external temperatures occurring relative to a region outside the physical environment (e.g., the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, and any suitable combination of current and / or historical failures experienced by one or more temperature control units (e.g., Figure 1 one or more temperature control systems 112 of). In some embodiments, the training data 1208 for training these one or more specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more models can be trained to identify the likelihood or confidence that a specific failure (partial or complete) of one or more temperature control systems 112 will occur during a future time period. In some embodiments, one or more models can be trained to identify the degree of failure of one or more temperature control systems 112 (e.g., 80% failure, 100% failure, etc.).

[0199] According to at least one embodiment, at least one of the (one or more) models can be trained to provide an estimated total power reduction required due to a partial and / or complete failure of one or more temperature control units (e.g., the (one or more) temperature control systems 112). The training data 1208 used to train one or more of the (one or more) models 1202 to identify the required estimated total power reduction can include any suitable combination of the following: current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current and / or historical external temperatures that currently or previously occurred relative to a region outside the physical environment (e.g., in the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, current and / or historical failures experienced by one or more temperature control units (e.g., Figure 1 the (one or more) temperature control systems 112), current and / or historical total power thresholds, and / or current and / or historical power consumption levels of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more of the (one or more) models can be trained to identify the likelihood or confidence that an estimated total power reduction will be required during a future time period.

[0200] According to at least one embodiment, at least one of the (one or more) models can be trained to predict future temperatures (e.g., ambient temperature and / or external temperature). The training data 1208 used to train one or more of the (one or more) models 1202 to identify future temperatures can include any suitable combination of the following: current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current and / or historical external temperatures that currently or previously occurred relative to a region outside the physical environment (e.g., in the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, one or more temperature control units (e.g., Figure 1the current and / or historical faults experienced by the (one or more) temperature control systems 112, the current and / or historical total power thresholds, and / or the current and / or historical power consumption levels of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 may include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, (one or more) models may be trained to identify the likelihood or confidence of experiencing a predicted temperature during a future time period.

[0201] Generally, the (one or more) models 1202 may include any suitable number of models. The (one or more) models 1202 may be individually trained to identify the likelihood or confidence or accuracy of the output provided by the (one or more) models 1202 from the training data 1208 discussed above. In some embodiments, the (one or more) models 1202 may be configured to determine values corresponding to the examples provided above, and the likelihood and / or confidence values may indicate the likelihood or confidence that the output provided by the (one or more) models is accurate.

[0202] Generally, the (one or more) models 1202 may be trained during the training phase 1204 using supervised learning algorithms and labeled data 1206 to identify the outputs described in the examples above. The likelihood value may be a binary indicator, a percentage, a confidence value, etc., which indicates the degree of likelihood. The likelihood value may be a binary indicator that indicates whether a specific budget amount is likely or unlikely to be breached, or the likelihood value may indicate the likelihood and / or confidence that a predicted output will occur in the future. The labeled data 1206 may be any suitable portion of the potential training data (e.g., training data 1208) that can be used to train the (one or more) models to produce the outputs described above. The labeled data 1206 may include any suitable number of examples of current and / or historical data corresponding to any suitable number of devices in the physical environment (e.g., data center 102, the (one or more) servers 104, the (one or more) temperature control systems 112, Figure 2 PDUs, etc.). In some embodiments, the labeled data 1206 may include labels identifying known likelihoods and / or actual values. Using the labeled data 1206, a model (e.g., an inference function) that maps inputs (e.g., one or more instances of the training data) to outputs (e.g., predicted values or the likelihood / confidence values that the predicted value or output will occur during a future time period) can be learned. In some embodiments, Figure 3Any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306, or a separate service or system, can train any suitable combination of the (one or more) models 1202. In some embodiments, any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306 can obtain a pre-trained version of the (one or more) models 1202.

[0203] (One or more) models 1202 and the various types of models discussed above can include any suitable number of models that are trained using unsupervised learning techniques to identify likelihoods / confidences and / or predictions corresponding to the examples provided above. Unsupervised machine learning algorithms are configured to learn patterns from unlabeled data. In some embodiments, the training phase 1204 can utilize an unsupervised machine learning algorithm to generate one or more of the (one or more) models 1202. For example, the training data 1208 can include unlabeled data 1210 (e.g., any suitable combination of current and / or historical values corresponding to power data 310, account data 12, environmental data 314, host / instance data 316, operational data 318, etc.). The unlabeled data 1210 can be used with an unsupervised learning algorithm to divide the entries of the unlabeled data 1210 into groups. The unsupervised learning algorithm can be configured to group similar entries together into common groups. Examples of unsupervised learning algorithms can include clustering methods such as k-means clustering, DBScan, etc. In some embodiments, the unlabeled data 1210 can be clustered with the labeled data 1206 such that the unlabeled instances of a given group can be assigned the same label as the other labeled instances within that group.

[0204] In some embodiments, any suitable portion of the training data 1208 can be used to train the model(s) 1202 during the training phase 1204. For example, 70% of the labeled data 1206 and / or the unlabeled data 1210 can be used to train the model(s) 1202. Once trained, or at any suitable time, the model(s) 1202 can be evaluated to assess its quality (e.g., the accuracy of the output(s) 1212 relative to the labels corresponding to the labeled data 1206). For example, a portion of the examples of the labeled data 1206 and / or the unlabeled data 1210 can be used as input to the model(s) 1202 to generate the output(s) 1212. For example, an example of the labeled data 1206 can be provided as input, and the corresponding output (e.g., the output(s) 1212) can be compared to the label known to be associated with that example. If a portion of the output (e.g., a label) matches the example label, then that portion of the output can be considered accurate. Any suitable number of labeled examples can be used, and the number of accurate labels can be compared to the total number of examples provided (and / or the total number of previously identified labels) to determine an accuracy value for a given model, which quantifies how accurate the model is. For example, if 90 out of 100 input examples generate output labels that match the previously known example labels, then the model being evaluated can be determined to be 90% accurate.

[0205] In some embodiments, when the model(s) 1202 is used for subsequent inputs, the subsequent output(s) generated by the model(s) 1202 can be added to the corresponding input and used to retrain and / or update the model(s) 1202 at 1216. In some embodiments, examples may not be used to retrain or update the model until the feedback process 1214 is performed. In the feedback process 1214, an example (e.g., an example including one or more historical consumption data instances corresponding to one or more devices and / or racks) and the corresponding output generated by one of the model(s) 1202 for the example are presented to a user, and the user identifies whether the generated output (e.g., a quantity and / or a likelihood confidence value) is correct for the given example.

[0206] Figure 12 The training process depicted (e.g., method 1200) can be performed any suitable number of times at any suitable interval and / or according to any suitable schedule such that the accuracy of the model(s) 1202 improves over time.

[0207] In some embodiments, any suitable number and / or combination of (one or more) models 1202 can be used to determine an output. In some embodiments, the power management service 402 can utilize any suitable combination of outputs provided by (one or more) models 1202 to determine whether a budget threshold for a given component (e.g., a row-level PDU) is likely to be breached (and / or likely to be breached by a certain amount). Thus, in some embodiments, the power management service 402 can utilize models trained using any suitable combination of supervised and unsupervised learning algorithms.

[0208] Figure 13 is a block diagram illustrating an example method for causing an application to associate a corresponding set of reduction actions with response levels according to at least one embodiment. Method 1300 can be performed by one or more components of the Figure 3 power orchestration system 300 or in combination with Figure 4 and Figure 7 its subcomponents discussed. By way of example. Method 1300 can be performed at least in part by any suitable combination of the Figure 3 VEPO orchestration service 302, power management service 304, and / or compute service 306 of Figure 13 . The operations of method 1300 can be performed in any suitable order. Method 1300 can include more or fewer operations than those depicted in

[0209] Method 1300 can begin at 1302, where configuration data (e.g., the Figure 2 configuration data 712 of Figure 7 ) can be obtained (e.g., by the Figure 8 power orchestration service 702 of Figure 8 ). In some embodiments, the configuration data can include multiple response levels (e.g., the Figure 8 VEPO response levels of

[0210] which specify the applicability of a corresponding set of reduction actions to multiple hosts (e.g., via one or more reduction actions corresponding to each response level). In some embodiments, a first response level among the multiple response levels specifies the applicability of a first set of reduction actions to multiple hosts (e.g., at least in part based on one or more reduction actions associated with that response level). By way of example, the Figure 8 VEPO response level can specify that all vacant / idle hosts and hypervisors that can be fully evacuated are applicable to an action (e.g., power off). Thus, a reduction action indicates that a subset of resources (e.g., vacant / idle hosts and hypervisors that can be fully evacuated) is applicable to a given reduction action (e.g., power off), and by association indicates the corresponding response level.At 1304, a current value of the total power consumption of multiple hosts can be obtained (e.g., obtained by power management service 402, which can provide such data to power orchestration service 702). For example, the power management service 402 can calculate the current value of the total power consumption of multiple hosts based at least in part on Figure 3 the power data 310 and / or the power consumption values obtained from / to the rack PDU 320 and / or the power consumption values obtained from any suitable combination of the host(s) 324 (e.g., the hosts in rack 319, all hosts, etc.).

[0211] At 1306, a current value of the total power threshold of multiple hosts can be obtained (e.g., by the Figure 7 demand manager 710 in). In some embodiments, the total power threshold can be an aggregation of all known power consumption related capabilities of any suitable device in the physical environment where the multiple hosts are located. As a non-limiting example, the total power threshold can be an aggregation of a budgeted power level and / or the heat / power consumption that the device (e.g., the temperature control unit such as Figure 1 the temperature control unit 112A of) is configured to dispose / handle when operating fully. In some embodiments, the current total power threshold can be calculated by the demand manager 710 in the manner described in connection with the above figure and based on the factors and / or data described in connection with the above figure. In some embodiments, the current total power threshold can be dynamically changed (e.g., by the demand manager 710) based on changing conditions of the devices in the physical environment where the multiple hosts are located (e.g., including changing conditions of the server(s) 104, changing conditions of the temperature control system(s) 112, changes in power consumption, changes in temperature, changes in the operating status of the devices in the physical environment, etc.). In some embodiments, the current total power threshold can be modified / changed at least in part based on receiving a request / requirement for reduction from a government entity (e.g., local power authority), where the aspects of the requested reduction can be specified in the request.

[0212] At 1308, a first response level can be selected (e.g., by the Figure 7 enforcement manager 708 in) from multiple response levels (e.g., the Figure 8 VEPO response level in) based at least on the difference between the current value of the total power consumption and the current value of the total power threshold. In some embodiments, the selection can be initiated at least in part based on receiving a user input (e.g., at the user interface 1100) indicating the selection of the first response level.

[0213] At 1310, the computer system may apply a first set of reduction actions to at least one of the multiple hosts based on the selected first response level. In some embodiments, causing the application of the first set of reduction actions may include performing and / or instructing the performance of any suitable operations discussed above in connection with the power orchestration service 302, the power management service 304, and / or the compute service 306. In some embodiments, if instructions are used, the instructions may originate from any one of the power orchestration service 302, the power management service 304, and / or the compute service 306, to one of the other services, which instructs that service to perform one or more operations discussed above in connection with Figure 3 , Figure 4 and Figure 7 .

[0214] Figure 14 FIG. is a block diagram illustrating an example method for pre-migrating workloads affected by a selected response level according to at least one embodiment. Method 1400 may be performed by one or more components of the power orchestration system 300 of Figure 3 or in conjunction with Figure 4 and Figure 7 discussed sub-components thereof. By way of example. Method 1400 may be performed at least in part by any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the compute service 306 of Figure 3 . The operations of method 1400 may be performed in any suitable order. Method 1400 may include more or fewer operations than those depicted in Figure 14 .

[0215] Method 1402 may begin at 1402, where configuration data (e.g., the configuration data 712 of Figure 2 ) may be obtained (e.g., by the power orchestration service 702 of Figure 7 ). In some embodiments, the configuration data may include multiple response levels (e.g., the VEPO response levels of Figure 8 ), which specify the applicability of a corresponding set of reduction actions to the multiple hosts (e.g., via one or more reduction actions corresponding to each response level). In some embodiments, the first response level of the multiple response levels specifies the applicability of the first set of reduction actions to the multiple hosts (e.g., at least in part based on one or more reduction actions associated with that response level). By way of example, Figure 8 the VEPO response level of

[0216] At 1404, predicted values of the total power consumption of multiple hosts over a future time period can be obtained (e.g., obtained by power management service 402, which can provide such data to power orchestration service 702). The predicted values of the total power consumption of multiple hosts can be calculated, for example, by power management service 402 at least in part based on Figure 3 the power data 310 and / or power consumption values obtained from the rack PDU 320 and / or power consumption values obtained from any suitable combination of (one or more) hosts 324 (e.g., the hosts of rack 319, all hosts, etc.). In some embodiments, the predicted values of the total power consumption of multiple hosts can utilize historical power data. In still other environments, the predicted values of the total power consumption can utilize a previously trained machine learning model (e.g., Figure 12 one or more of the models 1202). The machine learning model can be pre-trained using supervised or unsupervised machine learning algorithms and a training data set (e.g., training data 1208), as described in connection with Figure 12 to provide an output (e.g., a predicted value of the total power consumption) at least in part based on the provided inputs (e.g., Figure 3 any suitable combination of the power data 310, account data 312, environmental data 314, host / interface data 316, and / or operation data 318).

[0217] At 1406, predicted values of the total power threshold of multiple hosts can be obtained (e.g., by Figure 7 the demand manager 710). In some embodiments, the predicted total power threshold can be an aggregation of all known and / or predicted power consumption-related capabilities of any suitable device in the physical environment in which the multiple hosts are located. As a non-limiting example, the predicted total power threshold can be a budget and / or a predicted power level and / or the amount or prediction of heat / power consumption that a device (e.g., a temperature control unit such as Figure 1 the temperature control unit 112A) is configured to handle / dispose of when fully operational. In some embodiments, the predicted total power threshold can be calculated by the demand manager 710 in a manner described in connection with the above figure and based on the factors and / or data described in connection with the above figure. In some embodiments, the predicted total power threshold can be determined at least in part based on any suitable combination of current and historical data of the power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In still other environments, the predicted values of the total power consumption can utilize a previously trained machine learning model (e.g., Figure 12 one or more of the models 1202). The machine learning model can be previously trained using supervised or unsupervised machine learning algorithms and a training data set (e.g., training data 1208), as described in connection with Figure 12As described, to provide an output (e.g., a predicted value of a total power threshold) based at least in part on received input data (e.g., Figure 3 any suitable combination of power data 310, account data 312, environmental data 314, host / interface data 316, and / or operation data 318).

[0218] At 1408, it is possible to (e.g., by Figure 7 the enforcement manager 708) select a first response level from a plurality of response levels (e.g., Figure 8 the VEPO response levels) based at least on the difference between the current value of the total power consumption and the predicted value of the total power threshold. In some embodiments, the selection can be initiated based at least in part on user input received (e.g., at the user interface 1100) indicating the selection of the first response level.

[0219] At 1410, the computer system can identify one or more workloads according to the selected first response level, where the workloads (a) are currently being executed on a plurality of hosts and (b) will be affected by applying a first set of reduction actions to the plurality of hosts. The identification of the one or more workloads can be performed in the manner described above in connection with Figure 7 the impact identification manager 706.

[0220] At 1412, the computer system can pre-migrate the affected workloads from the plurality of hosts to one or more other hosts before a future time period. In some embodiments, these operations can be performed by the compute service 306 (e.g., based at least in part on instructions received by the service from the power orchestration service 302).

[0221] Figures 15 - 18 depicts a plurality of example environments that can be hosted by Figure 3 the (one or more) hosts 324. Figures 15 - 18 The environments depicted in [description] depict a cloud computing, multi-tenant environment. As described above, cloud computing and / or multi-tenant environments, as well as other environments, can benefit from leveraging the power management techniques disclosed herein. These techniques enable data centers including components implementing the environments described in connection with Figures 1 - 4 and Figure 7 etc. to utilize the power resources of the data center more efficiently than conventional techniques that result in a large amount of unused power.

[0222] IaaS Infrastructure Example

[0223] Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also provide various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Thus, since these services may be policy-driven, IaaS users can be able to implement policies to drive load balancing to maintain the availability and performance of applications.

[0224] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of an application stack. For example, a user can log in to an IaaS platform to create a virtual machine (VM), install an operating system (OS) on each VM, deploy middleware such as a database, create buckets for workloads and backups, and even install enterprise software into that VM. Then, the customer can use the provider's services to perform various functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, etc.

[0225] In most cases, the cloud computing model will require the involvement of a cloud provider. The cloud provider can be but is not necessarily a third-party service that specifically provides (e.g., provisions, rents, sells) IaaS. An entity may also choose to deploy a private cloud and thus become its own infrastructure service provider.

[0226] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It can also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider and is below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Thus, the customer can be responsible for the processing of the (OS), middleware, and / or application deployment (e.g., on a self-service virtual machine, etc. that can be launched on demand).

[0227] In some examples, IaaS provisioning can refer to obtaining computers or virtual hosts for use and even installing the required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0228] In some cases, there are two different challenges in IaaS provisioning. First, there is an initial challenge in provisioning the initial set of infrastructure before anything can run. Second, once everything has been provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two challenges can be addressed by enabling the configuration of the infrastructure to be defined in a declarative manner. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, a workflow for creating and / or managing the different components described in the configuration files can be generated.

[0229] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shareable computing resources), also known as the core network. In some examples, there can also be one or more inbound / outbound traffic group rules that are provisioned to define how inbound and / or outbound traffic of the network will be set up, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. can also be provisioned. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve incrementally.

[0230] In some cases, continuous deployment techniques can be employed to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is expected to be deployed to one or more but typically many different production environments (e.g., across various different geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, code can be deployed using deployment tools.

[0231] Figure 15 FIG. 1500 is a block diagram illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 1502 can be communicatively coupled to a secure host lease 1504, which can include a virtual cloud network (VCN) 1506 and a secure host subnet 1508. In some examples, the service operator 1502 can use one or more client computing devices, which can be portable handheld devices (e.g., cellular phones, a computing tablet, a personal digital assistant (PDA), or a wearable device (e.g., Google Head-Mounted Display), running software such as Microsoft Windows and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and enabling Internet, email, Short Message Service (SMS), or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, for example, including personal computers and / or laptop computers running various versions of Microsoft Apple and / or Linux operating systems. The client computing device can be a workstation computer running various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, such as, for example, Google Chrome OS). Alternatively, or additionally, the client computing device can be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a gesture input device) and / or a personal messaging device, which can communicate via a network that can access VCN 1506 and / or the Internet.

[0232] VCN 1506 can include a Local Peer Gateway (LPG) 1510, which can be communicatively coupled to a Secure Shell (SSH) VCN 1512 via the LPG 1510 included in the SSH VCN 1512. The SSH VCN 1512 can include an SSH subnet 1514, and the SSH VCN 1512 can be communicatively coupled to a control plane VCN 1516 via the LPG 1510 included in the control plane VCN 1516. Moreover, the SSH VCN 1512 can be communicatively coupled to a data plane VCN 1518 via the LPG 1510. The control plane VCN 1516 and the data plane VCN 1518 can be included in a service lease 1519 that can be owned and / or operated by an IaaS provider.

[0233] The control plane VCN 1516 may include a control plane demilitarized zone (DMZ) layer 1520 that acts as a perimeter network (e.g., a portion of a corporate network between an intranet and an external network). Servers based on the DMZ can assume limited liability and help control vulnerabilities. Additionally, the DMZ layer 1520 may include one or more load balancer (LB) subnets 1522, a control plane application layer 1524 that may include one or more application subnets 1526, and a control plane data layer 1528 that may include one or more database (DB) subnets 1530 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). The one or more LB subnets 1522 included in the control plane DMZ layer 1520 may be communicatively coupled to the one or more application subnets 1526 included in the control plane application layer 1524 and an Internet gateway 1534 that may be included in the control plane VCN 1516, and the one or more application subnets 1526 may be communicatively coupled to the one or more DB subnets 1530 included in the control plane data layer 1528, as well as a service gateway 1536 and a network address translation (NAT) gateway 1538. The control plane VCN 1516 may include a service gateway 1536 and a NAT gateway 1538.

[0234] The control plane VCN 1516 may include a data plane mirror application layer 1540, which may include one or more application subnets 1526. The one or more application subnets 1526 included in the data plane mirror application layer 1540 may include virtual network interface controllers (VNICs) 1542 that may execute compute instances 1544. The compute instances 1544 may communicatively couple the one or more application subnets 1526 of the data plane mirror application layer 1540 to the one or more application subnets 1526 that may be included in the data plane application layer 1546.

[0235] The data plane VCN 1518 may include a data plane application layer 1546, a data plane DMZ layer 1548, and a data plane data layer 1550. The data plane DMZ layer 1548 may include one or more LB subnets 1522, which may be communicatively coupled to the one or more application subnets 1526 of the data plane application layer 1546 and an Internet gateway 1534 of the data plane VCN 1518. The one or more application subnets 1526 may be communicatively coupled to a service gateway 1536 of the data plane VCN 1518 and a NAT gateway 1538 of the data plane VCN 1518. The data plane data layer 1550 may further include one or more DB subnets 1530 that may be communicatively coupled to the one or more application subnets 1526 of the data plane application layer 1546.

[0236] The Internet gateway 1534 for the control plane VCN 1516 and the data plane VCN 1518 can be communicatively coupled to a metadata management service 1552, and the metadata management service 1552 can be communicatively coupled to the public Internet 1554. The public Internet 1554 can be communicatively coupled to the NAT gateway 1538 for the control plane VCN 1516 and the data plane VCN 1518. The service gateway 1536 for the control plane VCN 1516 and the data plane VCN 1518 can be communicatively coupled to cloud services 1556.

[0237] In some examples, the service gateway 1536 for the control plane VCN 1516 or the data plane VCN 1518 can make application programming interface (API) calls to the cloud services 1556 without going through the public Internet 1554. The API calls from the service gateway 1536 to the cloud services 1556 can be one-way: the service gateway 1536 can make API calls to the cloud services 1556, and the cloud services 1556 can send the requested data to the service gateway 1536. However, the cloud services 1556 may not initiate API calls to the service gateway 1536.

[0238] In some examples, the secure host lease 1504 can be directly connected to the service lease 1519, which would otherwise be isolated. The secure host subnet 1508 can communicate with the SSH subnet 1514 via the LPG 1510, and the LPG 1510 can enable two-way communication on otherwise isolated systems. Connecting the secure host subnet 1508 to the SSH subnet 1514 can enable the secure host subnet 1508 to access other entities within the service lease 1519.

[0239] The control plane VCN 1516 can allow users of the service lease 1519 to set or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1516 can be deployed or otherwise used in the data plane VCN 1518. In some examples, the control plane VCN 1516 can be isolated from the data plane VCN 1518, and the data plane mirror application layer 1540 of the control plane VCN 1516 can communicate with the data plane application layer 1546 of the data plane VCN 1518 via the VNIC 1542, and the VNIC 1542 can be included in the data plane mirror application layer 1540 and the data plane application layer 1546.

[0240] In some examples, a user or customer of the system can make requests, such as create, read, update, or delete (CRUD) operations, via a public Internet 1554 that can transmit requests to a metadata management service 1552. The metadata management service 1552 can transmit requests to a control plane VCN 1516 via an Internet gateway 1534. The requests can be received by one or more LB subnets 1522 included in a control plane DMZ layer 1520. The one or more LB subnets 1522 can determine that the requests are valid, and in response to that determination, the one or more LB subnets 1522 can transmit the requests to one or more application subnets 1526 included in a control plane application layer 1524. If the requests are verified and require a call to the public Internet 1554, then the call to the public Internet 1554 can be transmitted to a NAT gateway 1538 that can make calls to the public Internet 1554. Metadata that the requests may expect to be stored can be stored in one or more DB subnets 1530.

[0241] In some examples, a data plane mirror application layer 1540 can facilitate direct communication between a control plane VCN 1516 and a data plane VCN 1518. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources included in the data plane VCN 1518. Via a VNIC 1542, the control plane VCN 1516 can communicate directly with resources included in the data plane VCN 1518 and thereby can perform configuration changes, updates, or other appropriate modifications.

[0242] In some embodiments, a control plane VCN 1516 and a data plane VCN 1518 can be included in a service tenancy 1519. In such a case, a user or customer of the system may not own or operate the control plane VCN 1516 or the data plane VCN 1518. Instead, an IaaS provider can own or operate the control plane VCN 1516 and the data plane VCN 1518, both of which can be included in the service tenancy 1519. This embodiment can enable isolation of networks that may prevent a user or customer from interacting with resources of other users or other customers. Moreover, this embodiment can allow a user or customer of the system to privately store databases without relying on the public Internet 1554, which may not have a desired threat prevention level, for storage.

[0243] In other embodiments, the (one or more) LB subnets 1522 included in the control plane VCN 1516 may be configured to receive signals from the service gateway 1536. In this embodiment, the control plane VCN 1516 and the data plane VCN 1518 may be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1554. A customer of the IaaS provider may desire this embodiment because the (one or more) databases used by the customer may be controlled by the IaaS provider and may be stored on the service lease 1519, which may be isolated from the public Internet 1554.

[0244] Figure 16 is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1602 (e.g., Figure 15 the service operator 1502) may be communicatively coupled to a secure host lease 1604 (e.g., Figure 15 the secure host lease 1504), which may include a virtual cloud network (VCN) 1606 (e.g., Figure 15 the VCN 1506) and a secure host subnet 1608 (e.g., Figure 15 the secure host subnet 1508). The VCN 1606 may include a local peering gateway (LPG) 1610 (e.g., Figure 15 the LPG 1510), which may be communicatively coupled to a secure shell (SSH) VCN 1612 (e.g., Figure 15 the SSH VCN 1512) via the LPG 1510 included in the SSH VCN 1612. The SSH VCN 1612 may include an SSH subnet 1614 (e.g., Figure 15 the SSH subnet 1514), and the SSH VCN 1612 may be communicatively coupled to a control plane VCN 1616 (e.g., Figure 15 the control plane VCN 1516) via the LPG 1610 included in the control plane VCN 1616. The control plane VCN 1616 may be included in a service lease 1619 (e.g., Figure 15 the service lease 1519), and a data plane VCN 1618 (e.g., Figure 15 the data plane VCN 1518) may be included in a customer lease 1621 that may be owned or operated by a user or customer of the system.

[0245] The control plane VCN 1616 may include a control plane DMZ layer 1620 (e.g., Figure 15 the control plane DMZ layer 1520), which may include the (one or more) LB subnets 1622 (e.g.,Figure 15 one or more LB subnets 1522), and may include one or more application subnets 1626 (e.g., Figure 15 one or more application subnets 1526) of the control plane application layer 1624 (e.g., Figure 15 control plane application layer 1524), and may include one or more database (DB) subnets 1630 (e.g., similar to Figure 15 one or more DB subnets 1530) of the control plane data layer 1628 (e.g., Figure 15 control plane data layer 1528). One or more LB subnets 1622 included in the control plane DMZ layer 1620 may be communicatively coupled to one or more application subnets 1626 included in the control plane application layer 1624 and an Internet gateway 1634 that may be included in the control plane VCN 1616 (e.g., Figure 15 Internet gateway 1534), and one or more application subnets 1626 may be communicatively coupled to one or more DB subnets 1630 included in the control plane data layer 1628, as well as a service gateway 1636 (e.g., Figure 15 service gateway 1536) and a network address translation (NAT) gateway 1638 (e.g., Figure 15 NAT gateway 1538). The control plane VCN 1616 may include a service gateway 1636 and a NAT gateway 1638.

[0246] The control plane VCN 1616 may include a data plane mirror application layer 1640 that may include one or more application subnets 1626 (e.g., Figure 15 data plane mirror application layer 1540). One or more application subnets 1626 included in the data plane mirror application layer 1640 may include a virtual network interface controller (VNIC) 1642 that may execute a compute instance 1644 (e.g., similar to Figure 15 compute instance 1544) of 1542). The compute instance 1644 may facilitate communication between one or more application subnets 1626 of the data plane mirror application layer 1640 and one or more application subnets 1626 that may be included in the data plane application layer 1646 (e.g., Figure 15 data plane application layer 1546) via the VNIC 1642 included in the data plane mirror application layer 1640 and the VNIC 1642 included in the data plane application layer 1646.

[0247] The Internet gateway 1634 included in the control plane VCN 1616 may be communicatively coupled to a metadata management service 1652 (e.g.,Figure 15 a metadata management service 1552), and the metadata management service 1652 can be communicatively coupled to a public Internet 1654 (e.g., Figure 15 the public Internet 1554). The public Internet 1654 can be communicatively coupled to a NAT gateway 1638 included in the control plane VCN 1616. A service gateway 1636 included in the control plane VCN 1616 can be communicatively coupled to a cloud service 1656 (e.g., Figure 15 the cloud service 1556).

[0248] In some examples, the data plane VCN 1618 can be included in a customer tenancy 1621. In such a case, the IaaS provider can provide a control plane VCN 1616 for each customer, and the IaaS provider can set up a unique compute instance 1644 included in a service tenancy 1619 for each customer. Each compute instance 1644 can permit communication between the control plane VCN 1616 included in the service tenancy 1619 and the data plane VCN 1618 included in the customer tenancy 1621. The compute instance 1644 can permit resources provisioned in the control plane VCN 1616 included in the service tenancy 1619 to be deployed or otherwise used in the data plane VCN 1618 included in the customer tenancy 1621.

[0249] In other examples, a customer of the IaaS provider can have a database that exists in the customer tenancy 1621. In this example, the control plane VCN 1616 can include a data plane mirror application layer 1640, which can include one or more application subnets 1626. The data plane mirror application layer 1640 can reside in the data plane VCN 1618, but the data plane mirror application layer 1640 may not be in the data plane VCN 1618. That is, the data plane mirror application layer 1640 can access the customer tenancy 1621, but the data plane mirror application layer 1640 may not exist in the data plane VCN 1618 or be owned or operated by the customer of the IaaS provider. The data plane mirror application layer 1640 can be configured to make calls to the data plane VCN 1618, but may not be configured to make calls to any entity included in the control plane VCN 1616. The customer may expect to deploy or otherwise use resources provisioned in the control plane VCN 1616 in the data plane VCN 1618, and the data plane mirror application layer 1640 can facilitate the customer's desired deployment or other use of the resources.

[0250] In some embodiments, a customer of an IaaS provider can apply filters to a data plane VCN 1618. In this embodiment, the customer can determine what the data plane VCN 1618 can access, and the customer can restrict access from the data plane VCN 1618 to the public Internet 1654. The IaaS provider may not be able to apply filters or otherwise control the data plane VCN 1618's access to any external network or database. Applying filters and controls by the customer to the data plane VCN 1618 included in the customer's tenancy 1621 can help isolate the data plane VCN 1618 from other customers and the public Internet 1654.

[0251] In some embodiments, a cloud service 1656 can be invoked by a service gateway 1636 to access services that may not be present on the public Internet 1654, the control plane VCN 1616, or the data plane VCN 1618. The connection between the cloud service 1656 and the control plane VCN 1616 or the data plane VCN 1618 may not be real-time or continuous. The cloud service 1656 can exist on a different network owned or operated by the IaaS provider. The cloud service 1656 can be configured to receive calls from the service gateway 1636 and can be configured not to receive calls from the public Internet 1654. Some cloud services 1656 can be isolated from other cloud services 1656, and the control plane VCN 1616 can be isolated from cloud services 1656 that may not be in the same region as the control plane VCN 1616. For example, the control plane VCN 1616 can be located in "Region 1", and the cloud service "Deployment 15" can be located in Region 1 and "Region 2". If the service gateway 1636 included in the control plane VCN 1616 located in Region 1 makes a call to Deployment 15, then the call can be transmitted to Deployment 15 in Region 1. In this example, the control plane VCN 1616 or Deployment 15 in Region 1 may not be communicatively coupled or otherwise communicate with Deployment 15 in Region 2.

[0252] Figure 17 is a block diagram 1700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1702 (e.g., Figure 15 the service operator 1502) can be communicatively coupled to a secure host tenancy 1704 (e.g., Figure 15 the secure host tenancy 1504), which can include a virtual cloud network (VCN) 1706 (e.g., Figure 15 the VCN 1506) and a secure host subnet 1708 (e.g., Figure 15 the secure host subnet 1508). The VCN 1706 can include an LPG 1710 (e.g.,Figure 15 The LPG 1510), which can be communicatively coupled to the SSH VCN 1712 via the LPG 1710 included in the SSH VCN 1712 (e.g., Figure 15 the SSH VCN 1512). The SSH VCN 1712 may include an SSH subnet 1714 (e.g., Figure 15 the SSH subnet 1514), and the SSH VCN 1712 may be communicatively coupled to the control plane VCN 1716 via the LPG 1710 included in the control plane VCN 1716 (e.g., Figure 15 the control plane VCN 1516) and coupled to the data plane VCN 1718 via the LPG 1710 included in the data plane VCN 1718 (e.g., Figure 15 the data plane 1518). The control plane VCN 1716 and the data plane VCN 1718 may be included in the service lease 1719 (e.g., Figure 15 the service lease 1519).

[0253] The control plane VCN 1716 may include a control plane DMZ layer 1720 that may include one or more load balancer (LB) subnets 1722 (e.g., Figure 15 one or more LB subnets 1522), a control plane application layer 1724 that may include one or more application subnets 1726 (e.g., similar to Figure 15 one or more application subnets 1526), and a control plane data layer 1728 that may include one or more DB subnets 1730 (e.g., Figure 15 the control plane data layer 1528). The one or more LB subnets 1722 included in the control plane DMZ layer 1720 may be communicatively coupled to the one or more application subnets 1726 included in the control plane application layer 1724 and an Internet gateway 1734 that may be included in the control plane VCN 1716 (e.g., Figure 15 the Internet gateway 1534), and the one or more application subnets 1726 may be communicatively coupled to the one or more DB subnets 1730 included in the control plane data layer 1728, a service gateway 1736 (e.g., Figure 15 the service gateway), and a network address translation (NAT) gateway 1738 (e.g., Figure 15 the NAT gateway 1538). The control plane VCN 1716 may include a service gateway 1736 and a NAT gateway 1738. Figure 15 the service gateway) and a network address translation (NAT) gateway 1738 (e.g., Figure 15 the NAT gateway 1538). The control plane VCN 1716 may include a service gateway 1736 and a NAT gateway 1738.

[0254] The data plane VCN 1718 may include a data plane application layer 1746 (e.g., Figure 15 the data plane application layer 1546 of), a data plane DMZ layer 1748 (e.g., Figure 15 the data plane DMZ layer 1548 of), and a data plane data layer 1750 (e.g., Figure 15 the data plane data layer 1550 of). The data plane DMZ layer 1748 may include one or more trusted application subnets 1760 and one or more untrusted application subnets 1762 that may be communicatively coupled to the data plane application layer 1746 and one or more LB subnets 1722 of an Internet gateway 1734 included in the data plane VCN 1718. The one or more trusted application subnets 1760 may be communicatively coupled to a service gateway 1736 included in the data plane VCN 1718, a NAT gateway 1738 included in the data plane VCN 1718, and one or more DB subnets 1730 included in the data plane data layer 1750. The one or more untrusted application subnets 1762 may be communicatively coupled to a service gateway 1736 included in the data plane VCN 1718 and one or more DB subnets 1730 included in the data plane data layer 1750. The data plane data layer 1750 may include one or more DB subnets 1730 that may be communicatively coupled to a service gateway 1736 included in the data plane VCN 1718.

[0255] The one or more untrusted application subnets 1762 may include one or more primary VNICs 1764(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1766(1)-(N). Each tenant VM 1766(1)-(N) may be communicatively coupled to a corresponding application subnet 1767(1)-(N) that may be included in a corresponding container egress VCN 1768(1)-(N), and the corresponding container egress VCNs 1768(1)-(N) may be included in corresponding customer tenancies 1770(1)-(N). The corresponding secondary VNICs 1772(1)-(N) may facilitate communication between the one or more untrusted application subnets 1762 included in the data plane VCN 1718 and the application subnets included in the container egress VCNs 1768(1)-(N). Each container egress VCN 1768(1)-(N) may include a NAT gateway 1738 that may be communicatively coupled to a public Internet 1754 (e.g., Figure 15 the public Internet 1554 of).

[0256] An Internet gateway 1734 that is included in the control plane VCN 1716 and that is included in the data plane VCN 1718 can be communicatively coupled to a metadata management service 1752 (e.g., Figure 15 the metadata management system 1552), and the metadata management service 1752 can be communicatively coupled to the public Internet 1754. The public Internet 1754 can be communicatively coupled to a NAT gateway 1738 that is included in the control plane VCN 1716 and that is included in the data plane VCN 1718. A service gateway 1736 that is included in the control plane VCN 1716 and that is included in the data plane VCN 1718 can be communicatively coupled to a cloud service 1756.

[0257] In some embodiments, the data plane VCN 1718 can be integrated with a customer lease 1770. In some cases, such as when it may be desirable to support during code execution, this integration may be useful or desirable for customers of an IaaS provider. A customer may provide code to run that may be disruptive, may communicate with other customer resources, or may otherwise cause undesired effects. In response to this, the IaaS provider can determine whether to run the code given to the IaaS provider by the customer.

[0258] In some examples, a customer of an IaaS provider can grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1746. The code that runs the functionality can be executed in VMs 1766(1)-(N), and the code can be not configured to run anywhere else on the data plane VCN 1718. Each VM 1766(1)-(N) can be connected to a customer lease 1770. The corresponding containers 1771(1)-(N) included in the VMs 1766(1)-(N) can be configured to run the code. In this case, there can be dual isolation (e.g., the containers 1771(1)-(N) run the code, where the containers 1771(1)-(N) may be at least included in the VMs 1766(1)-(N) included in one or more untrusted application subnets 1762), which can help prevent incorrect or otherwise undesired code from corrupting the IaaS provider's network or corrupting the networks of different customers. The containers 1771(1)-(N) can be communicatively coupled to the customer lease 1770 and can be configured to transmit or receive data from the customer lease 1770. The containers 1771(1)-(N) can be not configured to transmit or receive data from any other entity in the data plane VCN 1718. After the code execution is complete, the IaaS provider can terminate or otherwise dispose of the containers 1771(1)-(N).

[0259] In some embodiments, the (one or more) trusted application subnets 1760 may run code that may be owned or operated by an IaaS provider. In this embodiment, the (one or more) trusted application subnets 1760 may be communicatively coupled to the (one or more) DB subnets 1730 and configured to perform CRUD operations in the (one or more) DB subnets 1730. The (one or more) untrusted application subnets 1762 may be communicatively coupled to the (one or more) DB subnets 1730, but in this embodiment, the (one or more) untrusted application subnets may be configured to perform read operations in the (one or more) DB subnets 1730. Containers 1771(1)-(N) that may be included in each customer's VM 1766(1)-(N) and may run code from the customer may not be communicatively coupled to the (one or more) DB subnets 1730.

[0260] In other embodiments, the control plane VCN 1716 and the data plane VCN 1718 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1716 and the data plane VCN 1718. However, communication may occur indirectly through at least one method. The LPG 1710 may be established by the IaaS provider, which may facilitate communication between the control plane VCN 1716 and the data plane VCN 1718. In another example, the control plane VCN 1716 or the data plane VCN 1718 may invoke cloud services 1756 via the service gateway 1736. For example, an invocation of cloud services 1756 from the control plane VCN 1716 may include a request for a service that may communicate with the data plane VCN 1718.

[0261] Figure 18 is a block diagram 1800 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1802 (e.g., Figure 15 the service operator 1502) may be communicatively coupled to a secure host lease 1804 (e.g., Figure 15 the secure host lease 1504), which may include a virtual cloud network (VCN) 1806 (e.g., Figure 15 the VCN 1506) and a secure host subnet 1808 (e.g., Figure 15 the secure host subnet 1508). The VCN 1806 may include an LPG 1810 (e.g., Figure 15 the LPG 1510), which may be via an SSH VCN 1812 included (e.g., Figure 15LPG 1810 in SSH VCN 1512 of the present invention is communicatively coupled to SSH VCN 1812. SSH VCN 1812 may include SSH subnet 1814 (e.g., Figure 15 SSH subnet 1514 of the control plane VCN 1816), and SSH VCN 1812 can be communicatively coupled to control plane VCN 1816 via LPG 1810 contained in control plane VCN 1816 (e.g., Figure 15 1516) and is coupled to the data plane VCN 1818 via the LPG 1810 included in the data plane VCN 1818 (e.g., Figure 15 The control plane VCN 1816 and the data plane VCN 1818 may be included in a service lease 1819 (e.g., Figure 15 Service lease 1519).

[0262] The control plane VCN 1816 may include (one or more) LB subnets 1822 (e.g., Figure 15 LB subnet(s) 1522) of the control plane DMZ layer 1820 (e.g., Figure 15 The control plane DMZ layer 1520 may include (one or more) application subnets 1826 (e.g., Figure 15 (one or more) application subnets 1526) of the control plane application layer 1824 (e.g., Figure 15 The control plane application layer 1524 of FIG. 1524 may include (one or more) DB subnets 1830 (e.g., Figure 17 (one or more) DB subnets 1730) of the control plane data layer 1828 (e.g., Figure 15 1828). The LB subnet(s) 1822 contained in the control plane DMZ layer 1820 may be communicatively coupled to the application subnet(s) 1826 contained in the control plane application layer 1824 and the internet gateway 1834 (e.g., Figure 15 1534), and the application subnet(s) 1826 may be communicatively coupled to the DB subnet(s) 1830 and the service gateway 1836 (e.g., Figure 15 ) and a network address translation (NAT) gateway 1838 (e.g., Figure 15 The control plane VCN 1816 may include a service gateway 1836 and a NAT gateway 1838.

[0263] The data plane VCN 1818 may include a data plane application layer 1846 (e.g., Figure 15 the data plane application layer 1546 of Figure 15 ), a data plane DMZ layer 1848 (e.g., Figure 15 the data plane DMZ layer 1548 of Figure 17 ), and a data plane data layer 1850 (e.g., Figure 17 the data plane data layer 1550 of

[0264] ). The data plane DMZ layer 1848 may include one or more trusted application subnets 1860 (e.g., Figure 15 one or more trusted application subnets 1760 of

[0265] Figure 17 ) that may be communicatively coupled to the data plane application layer 1846 and one or more untrusted application subnets 1862 (e.g., Figure 17 one or more untrusted application subnets 1762 of

[0264] ) and one or more LB subnets 1822 of an Internet gateway 1834 included in the data plane VCN 1818. The one or more trusted application subnets 1860 may be communicatively coupled to a service gateway 1836 included in the data plane VCN 1818, a NAT gateway 1838 included in the data plane VCN 1818, and one or more DB subnets 1830 included in the data plane data layer 1850. The one or more untrusted application subnets 1862 may be communicatively coupled to the service gateway 1836 included in the data plane VCN 1818 and one or more DB subnets 1830 included in the data plane data layer 1850. The data plane data layer 1850 may include one or more DB subnets 1830 that may be communicatively coupled to the service gateway 1836 included in the data plane VCN 1818. (One or more) untrusted application subnets 1862 may include one or more primary VNICs 1864(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1866(1)-(N) residing within the one or more untrusted application subnets 1862. Each tenant VM 1866(1)-(N) may run code in a respective container 1867(1)-(N) and be communicatively coupled to an application subnet 1826 in the data plane application layer 1846 that may be included in a container egress VCN 1868. Respective secondary VNICs 1872(1)-(N) may facilitate communication between the one or more untrusted application subnets 1862 included in the data plane VCN 1818 and the application subnet included in the container egress VCN 1868. The container egress VCN may include a NAT gateway 1838 that may be communicatively coupled to a public Internet 1854 (e.g., Figure 15 the public Internet 1554 of

[0265] The Internet gateway 1834 included in the control plane VCN 1816 and included in the data plane VCN 1818 can be communicatively coupled to the metadata management service 1852 (e.g., Figure 15 1852), which can be communicatively coupled to the public Internet 1854. The public Internet 1854 can be communicatively coupled to a NAT gateway 1838 contained in the control plane VCN 1816 and contained in the data plane VCN 1818. The service gateway 1836 contained in the control plane VCN 1816 and contained in the data plane VCN 1818 can be communicatively coupled to the cloud service 1856.

[0266] In some examples, Figure 18 The architecture model shown in block diagram 1800 can be considered as Figure 17 1700 , and may be desired by customers of the IaaS provider if the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). The customer may have real-time access to the corresponding container 1867(1)-(N) contained in each customer's VM 1866(1)-(N). The container 1867(1)-(N) may be configured to make calls to the corresponding secondary VNIC 1872(1)-(N) contained in the (one or more) application subnets 1826 of the data plane application layer 1846, which may be contained in the container egress VCN 1868. The secondary VNIC 1872(1)-(N) may transmit the call to the NAT gateway 1838, which may transmit the call to the public Internet 1854. In this example, containers 1867(1)-(N), which may be accessed by customers in real time, may be isolated from control plane VCN 1816 and may be isolated from other entities contained in data plane VCN 1818. Containers 1867(1)-(N) may also be isolated from resources from other customers.

[0267] In other examples, a customer can use containers 1867(1)-(N) to invoke cloud service 1856. In this example, the customer can run code in containers 1867(1)-(N) that requests services from cloud service 1856. Containers 1867(1)-(N) can transmit the request to secondary VNICs 1872(1)-(N), which can transmit the request to a NAT gateway that can transmit the request to public Internet 1854. Public Internet 1854 can transmit the request via Internet gateway 1834 to (one or more) LB subnets 1822 included in control plane VCN 1816. In response to determining that the request is valid, (one or more) LB subnets can transmit the request to (one or more) application subnets 1826, which can transmit the request to cloud service 1856 via service gateway 1836.

[0268] It should be appreciated that the IaaS architectures 1500, 1600, 1700, 1800 depicted in the figures may have other components than those depicted. Additionally, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0269] In certain embodiments, the IaaS systems described herein may include application suites, middleware, and database service offerings that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) offered by the present assignee.

[0270] Figure 19 An example computer system 1900 in which various embodiments may be implemented is illustrated. System 1900 may be used to implement any of the computer systems described above. As shown, computer system 1900 includes a processing unit 1904 that communicates with a plurality of peripheral subsystems via a bus subsystem 1902. These peripheral subsystems may include a processing acceleration unit 1906, an I / O subsystem 1908, a storage subsystem 1918, and a communication subsystem 1924. Storage subsystem 1918 includes a tangible computer-readable storage medium 1922 and system memory 1910.

[0271] The bus subsystem 1902 provides a mechanism for enabling the various components and subsystems of the computer system 1900 to communicate with each other as intended. Although the bus subsystem 1902 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1902 can be any of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which can be implemented as Mezzanine buses manufactured to the IEEE P1386.1 standard.

[0272] The processing unit 1904, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1900. One or more processors may be included in the processing unit 1904. These processors may include single-core or multi-core processors. In certain embodiments, the processing unit 1904 may be implemented as one or more separate processing units 1932 and / or 1934, where each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1904 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0273] In various embodiments, the processing unit 1904 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in the (one or more) processors 1904 and / or the storage subsystem 1918. Through appropriate programming, the (one or more) processors 1904 can provide the various functions described above. The computer system 1900 may additionally include a processing acceleration unit 1906, which may include a digital signal processor (DSP), a dedicated processor, and the like.

[0274] The I / O subsystem 1908 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device having a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as Microsoft A motion sensor that enables a user to control and interact with an input device such as a Microsoft 360 game controller through a natural user interface using gestures and voice commands. The user interface input device may also include an eye gesture recognition device, such as one that detects eye activity from the user (e.g., a "blink" when taking a photo and / or making a menu selection) and converts the eye gesture into an input to the input device (e.g., a Google blink detector). Additionally, the user interface input device may include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., a navigator) through voice commands. The user interface input device may also include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, game pad, and graphics tablet, as well as audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye tracking devices. Additionally, the user interface input device may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. The user interface input device may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0275]

[0276] The user interface output device may include a display subsystem, indicator lights, or a non-visual display such as an audio output device, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1900 to a user or other computer. For example, the user interface output device may include, but is not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

[0277] The computer system 1900 may include a storage subsystem 1918 that provides a tangible non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., that provide the above functionality when executed by one or more cores or processors of the processing unit 1904. The storage subsystem 1918 may also provide a repository for storing data used in accordance with this disclosure.

[0278] As Figure 19 depicted in the example of [[ID=]], the storage subsystem 1918 may include various components, including a system memory 1910, a computer-readable storage medium 1922, and a computer-readable storage medium reader 1920. The system memory 1910 may store program instructions that can be loaded and executed by the processing unit 1904. The system memory 1910 may also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs may be loaded into the system memory 1910, including but not limited to client applications, web browsers, middle-tier applications, relational database management systems (RDBMSs), virtual machines, containers, etc.

[0279] The system memory 1910 may also store an operating system 1916. Examples of the operating system 1916 may include various versions of Microsoft Apple and / or Linux operating systems, various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems (such as iOS, Phone, OS, OS, and OS operating systems). In certain embodiments where the computer system 1900 executes one or more virtual machines, the virtual machines along with their guest operating systems (GOSs) may be loaded into the system memory 1910 and executed by one or more processors or cores of the processing unit 1904.

[0280] The system memory 1910 may be configured differently depending on the type of the computer system 1900. For example, the system memory 1910 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, the system memory 1910 may include a basic input / output system (BIOS) that contains basic routines that help transfer information between elements within the computer system 1900, such as during startup.

[0281] The computer-readable storage medium 1922 can represent a remote, local, fixed, and / or removable storage device and a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1904 of the computer system 1900) for use by the computer system 1900.

[0282] The computer-readable storage medium 1922 can include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for the storage and / or transmission of information. This can include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape cartridges, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0283] For example, the computer-readable storage medium 1922 can include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a disk drive that reads from or writes to a removable non-volatile disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk (such as a CD ROM, DVD, and disk or other optical media). The computer-readable storage medium 1922 can include, but is not limited to, drives, flash cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital audio tapes, and so on. The computer-readable storage medium 1922 can also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), SSDs based on volatile memory (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1900.

[0284] Machine-readable instructions executable by one or more processors or cores of processing unit 1904 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, which include volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0285] Communication subsystem 1924 provides an interface to other computer systems and networks. Communication subsystem 1924 serves as an interface for receiving data from other systems and sending data from computer system 1900 to other systems. For example, communication subsystem 1924 may enable computer system 1900 to connect to one or more devices via the Internet. In some embodiments, communication subsystem 1924 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution) advanced data network technologies, WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, as an addition or alternative to the wireless interface, communication subsystem 1924 may provide a wired network connection (e.g., Ethernet).

[0286] In some embodiments, communication subsystem 1924 may also receive input communications in the form of structured and / or unstructured data feeds 1926, event streams 1928, event updates 1930, etc. on behalf of one or more users who may use computer system 1900.

[0287] By way of example, communication subsystem 1924 may be configured to receive data feeds 1926 from users of social networks and / or other communication services in real time, such as feeds, updates, web feeds such as rich site summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0288] In addition, the communication subsystem 1924 can also be configured to receive data in the form of a continuous data stream, which can include an event stream 1928 and / or event updates 1930 of real-time events that can be continuous or unbounded in nature and have no explicit termination. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and so on.

[0289] The communication subsystem 1924 can also be configured to output structured and / or unstructured data feeds 1926, event streams 1928, event updates 1930, etc., to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1900.

[0290] The computer system 1900 can be one of various types, including handheld portable devices (e.g., cellular phones, computing tablets, PDAs), wearable devices (e.g., Glass head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing system.

[0291] Due to the ever-changing nature of computers and networks, the description of the computer system 1900 depicted in the figure is merely to serve as a specific example. Many other configurations with more or fewer components than the system depicted in the figure are possible. For example, custom hardware can also be used and / or specific elements can be implemented in hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices such as network input / output devices can also be employed. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will recognize other ways and / or methods of implementing various embodiments.

[0292] Embodiments can be implemented by using a computer program product, including computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure.

[0293] Although specific embodiments have been described, various modifications, changes, alternative constructs, and equivalent forms are also included within the scope of this disclosure. Embodiments are not limited to operating within certain specific data processing environments but can operate freely within multiple data processing environments. Additionally, although embodiments have been described using a specific series of transactions and steps, those skilled in the art should be clear that the scope of this disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above embodiments can be used alone or in combination.

[0294] In addition, although embodiments have been described using a specific combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the present disclosure. Embodiments can be implemented using only hardware, or only software, or using a combination thereof. The various processes described herein can be implemented in the same processor or in different processors in any combination. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished by, for example, designing electronic circuitry to perform the operations, programming a programmable electronic circuit such as a microprocessor to perform the operations, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes can use different techniques, or the same pair of processes can use different techniques at different times.

[0295] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is obvious that additions, subtractions, deletions, and other modifications and changes can be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalent forms are within the scope of the following claims.

[0296] In the context of describing the disclosed embodiments (especially in the context of the following claims), the terms “a,” “an,” “the,” and similar references are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” shall be construed to mean partly or wholly incorporated in, attached to, or joined together, even if there is something in between. Unless otherwise indicated herein, the recitation of a range of values herein is merely intended to be a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated into the specification as if it were individually recited herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein is merely intended to better illuminate the embodiments and does not pose a limitation on the scope of the present disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the present disclosure.

[0297] Disjunctive language, such as the phrase "at least one of X, Y, or Z," unless otherwise expressly specified, is intended to be understood in the context of generally representing items, terms, etc., and can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language generally is not intended to and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z each.

[0298] This disclosure describes preferred embodiments of the present disclosure, including known best modes for carrying out the present disclosure. Variations of these preferred embodiments may become apparent to those of ordinary skill in the art after reading the foregoing description. Those of ordinary skill in the art should be able to appropriately adopt such variations and practice the present disclosure in a manner different from that specifically described herein. Accordingly, the present disclosure includes all modifications and equivalent forms of the subject matter recited in the appended claims as permitted by applicable law. Moreover, unless otherwise indicated herein, the present disclosure includes any combination of the above elements in all possible variations.

[0299] Example embodiments of the present disclosure may be described in accordance with the following clauses.

[0300] Clause 1. A method is disclosed. The method may include obtaining, by a computer system, configuration data including a plurality of response levels that specify the applicability of a respective set of reduction actions to a plurality of hosts. In some embodiments, a first response level of the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts. The method may include obtaining, by the computer system, a current value of the total power consumption of the plurality of hosts. The method may include obtaining, by the computer system, a current value of the total power threshold of the plurality of hosts. The method may include selecting a first response level from the plurality of response levels based at least on a difference between the current value of the total power consumption and the current value of the total power threshold. The method may include causing the first set of reduction actions to be applied to at least one of the plurality of hosts according to the selected first response level.

[0301] Clause 2. The method of Clause 1, further comprising performing, by the computer system, the first set of reduction actions on the plurality of hosts according to the selected first response level.

[0302] Clause 3. The method of Clause 1 or 2, further comprising 1) determining whether the current value of the total power consumption exceeds the current value of the total power threshold, and 2) in response to determining that the current value of the total power consumption exceeds the current value of the total power threshold, making a selection from the plurality of response levels, wherein the selected response level is applicable to a subset of the plurality of hosts.

[0303] Clause 4. The method of any one of Clauses 1-3, wherein the first response level designates the applicability of a first set of reduction actions to the plurality of hosts based at least on host attributes.

[0304] Clause 5. The method of Clause 4, wherein the host attributes include a priority level.

[0305] Clause 6. The method of any one of Clauses 1-5, wherein each successive response level among the plurality of response levels designates multiple successive sets of strict reduction actions to be applied to the plurality of hosts.

[0306] Clause 7. The method of Clause 6, wherein a first power ceiling value applied to a first host by a first reduction action according to a first response level is lower than a second power ceiling value applied to the first host by a different reduction action according to a second response level.

[0307] Clause 8. The method of Clause 6, wherein: 1) the first response level designates applying a first reduction action to a first subset of the plurality of hosts associated with a priority level lower than a first level, 2) the second response level designates applying a second reduction action to a second subset of the plurality of hosts associated with a priority level lower than a second level, 3) the second level is higher than the first level, and 4) the second subset of the plurality of hosts does not include the first subset of the plurality of hosts.

[0308] Clause 9. The method of Clause 6, wherein the selected first response level is the response level among the plurality of response levels associated with the least strict set of reduction actions that achieves a new value of total power consumption that is less than or equal to the current value of the total power threshold.

[0309] Clause 10. The method of Clause 6, wherein selecting the first response level from the plurality of response levels based at least on the difference between the current value of the total power consumption and the current value of the total power threshold includes: 1) determining that a first reduction in total power consumption resulting from the first response level is greater than the difference, 2) determining that a second reduction in total power consumption resulting from the second response level is less than the difference, and 3) selecting the first response level.

[0310] Clause 11. The method of any one of Clauses 1-10, wherein due to a decrease in the total power threshold, the current value of the total power consumption has breached the current value of the total power threshold, and wherein the current value of the total power consumption has exceeded the current value of the total power threshold for a period of time.

[0311] Clause 12. The method of Clause 11, wherein the decrease in the total power threshold is caused by at least one of the following: 1) degradation or failure of a temperature control system of a physical environment including the plurality of hosts, or 2) government reduction of power supply, or 3) an increase in external temperature.

[0312] Clause 13. The method of any one of Clauses 1 - 12, wherein: 1) during a first time period, it is determined that a first value of a total power threshold exceeds a current value of total power consumption; and 2) during a current time period after the first time period, it is determined that the current value of total power consumption exceeds the current value of the total power threshold. In some embodiments, the current value of the total power threshold during the current time period is less than the first value of the total power threshold during the first time period.

[0313] Clause 14. The method of any one of Clauses 1 - 13, wherein the first reduction action includes at least one of the following: 1) enforcing a power ceiling on the host; 2) migrating a workload from the host; or 3) shutting down or pausing the host.

[0314] Clause 15. The method of any one of Clauses 1 - 14, further comprising:

[0315] obtaining a new value of the total power threshold by a computer system; and

[0316] performing a recovery from a first response level based at least on the new value of the total power threshold.

[0317] Clause 16. The method of Clause 15, wherein performing a recovery from the first response level includes at least one of the following: 1) stopping power capping on the host, 2) migrating the workload back to the host, or 3) restarting a host that was previously shut down or paused.

[0318] Clause 17. The method of any one of Clauses 1 - 16, wherein performing the first response level among the plurality of response levels is in response to receiving user input at a user interface, wherein the selected first response level is associated with a reduction amount that exceeds a difference between the current value of total power consumption and the current value of the total power threshold.

[0319] Clause 18. A method is disclosed. The method may include obtaining, by a computer system, configuration data including a plurality of response levels that specify the applicability of a corresponding set of reduction actions to a plurality of hosts. In some embodiments, a first response level of the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts. The method may include obtaining, by the computer system, a predicted value of the total power consumption of the plurality of hosts over a future time period. The method may include obtaining, by the computer system, a predicted value of the total power threshold of the plurality of hosts over a future time period. The method may include selecting the first response level from the plurality of response levels based at least on a predicted difference between the predicted value of the total power consumption and the predicted value of the total power threshold. The method may include identifying, according to the selected first response level, one or more workloads that are (a) currently being executed on the plurality of hosts and (b) will be affected by applying the first set of reduction actions to the plurality of hosts. The method may include migrating, prior to the future time period, the affected workloads from the plurality of hosts to one or more other hosts.

[0320] Clause 19. The method of Clause 18 may further include determining the predicted value of the total power threshold based on at least one of: 1) the health of a temperature control system of a physical environment including the plurality of hosts, or 2) an announced or foreseen government power supply reduction, or 3) a predicted increase in external temperature.

[0321] Clause 20. The method of Clause 18 or 19 may further include determining the predicted value of the total power consumption based on a historical pattern of the total power consumption.

[0322] Clause 21. The method of any one of Clauses 18 - 20, wherein identifying, according to the selected first response level, one or more workloads that are (a) currently being executed on the plurality of hosts and (b) will be affected by applying the first set of reduction actions to the plurality of hosts includes: 1) determining that a particular reduction action will be applied to a first host according to the selected second first level, 2) determining that a first workload is currently being executed on the first host, 3) determining that the first workload will be affected by applying the first set of reduction actions to the plurality of hosts according to the selected first response level.

[0323] Clause 22. A system is disclosed. The system may include a memory configured to store instructions and one or more processors configured to execute the instructions to perform the method of any one of Clauses 1 - 21.

[0324] Clause 23 discloses a non-transitory computer-readable medium. The non-transitory computer-readable medium may have instructions stored thereon that, when executed by a processor, cause the processor to perform the method of any one of Clauses 1-21.

[0325] All references cited herein, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in full herein.

[0326] In the foregoing specification, aspects of the disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects disclosed above may be used singly or in combination. Additionally, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method, comprising: obtaining, by a computer system, configuration data including a plurality of response levels that specify the applicability of a corresponding set of reduction actions to a plurality of hosts; wherein a first response level among the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts; obtaining, by the computer system, a current value of the total power consumption of the plurality of hosts; obtaining, by the computer system, a current value of the total power threshold of the plurality of hosts; selecting the first response level from the plurality of response levels based at least on a difference between the current value of the total power consumption and the current value of the total power threshold; and causing, according to the selected first response level, the first set of reduction actions to be applied to at least one of the plurality of hosts.

2. The method according to claim 1, further comprising: performing, by the computer system, the first set of reduction actions on the plurality of hosts according to the selected first response level.

3. The method according to claim 1 or 2, further comprising: determining whether the current value of the total power consumption exceeds the current value of the total power threshold; and in response to determining that the current value of the total power consumption exceeds the current value of the total power threshold, making a selection from the plurality of response levels, wherein the selected response level is applicable to a subset of the plurality of hosts.

4. The method according to any one of claims 1 to 3, wherein the first response level specifies the applicability of the first set of reduction actions to the plurality of hosts based at least on host attributes.

5. The method according to claim 4, wherein the host attributes include a priority level.

6. The method according to any one of claims 1 to 5, wherein each successive response level among the plurality of response levels specifies a successive set of increasingly strict reduction actions to be applied to the plurality of hosts.

7. The method according to claim 6, wherein a first power ceiling value applied to a first host by a first reduction action according to the first response level is lower than a second power ceiling value applied to the first host by a different reduction action according to a second response level.

8. The method according to claim 6, wherein: the first response level specifies applying a first reduction action to a first subset of the plurality of hosts associated with a priority level lower than a first level; the second response level specifies applying a second reduction action to a second subset of the plurality of hosts associated with a priority level lower than a second level; the second level is higher than the first level, and the second subset of the plurality of hosts does not include the first subset of the plurality of hosts.

9. The method according to claim 6, wherein the selected first response level is the response level among the plurality of response levels associated with the least strict set of reduction actions that achieves a new value of the total power consumption that is less than or equal to the current value of the total power threshold.

10. The method according to claim 6, wherein selecting the first response level from the plurality of response levels based at least on a difference between the current value of the total power consumption and the current value of the total power threshold comprises: determining that a first reduction in the total power consumption caused by the first response level is greater than the difference; Determine that a second reduction in the total power consumption caused by the second response level is less than the difference; and Select the first response level.

11. The method according to any one of claims 1 to 10, wherein due to a decrease in the total power threshold, the current value of the total power consumption has breached the current value of the total power threshold, and wherein the current value of the total power consumption has exceeded the current value of the total power threshold for a period of time.

12. The method according to claim 11, wherein the decrease in the total power threshold is caused by at least one of the following: Degradation or failure of a temperature control system of a physical environment including the plurality of hosts; or Government reduction of power supply; or Rise in external temperature.

13. The method according to any one of claims 1 to 12, further comprising: During a first time period, determine that a first value of the total power threshold exceeds the current value of the total power consumption; and During a current time period after the first time period, determine that the current value of the total power consumption exceeds the current value of the total power threshold, and the current value of the total power threshold during the current time period is less than the first value of the total power threshold during the first time period.

14. The method according to any one of claims 1-13, wherein the first reduction action includes at least one of the following: 1) Enforce a power ceiling on the host; 2) Migrate workloads from the host; or 3) Shut down or suspend the host.

15. The method according to any one of claims 1 to 14, further comprising: Obtain a new value of the total power threshold by the computer system; and Perform a recovery from the first response level based at least on the new value of the total power threshold.

16. The method according to claim 15, wherein performing a recovery from the first response level includes at least one of the following: 1) Stop power capping on the host, 2) Migrate the workload back to the host, or 3) Restart a previously shut down or suspended host.

17. The method according to any one of claims 1 to 16, wherein the application that performs the first set of reduction actions is in response to receiving user input at a user interface, and wherein the selected first response level is associated with a reduction amount that exceeds the difference between the current value of the total power consumption and the current value of the total power threshold.

18. A method, comprising: Obtain configuration data by a computer system, the configuration data including the plurality of response levels, the plurality of response levels specifying the applicability of a corresponding set of reduction actions to a plurality of hosts; wherein a first response level among the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts; Obtain a predicted value of the total power consumption of the plurality of hosts within a future time period by the computer system; Obtain a predicted value of the total power threshold of the plurality of hosts within a future time period by the computer system; Select a first response level from the plurality of response levels based at least on a predicted difference between the predicted value of the total power consumption and the predicted value of the total power threshold; Identify, according to the selected first response level, one or more workloads that (a) are currently being executed on the plurality of hosts and (b) will be affected by applying the first set of reduction actions to the plurality of hosts; and Prior to a future time period, migrate the affected workloads from the plurality of hosts to one or more other hosts in advance.

19. The method according to claim 18, further comprising: determine a predicted value of the total power threshold based on at least one of the following: the health status of the temperature control system of the physical environment including the plurality of hosts; or announced or foreseen government power supply reduction; or predicted external temperature increase.

20. The method according to claim 18 or 19, further comprising: determine a predicted value of the total power consumption based on the historical pattern of the total power consumption.

21. The method according to any one of claims 18-20, wherein one or more workloads that (a) are currently being executed on the plurality of hosts and (b) will be affected by applying a first set of reduction actions to the plurality of hosts are identified according to the selected first response level comprising: determine that a specific reduction action will be applied to the first host according to the selected second first level; determine that the first workload is currently being executed on the first host; and determine that the first workload will be affected by applying a first set of reduction actions to the plurality of hosts according to the selected first response level.

22. A system, comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform the method according to any one of claims 1 to 21.

23. A non-transitory computer-readable medium having instructions stored thereon, which when executed by a processor cause the processor to perform the method according to any one of claims 1 to 21.

Images