Techniques for orchestrating load shedding

Abstract

The disclosed technology relates to orchestrating power consumption reduction for multiple hosts. A current value of a total power threshold for the plurality of hosts may be identified. During a first time period, a current value of a total power threshold may be used to manage total power consumption of the plurality of hosts. A trigger event may be detected indicating that a total power threshold needs to be modified. A new value of the total power threshold may be determined based on the trigger event. During the second time period, the total power consumption of the plurality of hosts may be managed using a new value of the total power threshold.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 423,762, filed on November 8, 2022, entitled "Orchestrated DC - Scale Load Shedding", U.S. Provisional Application No. 63 / 439,576, filed on January 18, 2023, entitled "Orchestrated DC - Scale Load Shedding", and U.S. Non - Provisional Application No. 18 / 339,114, filed on June 21, 2023, entitled "Techniques for Orchestrated Load Shedding", the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to techniques for orchestrating the reduction of power consumption in a data center. The disclosed systems, methods, devices, and services implement a dynamic orchestration method that uses a total power threshold to limit power consumption. A triggering event may cause the total power threshold to be modified up or down over time, and power consumption can similarly be reduced or increased based on the currently adopted total power threshold. Background Art

[0004] Data centers are configured with a power infrastructure that provides various security features based on a hierarchical power distribution. Power is provided by a local power utility and is distributed to various components of the data center according to this power distribution hierarchy, including power distribution units (PDUs) (e.g., transformers, switchboards, bus ducts, rack PDUs, etc.) and power - consuming devices (e.g., servers, network devices, etc.). This ensures that the power consumed by all downstream devices complies with the power limits of each upstream device. When demand reaches a peak and / or when a component fails, the data center may no longer be able to effectively handle the demand, which may lead to a widespread power failure, causing at least severe damage to downstream devices, resulting in a reduction in processing capacity within the data center and / or a data center power failure, which in turn leads to a poor user experience. In the worst - case scenario, a power failure in one data center may trigger a cascading power failure of other devices within the same data center or other data centers as workloads are redistributed in an attempt to recover from the initial power failure. Additionally, in some embodiments, external factors (e.g., government regulations) may require reducing the power consumption of a particular data center within a specific time.

[0005] Conventional methods of managing load shedding include manual tasks where an operator shuts down host racks or individual devices one by one to reduce power consumption. Alternatively, conventional techniques include pressing an emergency shutdown switch or button that is configured to shut off the power to an entire data center or a majority of the data center. The operator determining which components to shut down typically has no knowledge of the workloads or customers affected by these actions or the extent of the impact of taking these actions. Conventional methods like those mentioned here result in suboptimal approaches to affected customers and workloads. More sophisticated methods may have a lesser impact on customers and / or devices / workloads while still providing the desired power reduction for the situation. Accordingly, it is desirable to improve power management techniques, particularly with respect to load shedding, to reduce power consumption to a sufficient impact and amount and avoid the deficiencies of conventional methods. SUMMARY OF THE INVENTION

[0006] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is apparent that the various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0007] Some embodiments may include a method. The method may include a computer system identifying a current value of a total power threshold for a plurality of hosts. The method may include using the current value of the total power threshold to manage the total power consumption of the plurality of hosts during a first time period. The method may include a computer system detecting a trigger event indicating a need to modify the total power threshold. The method may include a computer system determining a new value of the total power threshold based on the trigger event. The method may include using the new value of the total power threshold to manage the total power consumption of the plurality of hosts during a second time period.

[0008] In some embodiments, detecting the trigger event includes at least one of the following: 1) a computer system monitoring a temperature control system associated with the plurality of hosts, or 2) a computer system determining that the temperature control system is operating at a reduced capacity.

[0009] In some embodiments, determining the new value of the total power threshold includes at least one of the following: 1) a computer system identifying a table specifying corresponding power consumption values applicable to each of a plurality of temperature control systems associated with the plurality of hosts, the plurality of temperature control systems including the temperature control system operating at a reduced capacity, 2) a computer system identifying from the table the power consumption value corresponding to the temperature control system operating at a reduced capacity. In some embodiments, the new value of the total power threshold is identified at least in part based on the current power consumption value being less than the power consumption amount associated with the temperature control system operating at a reduced capacity.

[0010] In some embodiments, determining a new value for the total power threshold includes at least one of the following: 1) obtaining, by a computer system, a machine learning model that has been previously trained to identify, from input data including an identifier of a corresponding temperature control system, a corresponding power consumption value applicable to the corresponding temperature control system, 2) providing, by the computer system, a corresponding identifier associated with a temperature control system operating at a reduced capacity as an input to the machine learning model, or 3) receiving, by the computer system, a power consumption value applicable to the temperature control system as an output from the machine learning model, wherein the new total power consumption is determined based at least in part on the power consumption value applicable to the temperature control system.

[0011] In some embodiments, the machine learning model is trained at least in part based on a training data set including historical power consumption values and historical metadata associated with one or more temperature control systems.

[0012] In some embodiments, detecting a trigger event includes at least one of the following: 1) receiving, by a computer system, from a utility grid, a curtailment request from a power authority that indicates the extent to which power consumption in the computer system is to be curtailed, 2) identifying, by the computer system, a degradation or failure of a temperature control system of a physical environment including a plurality of hosts, or 3) identifying, by the computer system, an increase in an external temperature.

[0013] In some embodiments, detecting a trigger event includes a computer system predicting a failure of a temperature control system associated with a plurality of hosts based at least in part on an external temperature associated with a physical environment in which the plurality of hosts are located.

[0014] In some embodiments, predicting a failure of a temperature control system associated with a plurality of hosts is based at least in part on identifying that the external temperature exceeds a temperature threshold associated with a physical environment in which the plurality of hosts are located.

[0015] In some embodiments, predicting a failure of a temperature control system associated with a plurality of hosts is also based at least in part on identifying that a redundant temperature control system corresponding to the temperature control system is associated with a second failure.

[0016] In some embodiments, using the new value of the total power threshold to manage the total power consumption of a plurality of hosts includes at least one of the following: 1) enforcing a power ceiling on the hosts; 2) migrating workloads from the hosts; or 3) shutting down or suspending the hosts.

[0017] In some embodiments, a computer system is disclosed. The computer system may include a memory configured to store instructions and one or more processors configured to execute the instructions to perform a method. In some embodiments, the method may include identifying a current value of a total power threshold for a plurality of hosts. The method may include using the current value of the total power threshold to manage the total power consumption of the plurality of hosts during a first time period. The method may include detecting a trigger event indicating a need to modify the total power threshold. The method may include determining a new value of the total power threshold based on the trigger event. The method may include using the new value of the total power threshold to manage the total power consumption of the plurality of hosts during a second time period.

[0018] In some embodiments, executing the instructions further causes the one or more processors to identify a plurality of response levels that specify the applicability of respective sets of reduction actions to the plurality of hosts. In some embodiments, a first response level of the plurality of response levels may specify the applicability of a first set of reduction actions to the plurality of hosts such that a first reduction action of the first set of reduction actions is applicable to a first host of the plurality of hosts and a second reduction action of the first set of reduction actions is applicable to a second host of the plurality of hosts. In some embodiments, executing the instructions further causes the one or more processors to select the first response level of the plurality of response levels at least in part based on a first power reduction estimate for applying the first set of reduction actions to the first host of the plurality of hosts and a second power reduction estimate for applying a second set of reduction actions to the second host of the plurality of hosts.

[0019] In some embodiments, executing the instructions further causes the one or more processors to determine that a total power reduction estimated for the first response level is greater than a difference between a current total power consumption and the new value of the total power threshold. In some embodiments, the total power reduction estimated for a response level is determined at least in part based on a first power reduction estimate for applying a first set of reduction actions to a first host of the plurality of hosts and a second power reduction estimate for applying a second set of reduction actions to a second host of the plurality of hosts.

[0020] In some embodiments, using the new value of the total power threshold to manage the total power consumption of the plurality of hosts during the second time period includes at least one of the following: 1) determining a difference between a current value of the total power consumption of the plurality of hosts during the second time period and a current value of the total power threshold of the plurality of hosts during the second time period, 2) determining that a first value of power reduction resulting from the first response level is greater than the difference, or 3) selecting the first response level at least based on determining that the first value of power reduction resulting from the first response level is greater than the difference.

[0021] In some embodiments, the detected trigger event is associated with an external temperature outside the physical environment in which multiple hosts are located. In some embodiments, the execution of the instructions also causes one or more processors to determine, from the environmental data, a difference between the current ambient temperature of the physical environment in which the multiple hosts are located and the external temperature outside the physical environment. In some embodiments, the environment includes an association between the difference and a corresponding power consumption value. In some embodiments, a new value of the total power threshold is determined at least in part based on the corresponding power consumption value associated with the difference between the current ambient temperature and the external temperature.

[0022] In some embodiments, the execution of the instructions also causes one or more processors to perform at least one of the following: 1) obtain a machine learning model that has been previously trained to identify, from input data including a corresponding external temperature outside the physical environment in which the multiple hosts are located, an output indicating an amount by which the total power consumption of the data center should be reduced, 2) provide the current external temperature outside the physical environment in which the multiple hosts are located as input to the machine learning model, or 3) receive, by the computer system, the reduction amount from the machine learning model, wherein the new value of the total power threshold is determined at least in part based on the reduction amount received from the machine learning model.

[0023] In some embodiments, the machine learning model has been previously trained using a supervised learning algorithm and a training dataset that includes historical external temperatures, historical ambient temperatures, historical total power thresholds, and historical current power consumption values corresponding to the multiple hosts.

[0024] In some embodiments, detecting the trigger event is associated with detecting a failure of a temperature control system associated with the multiple hosts. In some embodiments, the execution of the instructions also causes one or more processors to perform at least one of the following: 1) detect that the temperature control system is operable, 2) determine an updated value of the total power threshold at least in part based on determining that the temperature control system is operable, or 3) manage the total power consumption of the multiple hosts using the updated value of the total power threshold during a third time period.

[0025] In some embodiments, the execution of the instructions also causes one or more processors to perform one or more recovery actions at least in part based on the updated value of the total power threshold.

[0026] In some embodiments, one or more recovery actions include at least one of the following: 1) stop power capping of a host among the multiple hosts, 2) migrate a workload back to the host, or 3) restart a host that was previously shut down or suspended.

[0027] Systems, devices, and computer media are disclosed, where each of the systems, devices, and computer media can include one or more memories, where instructions corresponding to the methods disclosed herein can be stored. These instructions can be executed by one or more processors of the disclosed systems and devices to perform the methods disclosed herein. One or more computer programs can be configured to perform specific operations or actions corresponding to the described methods by including instructions that, when executed by a data processing device, cause the device to perform these actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Depicts an example physical environment (e.g., a data center or a portion thereof) including various components according to at least one embodiment.

[0029] Figure 2 Illustrates a simplified diagram of an exemplary power distribution infrastructure including various components of a data center according to at least one embodiment.

[0030] Figure 3 Illustrates an example architecture of an exemplary power orchestration system configured to orchestrate power consumption reduction of various resources applicable to a physical environment (e.g., a data center, a room of a data center, etc.) according to at least one embodiment.

[0031] Figure 4 Illustrates an example architecture of a power management service for detecting consumption overages and constraining power consumption overages according to at least one embodiment.

[0032] Figure 5 Illustrates according to at least one embodiment the Figure 2 corresponding example power distribution hierarchy of the arrangement of components.

[0033] Figure 6 Is a flowchart illustrating an example method for managing excessive power consumption according to at least one embodiment.

[0034] Figure 7 Illustrates an example architecture of a VEPO orchestration service for orchestrating power consumption constraints and / or power-off tasks according to at least one embodiment.

[0035] Figure 8 Is a table illustrating a set of example response levels and multiple sets of corresponding reduction actions for each response level according to at least one embodiment.

[0036] Figure 9 Is a diagram illustrating an example scope of components affected by one or more response levels used by a VEPO orchestration service for orchestrating power consumption constraints and / or power-off tasks according to at least one embodiment.

[0037] Figure 10 FIG. is a block diagram illustrating an example use case applying multiple response levels based on current conditions of a set of hosts according to at least one embodiment.

[0038] Figure 11 FIG. is a schematic diagram of an example user interface according to at least one embodiment.

[0039] Figure 12 FIG. illustrates a flow of an example method for training one or more machine learning models according to at least one embodiment;

[0040] Figure 13 FIG. is a block diagram illustrating an example method for managing total power consumption based on a total power threshold according to at least one embodiment;

[0041] Figure 14 FIG. is a block diagram illustrating a mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0042] Figure 15 FIG. is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0043] Figure 16 FIG. is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0044] Figure 17 FIG. is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0045] Figure 18 FIG. is a block diagram illustrating an example computer system according to at least one embodiment. DETAILED DESCRIPTION

[0046] In the following description, various embodiments will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, those skilled in the art will also appreciate that the embodiments may be practiced without specific details. Additionally, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0047] The present disclosure relates to managing power consumption and orchestrating power consumption reduction within an environment (e.g., hosts or power distribution units in a data center, or portions thereof). More particularly, techniques for enabling orchestrated load shedding to achieve power consumption reduction due to current conditions and according to a total power threshold are described.

[0048] Balancing the power supply and consumption within a data center is desirable. When the consumed power exceeds the available supply, the balance can be restored by increasing the power supply or reducing the power consumption rate. If this balance is not maintained and a component attempts to consume more power than is available, then a circuit breaker may trip, disconnecting the component from the power source.

[0049] The maximum total power threshold for a particular environment may depend on many factors, including but not limited to the current power consumption value at a host within the environment, the operating conditions of associated components such as temperature control systems (e.g., HVAC, coolers, etc.), environmental conditions (e.g., ambient temperature, external temperature outside the environment), etc. Conventional systems are not configured to manage power consumption based on these factors. When power consumption peaks and power failures are impending, conventional techniques mainly utilize manual efforts to shut down or suspend hosts. This results in suboptimal load shedding strategies because the operators implementing the load shedding typically do not understand the impact of their actions. For example, when selecting which devices to shut down or suspend, the operator rarely (if ever) knows the customers, workloads, instances, or hosts or their respective priorities. A power failure can cause interruptions to the operations performed by the components of the data center. For example, if a circuit breaker trips, disconnecting a server from the power source of the data center, then the websites hosted on the data center server will crash. Suboptimal load shedding strategies can be inefficient and may potentially exceed the amount of power reduction sufficient to reduce the total power consumption to the desired amount. Additionally or alternatively, suboptimal load shedding strategies may have a broader / greater impact on the underlying hosts, instances, workloads, and corresponding customers than might be expected.

[0050] The balance between the power supply and power consumption of a data center can be managed by maintaining a balance between the available supply and demand and / or consumption. Sometimes, it may not be feasible to expand the power supply in a data center because power is typically statically allocated in long-term contracts with power utilities. Although the supply may be statically allocated, the power consumption in a data center can vary, sometimes significantly. As a simple example, when the number of threads executed by a server's processor increases, the power consumption of the server may increase. The ambient temperature within the data center affects the power consumed by the data center's cooling system. When the cooling system works harder, it consumes more power as it tries to reduce the ambient temperature experienced in the data center. When the cooling system fails, the data center may no longer be able to tolerate the heat generated by the current power consumption level. The demand caused by some components in the data center, such as uninterruptible power supplies, power distribution units, cooling systems, or bus ducts, may be difficult to adjust. However, some components, such as servers, virtual machines, and / or bare-metal instances, may be more susceptible to being constrained. Additionally, workloads and / or instances may be migrated to other hosts and / or instances to concentrate resources on a smaller subset of hosts, resulting in more idle and / or vacant hosts. Subsequently, applying power reduction to the idle / vacant hosts can ensure that the impact (e.g., the number of affected hosts, customers, instances, workloads) is minimized.

[0051] Many conventional power management methods utilize power capping to constrain the operation of power-consuming devices (e.g., servers, network devices, etc.) within a data center. When using power capping, a power limit ceiling can be used to constrain the power consumed by a device. The power limit ceiling is used to constrain (e.g., throttle) the operation at a server to ensure that the power consumption of the server does not exceed the power limit ceiling. Using power capping ensures that the allocated power limit of upstream devices is not breached, causing each upstream device to be supplied based on the worst-case scenario, where it is assumed that each downstream device consumes its corresponding maximum allocated power. However, these downstream devices may typically consume less power than their maximum allocated power, leaving at least a portion of the power allocated to the upstream devices unutilized. These methods waste valuable power and limit the density of power-consuming devices that can be used within the data center.

[0052] An efficient power infrastructure within a data center is necessary to increase a provider's profit margin, manage scarce power resources, and make the services provided by the data center more environmentally friendly. A data center that includes components of a hosted multi-tenant environment (e.g., a public cloud) may experience above-average consumption because not all cloud tenants use it simultaneously. To push power consumption closer to the statically allocated capacity to increase the efficiency and resource utilization of the data center, the data center provider can increase servers and / or leases such that the power consumption across all power-consuming devices is closer to the allocated power capacity of the data center. However, in some cases, narrowing the gap between the allocated power capacity and power consumption increases the risk of circuit breaker tripping and the loss of the ability to utilize computing resources. The techniques described herein minimize the frequency at which operations at downstream devices are constrained, enabling these devices to utilize previously unused power while maintaining a high level of safety in avoiding power failures.

[0053] Technical Effects

[0054] The disclosed systems and methods provide an automated and dynamic orchestration approach to constrain power consumption (e.g., via power capping, migrating workloads, etc.), suspend hosts, migrate instances and / or hosts, and / or shut down hosts to achieve a desired power reduction. In some embodiments, at least a portion of the functions performed for these actions are user-selectable and / or based on user input. The techniques described provide multiple response levels. Each response level can be associated with a set of reduction actions (referred to herein as "actions" for brevity). Each response level (referred to herein as a "level" for brevity) can provide multiple sets of increasingly stringent reduction actions to be applied. When a total power threshold is breached, a response level can be selected based on the estimated power reduction corresponding to each response level (e.g., by the system based on user input, etc.). The least stringent level and / or the level with the least impact can be selected (e.g., by the system based on user input, etc.) that, if implemented, is sufficient to cause the total power consumption of the data center to drop below the total power threshold.

[0055] The estimated impact of applying level - associated reduction actions can be determined at runtime based on the current properties of the workloads implemented on virtual machine and / or bare - metal instances, the customers associated with the workloads and / or affected hosts, and the priorities corresponding to these workloads, hosts, and / or customers. Demands (e.g., corresponding to a total power threshold) can be dynamically modified based on the operational status data and environmental data corresponding to the various components of the data center. These factors can be used to identify changes in the total power threshold (e.g., the threshold amount of total power consumption / heat / demand that the components of the data center can collectively manage) as the power management capabilities within the data center occur. The response level selection can be triggered by real - time demand changes, via a request (e.g., a user request, a request submitted by a government entity to enforce power consumption reduction, etc.), or any appropriate trigger, and can be based on the impact of implementing the corresponding reduction action.

[0056] Using the techniques described herein, the power constraints, migration tasks, and / or pause or shutdown tasks employed can be adjusted according to the current conditions and resources (e.g., hosts, instances, workloads) in the data center, thereby mitigating or completely avoiding over - reduction. These techniques provide real - time capabilities for reducing the impact of power management response actions on customers, hosts, instances, and / or workloads while ensuring an effective avoidance of the risk of power failures. The systems and methods described herein provide a more efficient and effective power orchestration method than conventional systems, which can lead to a more satisfactory user experience. The specific actions to be implemented can be selected based on at least any appropriate combination (e.g., automatically by the system, through user selection, etc.): 1) maximizing the overall power consumption reduction, 2) identifying / implementing the least - stringent response level, 3) identifying / implementing the least - impactful response level, and / or 4) identifying the response level whose estimated power reduction is sufficient to bring the current total power consumption value below the current or predicted total power threshold while also providing the least amount of over - power reduction (e.g., reduction exceeding that required to bring the current power consumption below the current total power threshold). Thus, these techniques provide a more flexible and intelligent approach to shedding load in various contexts and / or due to various trigger events.

[0057] Figure 1Depicts an example environment (e.g., environment 100) including various components according to at least one embodiment. Environment 100 can be a physical environment, such as a data center (e.g., data center 102) or a portion thereof (e.g., a room of the data center, such as room 110A). Environment 100 can include a dedicated space for hosting any suitable number of servers, such as servers 104A-P (collectively referred to as "servers 104"), and the infrastructure for hosting these servers, such as network hardware, a cooling system (also referred to as a "temperature control system"), and storage devices. Servers 104A-P can also be referred to as "hosts". The networking hardware of data center 100 (not depicted here) can enable remote users to interact with the servers via a network (e.g., the Internet). Any suitable number of servers 104 (e.g., 10, 14, 21, 42, etc.) can be maintained in various racks, such as racks 106A-H (collectively referred to as "racks 106"). Racks 106 can include a framework or enclosure in which a corresponding set of servers is placed and / or mounted.

[0058] Various subsets of racks 106 can be organized into groups called "rows" (e.g., rows 108A-D, collectively referred to as "rows 108"). In some embodiments, rows 108 can include any suitable number of racks (e.g., 5, 8, 10, up to 10, etc.) that are co-located (e.g., within a threshold distance of each other). In other embodiments, a row can be an organizational unit, and the racks having a given row can be placed in different locations (not necessarily within a threshold distance of each other). As an example, rows 108 can be located in rooms (e.g., room 110A, room 110N, etc.). A room (e.g., room 110A) can be a partition or physical enclosure or physical environment of a building in which any suitable number of racks 106 are placed. In other embodiments, a room can be an organizational unit, and the rooms can be located in different physical locations, or multiple rooms can be located in a single partition of a building.

[0059] Various temperature control systems (e.g., one or more temperature control systems 112A-N, one or more temperature control systems 114A-N, etc.) can be configured to manage the ambient temperature in a data center or a portion thereof. As a non-limiting example, one or more temperature control systems 112A-N can be associated with room 110A, while one or more temperature control systems 114A-N can be associated with room 110N. Any suitable number of temperature control systems can be associated with the data center and / or a portion thereof. In some embodiments, these temperature control systems can be any suitable heating, ventilation, and air conditioning (HVAC) equipment (e.g., air conditioning units), coolers (e.g., chilled water circulation equipment that controls temperature by circulating a liquid such as water), etc. In some embodiments, each temperature control system can be associated with the corresponding amount of heat that it is capable of and / or configured to manage (e.g., the amount of heat generated by the corresponding power consumption of server 104).

[0060] Figure 2 A simplified diagram of an exemplary power distribution infrastructure 200 including various components (e.g., Figure 1 components of data center 102) is shown in accordance with at least one embodiment. The power distribution infrastructure 200 can be connected to a utility power source (not shown), and power can be initially received at one or more uninterruptible power supplies (uninterruptible power supplies ((one or more) UPSs) 202). In some embodiments, power can be received from the utility at the (one or more) UPSs via a substation on-site (not shown) that is configured to establish an appropriate voltage level for distributing power through the data center. The (one or more) UPSs 202 can each individually include a dedicated battery or generator that provides emergency power in the event of a failure of the input power source. The (one or more) UPSs 202 can monitor the input power and provide backup power when a drop in the input power is detected.

[0061] The power distribution infrastructure 200 may include any suitable number of intermediate power distribution units ((one or more) PDUs) (e.g., (one or more) intermediate PDUs 204), which are connected to (one or more) UPSs 202 and receive power / electricity therefrom. Any suitable number of (one or more) intermediate PDUs 204 may be deployed between the UPS (in (one or more) UPSs 202) and any suitable number of row PDUs (e.g., row PDU 206). The power distribution units (e.g., (one or more) intermediate PDUs 204, row PDUs 206, (one or more) rack PDUs 208, etc.) may be any suitable devices configured to control and distribute power / electricity. Example power distribution units may include, but are not limited to, main distribution boards, distribution boards, remote distribution boards, busbars, power strips, transformers, etc. Power may be provided from (one or more) UPSs 202 to (one or more) intermediate PDUs 204. The (one or more) intermediate PDUs 204 may distribute power to downstream components of the power distribution infrastructure 200 (e.g., row PDU 206).

[0062] The power distribution infrastructure 200 may include any suitable number of row power distribution units (including row PDU 206). The row PDU may include any suitable PDU (e.g., remote distribution board, busbar / raceway, etc.) that is deployed between an intermediate PDU (e.g., the PDU in (one or more) intermediate PDUs 204) and one or more rack PDUs (e.g., rack PDU 208A, rack PDU 208N, collectively referred to as "(one or more) rack PDUs 208"). A "row PDU" refers to a PDU configured to distribute power to one or more rows of devices (e.g., row 210, including servers 212A-D, collectively referred to as "servers 212"). As described above, a row (e.g., row 210) may include any suitable number of racks (e.g., racks 214A-N, collectively referred to as "racks 214") in which the servers 212 are located.

[0063] The power distribution infrastructure 200 may include any suitable number of rack power distribution units (including (one or more) rack PDUs 208). The rack PDU may include any suitable PDU that is deployed between a row PDU (e.g., row PDU 206) and a rack (e.g., rack 214A, Figure 1Between one or more servers (e.g., server 212A, server 212B, etc.) corresponding to the rack 106 of the example). A "rack PDU" refers to any suitable PDU configured to distribute power to one or more servers within a rack. A rack (e.g., rack 214A) can include any suitable number of servers 212. In some embodiments, the (one or more) rack PDUs 208 can include intelligent PDUs that are also configured to monitor, manage, and control power consumption at multiple devices (e.g., rack PDU 208A, server 212A, server 212B, etc.).

[0064] Each of the servers 212 (for each example of the server 104 of Figure 1 can separately include a power controller ((one or more) power controllers 216A - D, collectively referred to as "power controller 216"). A power controller refers to any suitable hardware or software component operating at a device (e.g., a server) that is configured to monitor and / or manage power consumption at the device. The power controllers 216 can separately monitor the power consumption of the respective servers on which they operate. The power controllers 216 can each be configured to enforce a power capping limit to constrain the power consumption at the respective servers. Enforcing the power capping limit can include any suitable combination of the following: monitoring the power consumption at the server, determining whether to constrain (e.g., limit, cap, etc.) the power consumption at the server (e.g., at least in part based on a comparison between the current power consumption of the server and the stored power capping limit), and limiting / capping the power consumption at the server (e.g., using processor and memory dynamic voltage and frequency scaling to suppress server power consumption). Enforcing the power capping limit can be referred to as "power capping".

[0065] Figure 1 The data center 102 in can include various components depicted in the power distribution infrastructure 200. For example, room 110A can include one or more bus ducts (each bus duct being an example of a row PDU 206). A bus bar (also referred to as a "bus duct") refers to a conduit of conductive material that can distribute power (e.g., within room 110A). The bus duct can receive power from the power distribution unit of the (one or more) intermediate PDUs 204 and supply power to one or more racks (e.g., Figure 1 of row 108A) associated with the row (e.g., Figure 1 rack 106A of Figure 1 rack 106B, etc.). Each power infrastructure component that distributes / provides power to other components also consumes a portion of the power passing through it. This loss can be due to heat loss generated when power flows through the component or power directly consumed by the component (e.g., power consumed by the processor in a rack PDU).

[0066] Figure 3 Illustrates an example architecture of an example power orchestration system 300 according to at least one embodiment, the power orchestration system being configured to orchestrate power consumption reduction of various resources applicable to a physical environment (e.g., a data center, a room of a data center, etc.). The term "resource" may be considered to include hosts, workloads (e.g., virtual machines and / or bare-metal instances running these workloads) and / or customers associated with these hosts and / or workloads / instances. The power orchestration system 300 may be configured to monitor power consumption levels (e.g., current individual and / or total power consumption values corresponding to various hosts such as (one or more) hosts 324, each of which is an example of (one or more) servers 104). The power orchestration system 300 may manage power consumption within the physical environment at least in part based on this monitoring such that a total power threshold is enforced. As used herein, the total power threshold represents the maximum amount of power consumption that the components of the system 300 can manage under current conditions. The total power threshold may be dynamically adjusted as conditions change (e.g., based on an order from a government entity to reduce power consumption, based on environmental conditions (actual or predicted), based on current power consumption values (actual or predicted), based on the operating conditions (actual or predicted) of various components such as the temperature control system of the physical environment, etc.). The specific impact (e.g., scope, applicability) of the changes made to enforce the current total power threshold may vary depending on real-time conditions. Some example actions that may be employed to enforce the current total power threshold may be power capping (e.g., enforcing a maximum power consumption ceiling at a particular device such as one or more hosts 324) or otherwise allocating a budgeted amount of power to any suitable device (e.g., Figure 1 one or more hosts 324, (one or more) PDUs 202, 204, 206, 208, etc. in), pausing or shutting down hosts and / or instances (e.g., VMs or BMs) and / or workloads running at the instances, migrating a workload from one instance to another, migrating a workload from one host to another, etc.). Figure 2

[0067] The power orchestration system 300 may include various components such as Figure 3 the components depicted in. For example, the power orchestration system 300 may include a VEPO orchestration service 302. The VEPO orchestration service 302 may be configured to obtain various data based on which impact and reduction actions may be determined. By way of example, the VEPO orchestration service 302 may be configured to obtain any suitable combination of the following: power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318.

[0068] ​Power data 310 may include any suitable budget / assignment value, power ceiling value, and / or current power consumption value of (one or more) hosts 324 or (one or more) PDUs (including the rack PDU 320 ( Figure 2 example of the rack PDU 208 of Figure 2 )), which indicates the amount of power currently consumed by the device. The power data 310 may be stored in a location accessible to the VEPO orchestration service 302, and / or the power data 310 may be obtained from a source and / or service configured to manage and / or obtain such data. In some embodiments, the power management service 304 may be configured to obtain the power data 310 and store such data in a location from which the VEPO orchestration service 302 can retrieve the data. In some embodiments, the power management service 304 or another component configured to obtain such data may provide the power data 310 to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or directly via a request.

[0069] Account data 312 may include any suitable attributes of the customer, any suitable attributes of the (one or more) hosts and / or (one or more) instances associated with the customer, etc. Some example attributes may include the category associated with the customer account (e.g., free tier category, which indicates that the customer is using the service in a non-paying manner that may be provided with limited functionality or resources; free trial category, which indicates that the customer is using the service in a non-paying manner for a limited time). The category may correspond to the priority associated with the customer, or the priority may be assigned to the customer, host, or workload in other ways. The priority may indicate the degree of relationship importance, such as low priority, medium priority, and high priority, but other priority schemes may also be considered. Another example attribute may include an identifier associated with the customer, host, instance, and / or workload. The account data 312 may be stored in a location accessible to the VEPO orchestration service 302, and the account data 312 may be retrieved from that location, or the account data 312 may be obtained from a service and / or source (e.g., an account service of a cloud computing environment not shown) configured to manage and / or obtain such data. In some embodiments, the account data 310 is provided directly from the source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0070] Environmental data 314 may include any suitable data associated with the physical environment or environmental data indicating external conditions. By way of example, environmental data 314 may include an environmental temperature reading of the physical environment, an external temperature outside the physical environment, a design point indicating the external temperature that the physical environment is designed to withstand, and the like. At least a portion of the environmental data 314 may be collected from one or more sensors (not shown) configured to measure specific conditions such as the environmental temperature within the physical environment, the external temperature outside the physical environment, and the like. In some embodiments, at least a portion of the environmental data 314 may be provided by a weather source (such as a weather forecast stored in a storage device of the power orchestration system 300) or obtained from an external source (such as a weather forecast website). In some embodiments, the environmental data 314 may include tables and / or protocols from which a reduced capacity / capability may be determined / identified. By way of example, a table or mapping may be included in the environmental data 314 indicating that a particular difference between the environmental temperature and the temperature outside the physical environment (referred to as the "external" temperature) is associated with a particular reduced capacity of power consumption that the components of the system or individual components can manage. The environmental data 314 may be stored in a location accessible to the VEPO orchestration service 302 and the environmental data 314 may be retrieved from that location, or the environmental data 314 may be obtained from a service and / or source configured to manage and / or obtain such data (e.g., a metrics service of a cloud computing environment, the power management service 304, etc.). In some embodiments, the environmental data 314 is provided directly from a source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0071] Host / instance data 316 may include any suitable data associated with a host and / or an instance. The host / instance data 316 may include workload metadata that identifies the workloads being executed at the host and / or via a particular instance. In some embodiments, the host / instance data 316 may identify the corresponding customer, host, and / or instance and / or the corresponding priority of the workload, and the like. At least a portion of the host / instance data 316 may initially be obtained and / or maintained by a separate service (e.g., the compute service 306, an example of a compute service control plane of a cloud computing environment). The host / instance data 316 may be stored in a location accessible to the VEPO orchestration service 302 and the host / instance 316 may be retrieved from that location, or the host / instance 316 may be obtained from a service and / or source configured to manage and / or obtain such data (e.g., the compute service 306, etc.). In some embodiments, the host / instance data 316 may be provided directly from a source or a component configured to obtain such data to the VEPO orchestration service 302 according to a predefined schedule, frequency, or periodicity or via a request.

[0072] The operational data 318 can include any suitable data associated with the operating conditions or states of one or more devices or components corresponding to a physical environment. As a non-limiting example, the operational data 318 can include the operating conditions of one or more temperature control systems (e.g., Figure 1 the temperature control system(s) 112 of Figure 1 ). In some embodiments, the operational data 318 can indicate which temperature control systems are operable and / or the operable capacity of these systems. In some embodiments, the environmental data 314 can include tables and / or protocols from which a reduced capacity / capability can be determined / identified. By way of example, a table or mapping indicating that a failure (e.g., complete or partial) of a particular component (e.g., a particular temperature control system) is associated with a particular reduced capacity of power consumption that can be managed by components of the overall system or a single component of the system can be included in the operational data 318. At least some portion of the operational data 318 can be initially obtained and / or maintained by a separate service (e.g., the power management service 304, etc.). The operational data 318 can be stored in a location accessible to the VEPO orchestration service 302 and the operational data 318 can be retrieved from that location, or the operational data 318 can be obtained from a service and / or source (e.g., the compute service 306, etc.) configured to manage and / or obtain such data. In some embodiments, the operational data 318 can be provided directly to the VEPO orchestration service 302 from a source or a component configured to obtain such data according to a predefined schedule, frequency, or periodicity or via a request.

[0073] It should be appreciated that the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 can include current data indicating current values and / or historical data indicating corresponding historical values. In some embodiments, future attributes of the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 can be predicted at least in part based on these historical values. In some embodiments, machine learning can be utilized to predict any suitable portion of these future attributes. Example methods for predicting future values of the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 are discussed in more detail with respect to Figure 12 In some embodiments, the VEPO orchestration service 302 can be configured to aggregate data of any suitable combination of the power data 310, the account data 312, the environmental data 314, the host / instance data 316, or the operational data 318 into tables, maps, or databases from which the data can be filtered or sorted to identify subsets of resources (e.g., hosts, instances, workloads, customers) for a given set of constraints (e.g., low-priority workloads, free-tier customers, etc.).

[0074] The VEPO orchestration service 302 can be configured to identify and / or modify the total power threshold associated with the physical environment based on at least any suitable combination of the following: current / total power consumption values (actual or predicted) associated with any suitable combination of (one or more) hosts 324; a received request associated with a government entity (e.g., a local power authority) that commands / requests a specific power reduction or an overall power reduction, possibly for a specific time period (e.g., the next 24 hours); current or predicted environmental conditions (e.g., current or predicted ambient temperature, current or future external temperature, etc.); current or predicted operating conditions corresponding to the temperature control system (e.g., current or predicted total / partial failure), etc.

[0075] The VEPO orchestration service 302 can be configured to manage the actual power consumption values corresponding to the hosts / instances / workloads of the physical environment through associated functions or functions provided by other systems and / or services. Managing the actual power consumption values can include power capping, pausing workloads, instances, hosts, shutting down workloads / instances / hosts, migrating an instance from one host to another, migrating a workload from one instance and / or host to another instance and / or host, etc.

[0076] The VEPO orchestration service 302 can obtain configuration data (e.g., mappings, protocol sets, rules, etc.) corresponding to multiple response levels. Each response level can be associated with a corresponding set of reduction actions (e.g., power capping, migration, pause, shutdown, etc.) that can be performed on the components of the physical environment (e.g., (one or more) servers 104, (one or more) PDUs 202, 204, 206, 208, etc.). In some embodiments, each set of reduction actions corresponding to a specific level is associated with potentially different reductions in the total power consumption of the data center. In some embodiments, the multiple response levels indicate an increase in the severity of performing reduction actions to reduce the total power consumption of the data center. "Severity" is intended to refer to the relative degree of destructiveness of the action. For example, the action of power capping a server may be considered less severe than the action of completely shutting down the server because in the former, the server can still provide some processing power, although the capacity is reduced, while in the latter, the server provides no processing power. A set of example response levels is discussed in more detail. Figure 9 A set of example response levels is discussed in more detail.

[0077] The VEPO orchestration service 302 can be configured to determine the impact of applying a reduction action of a given level to hosts / instances / workloads (collectively referred to as "resources") of a physical environment using configuration data corresponding to the specification of (one or more) response levels and their corresponding reduction actions. The impact of applying a given action may depend on the current conditions of the runtime resources. In some embodiments, identifying the impact of a given reduction action may include identifying a set of resources and / or customers for which the action is applicable if implemented. Thus, the estimated impact is intended to refer to the scope or applicability of a given action or level. Identifying the scope and / or applicability of a given action or level may include identifying the specific resources and / or customers that will be affected by implementing the action or level and / or identifying any appropriate attributes associated with the applicable resources and / or customers. As a simple example, the VEPO orchestration service 302 can identify that if the action of shutting down all idle hosts is taken, then X idle hosts among the (one or more) hosts 324 will be affected, and these (one or more) hosts are associated with a specific customer or a specific number of customers, and / or the (one or more) hosts and / or customers are associated with other attributes such as category (e.g., free tier), priority (e.g., high priority), etc.

[0078] The VEPO orchestration service 302 can be configured to estimate the impact and / or actual power reduction that one or more levels may experience based at least in part on any suitable combination of configuration data specifying levels and corresponding actions and power data 310, account data 312, host / instance data, etc. The estimated impact (e.g., which (one or more) hosts, (one or more) instances, (one or more) workloads, (one or more) customers may be affected, which attributes are associated with the affected (one or more) hosts, (one or more) instances, (one or more) workloads, and / or customers, how many of the affected (one or more) hosts / (one or more) instances / (one or more) workloads / (one or more) customers, etc.) can be identified for one or more response levels, one or more reduction actions corresponding to a given level, etc. In some embodiments, the estimated impact of a given action can be aggregated with the estimated impact of all actions of a given level to identify the estimated impact of the given level.

[0079] The VEPO orchestration service 302 can be configured to determine the estimated impact and / or actual power reduction that one or more levels may experience based at least in part on current data, historical data, or predictive data (e.g., current values, historical values, or predictive values corresponding to power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318).

[0080] In some embodiments, the VEPO orchestration service 302 can be configured to present, recommend, or automatically select a given response level from a set of possible response levels based at least in part on any suitable combination of the estimated impact and / or estimated power reduction that might be experienced if a reduction action corresponding to a given level is implemented (e.g., enforced). Thus, in some embodiments, the power orchestration system 300 can recommend and / or select a particular level based on any suitable combination of: 1) determining a level that, if implemented, would likely result in a sufficient but not excessive reduction in power consumption at the resources of the physical environment (e.g., the minimum power consumption reduction amount sufficient to bring the current power consumption value below the current total power threshold), 2) determining a level that includes a least restrictive set of actions, or 3) determining a level or action with the least impact (e.g., a set of resources and / or customers that are likely to be affected by the fewest number of resources / customers, have a priority or category indicating the lowest degree of collective importance, etc.). Determining the least restrictive level or action, or the level or action with the least impact, can be determined independent of the estimated power consumption reduction that might be experienced by implementing that level and / or action, or determining the least restrictive level / action and / or the level / action with the least impact can be determined from one or more levels that, if implemented under the given current conditions, are estimated to result in a sufficient reduction in power consumption such that the power consumption value drops to an aggregate value below the current total power threshold).

[0081] The power management service 304 can be configured to provide functionality for identifying power ceiling values for one or more hosts and / or instances based at least in part on the total power threshold, the budget / assigned power threshold associated with the (one or more) PDUs, and / or the current individual and / or total power consumption values. In some embodiments, the power management service 304 can be configured to identify the resources to which power ceilings are to be applied and / or the specific values of these power ceilings based at least in part on any suitable combination of current, historical, or predicted values of the power data 310, the account data 312, the environmental data 314, and the host / instance data 316. In some embodiments, the VEPO orchestration service 302 can trigger and utilize the functionality provided by the power management service 304 for identifying / modifying the budget / assigned power corresponding to one or more resources (e.g., hosts, instances, PDUs, etc.), identifying which or how many resources are applicable, and any suitable combination of the estimated power reduction values that might be experienced if a given level of (one or more) actions is implemented. In some embodiments, the power management service 304 can perform some or all of these functions.

[0082] The VEPO orchestration service 302 and / or the power management service 304 may host the user interface 308. The user interface 308 may be configured to provide any suitable application metadata corresponding to a combination of: current individual and / or power consumption values corresponding to any suitable number or type of resources (e.g., hosts, instances, workloads, etc.) or customers, any suitable attributes associated with these resources or customers (e.g., priority, category, etc.), prediction conditions (e.g., changes in predicted total power thresholds), current conditions (e.g., current total power thresholds), estimated impacts (e.g., the number, identifier, or other attributes associated with the affected host / instance / workload customers), and / or estimated power reduction (e.g., the estimated reduction expected to be experienced if the action at that level is applied), and / or the identification of the action(s) corresponding to a given level. The user interface 308 may present any suitable combination of this application metadata. The application metadata may correspond to any suitable combination of available response levels. In some embodiments, the application metadata corresponding to the current / predicted power consumption values and / or the current / predicted total power thresholds may be displayed. In some embodiments, the application metadata presented at the user interface 308 may correspond to any suitable number of levels of estimated power consumption reduction and / or estimated impact. In some embodiments, the VEPO orchestration service 302 may provide a particular level of recommendation (e.g., via the provided application metadata corresponding to that level or the reduction action associated with that level), and the confirmation / or rejection of the recommended level may be input via an optional option at the user interface 308. If confirmed, the VEPO orchestration service 302 may be configured to perform operations to effectuate the implementation of the confirmed level and its corresponding actions. In some embodiments, the user interface 308 may present the application metadata for any suitable number of response levels and / or corresponding reduction actions, and one or more options may be provided at the user interface 308 to enable the user to select a particular level and / or action.

[0083] Receiving user input that indicates selection of a particular level and / or action or that confirms selection of a particular level and / or action can cause the VEPO orchestration service 302 to perform operations to effectuate the implementation of the confirmed level and its corresponding action. Effectuating or implementing the confirmed level and / or its corresponding action can include instructing any suitable component of the power orchestration system (e.g., the power management service 304, the compute service 306) to effectuate any suitable portion of the level and / or the corresponding (one or more) actions. Instructing one or more components can include providing any suitable data, such as any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, operation data 318, to the instructed (one or more) components. In some embodiments, the instructed components can be configured to effectuate the (one or more) actions or levels based on the data provided by the VEPO orchestration service 302, or the instructed components can obtain any suitable portion of the data from a location storing the power data 310, account data 312, environmental data 314, host / instance data 316, or operation data 318.

[0084] As a non-limiting example, if the (one or more) actions for the level of reduction or impact to be estimated include power capping, then the power management service 304 can be configured to estimate a power ceiling, estimate individual or total power consumption reduction, and / or effectuate the estimated power ceiling to achieve the estimated individual / total power consumption reduction. In some embodiments, the power ceiling can be effectuated directly by the power management service 304, or by instructing the compute service 306. Other actions, such as migrating instances or workloads, pausing instances, workloads, or hosts, shutting down hosts, or preventing (at least for a period of time) future assignment or start of instances and / or workloads, can be identified by the VEPO orchestration service 302 and / or the compute service 306 and effectuated by the VEPO orchestration service 302 through the functionality provided by the compute service 306. In some embodiments, the compute service 306 can be configured to identify potentially affected hosts / instances / workloads / customers and provide such information to the VEPO orchestration service 302 and / or the power management service 304.

[0085] The compute service 306 can be configured to communicate with the baseboard management controller (BMC) / integrated lights-out manager (ILOM) 322 of the rack PDU 320. ILOM is an example of a BMC for illustrative purposes. In some embodiments, the BMC / ILOM 322 can be a dedicated service processor that monitors the physical state of the host machine, which in this case is the rack PDU 320. Similarly, the (one or more) hosts 324 (e.g., the (one or more) hosts of the rack 319, to which the rack PDU 320 is coupled Figure 5The power distribution hierarchy being discussed manages its power and can include a BMC / ILOM 328, which can be a dedicated service processor that monitors the physical state of the host, which in this case is one of the host(s) 324. The BMC / ILOM 322 and the BMC / ILOM 328 can be configured to manage and / or enforce, etc., the budgeted power and / or power capping on the device on which the BMC / ILOM 328 operates or on downstream devices. For example, the BMC / ILOM 322 can receive a power capping value for use by the BMC / ILOM 328 to constrain the power consumption of a given host. The BMC / ILOM 322 can distribute the power capping to the corresponding BMC / ILOM 328 associated with that power capping. In some embodiments, the BMC / ILOM 322 can distribute the (one or more) power cappings stored at the applicable host(s), but the enforcement of these power cappings does not occur until the BMC / ILOM 322 provides further instructions to the BMC / ILOM 328. The BMC / ILOM 328 can enforce the power capping at the host level and / or instance level and / or at the workload level of one or more workloads (not shown) executed at the instance.

[0086] The BMC / ILOM 322 to the BMC / ILOM 328 can be used to instruct a device (e.g., the rack PDU 320 and / or the host(s) 324) to pause operation or shut down the host / instance / workload. In some embodiments, the BMC / ILOM 322 to the BMC / ILOM 328 can be instructed to resume the operation of the host / instance / workload. If shut down, then the device (e.g., via its BMC / ILOM) can be instructed to start from the shut-down state (e.g., by the computer service 306 and / or by an upstream device such as its PDU). This can include the computing service 306 sending the (one or more) instructions to the BMC / ILOM 322 to the BMC / ILOM 328 to pause and / or shut down the host / instance / workload.

[0087] In some embodiments, the computing service 306 can perform any suitable operation to migrate a workload from one instance to another (e.g., migrate to an instance on the same host, migrate to a different instance operating on a different host, etc.), migrate an instance from one host to another (e.g., hosts in the same rack, hosts in different racks, etc.), migrate the instance / workload back to the host that initially hosted the previously migrated instance / workload, etc. In some embodiments, the computing service 306 can be configured to ensure that a host / instance / workload is not assigned to a host and / or instance to which a level and / or action applicable to the implementation or ongoing implementation pertains.

[0088] Any suitable combination of devices (e.g., one or more hosts 324, rack PDU 320) may include one or more power controllers (e.g., BMC / ILOM 328, BMC / ILOM 322, respectively). A power controller may be any suitable hardware or software component operating at a device (e.g., a server) that is configured to monitor and / or manage power consumption at the device. A power controller may individually monitor the power consumption of the corresponding device on which it operates. Each power controller may be configured to enforce a power capping limit to constrain the power consumption at the corresponding device. Enforcing a power capping limit may include monitoring the power consumption at the device, determining whether to constrain (e.g., limit, cap, etc.) the power consumption at the device (e.g., at least in part based on a comparison between the device's current power consumption and a stored power capping limit), and limiting / capping the power consumption at the device (e.g., using dynamic voltage and / or frequency scaling of the device's one or more processors and / or memory to throttle the power consumption at the device). Any suitable operation associated with power capping may be implemented by the power controller.

[0089] In some embodiments, a power controller (e.g., BMC / ILOM 322) may communicate with another power controller (e.g., BMC / ILOM 328) corresponding to one of the one or more hosts 324 via a direct connection (e.g., via a cable) and / or via a network or networks. A power controller (e.g., BMC / ILOM 328) may provide power consumption data indicative of the current power consumption of the device (e.g., cumulative power consumption over a period of time, current power consumption rate, etc.). The power consumption data may be provided to a power controller (e.g., BMC / ILOM 322) at any suitable frequency, periodically, or according to a predefined schedule or event (e.g., upon breaching a predefined consumption threshold, upon a threshold amount change in the consumption rate, upon determining that one or more predefined conditions are met, upon determining the thermal properties of the device, etc.).

[0090] In some embodiments, a power controller (e.g., BMC / ILOM 328) may receive a power capping value (also referred to as a "power cap") from a power controller (e.g., BMC / ILOM 322). In some embodiments, additional data may be provided with the power capping value. By way of example, an indicator may be included in the power capping value that indicates whether the power capping value is to be applied immediately. In some embodiments, it may be default to immediately apply / enforce the received power cap. In other embodiments, it may be default not to immediately apply / enforce the received power cap.

[0091] When applying / enforcing a power limit, a power controller (e.g., BMC / ILOM 328) can monitor the power consumption at a device. This can include using metering devices or software that are configured to identify / calculate the power consumption data of the device (e.g., cumulative power consumption over a period of time, current power consumption rate, change in the current power consumption rate within a time window, etc.). As part of applying / enforcing a power limit (also referred to as "power capping"), the power controller (e.g., BMC / ILOM 328) can determine whether to constrain (e.g., limit, cap, etc.) the power consumption at the device (e.g., at least in part based on comparing the current consumption rate of the device and a stored power limit value). When constraining the power consumption at the device, the power controller (e.g., BMC / ILOM 328) can limit / cap the power consumption at the device (e.g., using dynamic voltage and frequency scaling of (one or more) processors and / or memories to suppress the power consumption at the device). In some embodiments, when the current consumption data of the device (e.g., cumulative consumption, consumption rate within a time window, etc.) approaches the enforced power limit value (e.g., breaches a threshold less than the power limit value), the power controller (e.g., BMC / ILOM 328) can execute instructions to limit / cap the power consumption of the device. When constraining / capping the power consumption at the device (also referred to as "throttling"), the power controller (e.g., BMC / ILOM 328) ensures that the power consumption of the device remains below the power consumption indicated by the power limit value. The power controller (e.g., BMC / ILOM 328) can be configured to generally constrain the power at the host, or to constrain the power at the host according to instances and / or workloads to which the power limit may apply.

[0092] In some embodiments, the power controller of (one or more) hosts 324 can be configured to allow (one or more) hosts 324 to run unconstrained (e.g., not capped based on power consumption and power limit) until the power controller (e.g., BMC / ILOM 328) is instructed to apply / enforce a power limit. In some embodiments, the power controller (e.g., BMC / ILOM 328) can receive a power limit value that may or may not later be instructed to be enforced, but when received, the power limit value can be stored in memory and not used for power management at the device. Thus, in some embodiments, the power controller (e.g., BMC / ILOM 328) can avoid starting a power capping operation (e.g., the above comparison and determination) until it is instructed to do so (e.g., via an indicator provided by power controller 448). The indication to start enforcing the power limit can be received together with the power limit value, or can be received as a separate communication from power controller 448.

[0093] According to the power distribution hierarchy 500, each power controller of one or more PDUs (e.g., BMC / ILOM 322) can be configured to distribute, manage, and monitor power for any suitable number of (one or more) devices (e.g., (one or more) hosts 324 of the rack 319). In some embodiments, the power controller (e.g., BMC / ILOM 322) can be a computing agent or program installed at a given PDU (e.g., Figure 3 the rack PDU 320). The power controller (e.g., BMC / ILOM 322) can be configured to obtain power consumption data from (one or more) hosts 324 of the rack 319 corresponding to the devices (e.g., the devices for which the given PDU is configured to distribute, manage, or monitor power) associated therewith. The power controller (e.g., BMC / ILOM 322) can receive the consumption data according to a predefined frequency, periodicity, or schedule implemented by the power controller (e.g., BMC / ILOM 328), and / or the power controller (e.g., BMC / ILOM 322) can receive the consumption data in response to a request for the consumption data to the power controller (e.g., BMC / ILOM 328). The power controller (e.g., BMC / ILOM 322) can be configured to request the consumption data from the power controller (e.g., BMC / ILOM 328) according to a predefined frequency, periodicity, or schedule implemented by the power controller (e.g., BMC / ILOM 322).

[0094] The power controller (e.g., BMC / ILOM 322) can transmit the received consumption data (directly or through the computing service 306) to the power management service 402 at any suitable time according to any suitable frequency, periodicity, or schedule, or due to the satisfaction of one or more predefined conditions (e.g., the individual / cumulative / collective consumption rate of the (one or more) hosts 324 changes by more than a threshold). In some embodiments, the power controller (e.g., BMC / ILOM 322) can aggregate (e.g., via the power controller (e.g., BMC / ILOM328) of each device) the power consumption data received from the (one or more) hosts 324 before transmitting the aggregated consumption data to the power management service 402. In some embodiments, the consumption data can be aggregated by host, instance type / category / priority, workload type / category / priority, or customer.

[0095] A power controller (e.g., BMC / ILOM 322) can receive one or more power ceiling values corresponding to one or more hosts 324 at any appropriate time (e.g., from the execution manager 410 of the power management service 402). These one or more power ceiling values can be calculated by the power management service 402 (e.g., via the constraint identification manager 408). In some embodiments, the power controller (e.g., BMC / ILOM 322) can receive a timing value, where the power ceiling value indicates the duration to be used for a timer. The power controller (e.g., BMC / ILOM 322) can be configured to generate and initiate a timer having an associated duration / time period corresponding to the timing value. When the timer expires, indicating that the time period corresponding to that time period has passed, the power controller (e.g., BMC / ILOM 322) can transmit data to one or more hosts, including an indicator or other appropriate data instructing the one or more hosts to continue power capping using the power ceiling values previously stored at each device. In some embodiments, the power ceiling value can be provided with an indicator. In other embodiments, the power ceiling value can be provided by the power controller (e.g., BMC / ILOM 322) to the power controller of the device (e.g., BMC / ILOM 328) immediately after receiving the one or more power ceiling values from the power management service 402. Thus, in some embodiments, the power ceiling value can be transmitted by the power controller (e.g., BMC / ILOM 322) to the power controller (e.g., BMC / ILOM 328), stored in memory, but not enforced by the power controller (e.g., BMC / ILOM 322) until the power controller (e.g., BMC / ILOM 328) receives subsequent data from the power controller (e.g., BMC / ILOM 322) indicating that the power controller (e.g., BMC / ILOM 328) should start power capping.

[0096] The VEPO orchestration service 302, the power management service 304, the compute service 306, the rack PDU 320, one or more hosts 324 (collectively referred to as " Figure 3The "(one or more) devices" can communicate via one or more wired or wireless networks (e.g., (one or more) networks 808). The storage device storing power data 310, account data 312, environmental data 314, host / instance data 316, and operation data 318 can also communicate with the VEPO orchestration service 302, power management service 304, computing service 306, rack PDU 320, (one or more) hosts 324 via one or more wired or wireless connections (e.g., via one or more networks). In some embodiments, (one or more) networks can include any one or a combination of various different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks.

[0097] Figure 3 The (one or more) devices can be any suitable type of computing device, such as, but not limited to, server devices, networking devices, or any suitable device within a data center. In some embodiments, some devices (e.g., rack PDU 320 and (one or more) hosts 324) are arranged in a power distribution hierarchy (such as the power distribution hierarchy 500 discussed in conjunction with Figure 5 In some embodiments, (one or more) hosts 324 can correspond to and be represented by level 1 nodes of the power distribution hierarchy 500. The rack PDU 320 ("rack leader") can correspond to and be represented by a higher-level node of the power distribution hierarchy (e.g., a level 2 node).

[0098] Figure 3 Each of the (one or more) devices can include at least one memory. The (one or more) processors can each be suitably implemented in hardware, computer-executable instructions, firmware, or a combination thereof. Figure 3 The computer-executable instruction or firmware implementation of the (one or more) processors of the (one or more) devices can include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0099] Figure 3 The memory of the (one or more) devices can store program instructions that can be loaded and executed on the (one or more) corresponding processors of a given device, as well as data generated during the execution of these programs. Depending on the configuration and type of the user computing device, the memory can be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). Figure 3The (one or more) devices may also include additional removable and / or non-removable storage devices, including but not limited to magnetic storage devices, optical disks, and / or tape storage devices. The disk drive and its associated computer-readable medium can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device. In some embodiments, the memory may separately include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM.

[0100] Turning to the contents of the memory in more detail, the memory may include an operating system, one or more data repositories, and one or more applications, modules, instances, workloads, or services, such as the VEPO orchestration service 302, the power management service 304, the computing service 306, etc.

[0101] Figure 3 The (one or more) devices may include (one or more) communication connections that allow the (one or more) devices to communicate with each other via (one or more) networks (not shown). Figure 3 The (one or more) devices may also include I / O devices, such as a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc.

[0102] Figure 4 Illustrated is an example architecture 400 of a power management service 402 ( Figure 3 an example of the power management service 304) according to at least one embodiment. The power management service may include an input / output processing manager 404, which may be configured to receive and / or transmit any appropriate data between the power management service 402 and Figure 3 any other appropriate devices. As depicted, the power management service 402 may include a consumption monitoring manager 406, a constraint identification manager 408, and an enforcement manager 410, but more or fewer computing components or subroutines may be similarly utilized.

[0103] The consumption monitoring manager 406 may be configured to determine current individual and / or total power consumption values (e.g., corresponding to Figure 3 the (one or more) hosts 324, each host being Figure 1(example of one or more) servers 104). The constraint recognition manager 408 can be configured to perform any suitable function corresponding to identifying one or more power capping values for a host, instance, or workload (e.g., (one or more) hosts 324). According to at least one embodiment, the enforcement manager 410 can be configured to enforce and / or trigger the enforcement of a power cap at any suitable device. Allocating power can refer to the process of assigning a budgeted amount of electrical power (e.g., expected load / power draw) to any suitable component. A power cap can be an example of a budgeted amount of electrical power.

[0104] In some embodiments, the functions described in connection with the power management service 402 can be performed by one or more virtual machines implemented in a hosted computing environment (e.g., on (one or more) hosts 324, corresponding to Figure 1 any suitable number of servers 104). The hosted computing environment can include one or more rapidly provisioned and released computing resources, which can include computing, networking, and / or storage devices. The hosted computing environment can also be referred to as a cloud computing environment. Multiple example cloud computing environments are provided and discussed in more detail below. Figures 14 - 18 Provide and discuss multiple example cloud computing environments in more detail.

[0105] The power management service 450 (e.g., the consumption monitoring manager 406) can be configured to receive / obtain consumption data from (one or more) PDUs (e.g., from a rack PDU 320 of a power controller (such as Figure 3 the BMC / ILOM 322). As described above, the consumption data can be aggregated or accumulated for one or more devices. As a non-limiting example, an instance of the consumption data received by the power management service 402 can include the cumulative / aggregated consumption data for each device associated with a given PDU (e.g., managed by a given PDU). For example, the PDU providing the consumption data can be a rack PDU (e.g., rack PDU 320), and the consumption data provided by that PDU can include the cumulative / aggregated and / or individual consumption data values for each device in the rack (e.g., a host in (one or more) hosts 324 associated with a given rack 319). In some embodiments, the cumulative / aggregated consumption data values can additionally include the power consumption of the PDU. The power management service 402 can access the individual and / or aggregated or cumulative power consumption data for individual devices or all devices associated with that instance of the consumption data from the consumption data. The power management service 402 can be configured to be at least partially based on representing Figure 5Perform any suitable operations (e.g., aggregate consumption data, calculate power ceiling values, calculate timing values, determine budget power values, identify whether power capping is required (e.g., when the aggregated consumption breaches / may breach the budget power, etc.)) on the power distribution hierarchy 500's data and / or any suitable configuration data that identifies the arrangement of components within the data center (e.g., indicating which devices distribute power to which devices). For example, the configuration data can include data representing the power distribution hierarchy 500.

[0106] Figure 5 Illustrates an example of a power distribution hierarchy 500 corresponding to the arrangement of components of Figure 2 The power distribution hierarchy 500 can represent the arrangement of any suitable number of components of a power system (such as the power distribution infrastructure components discussed in connection with Figure 2 the power distribution infrastructure 200). The power distribution hierarchy 500 can include any suitable number of nodes organized according to any suitable number of levels. Each level can include one or more nodes. The root level (e.g., level 5) can include a single root node of the power distribution hierarchy 500. Each node of the power distribution hierarchy 500 can represent a corresponding component of the power distribution infrastructure 200. A group of one or more nodes at a given level can descend from a particular node corresponding to a higher level of the power distribution hierarchy 500. A group of lower-level components represented by nodes at a lower level (e.g., level 1) can receive power distributed by a higher-level component represented by a node at level 2 (e.g., a component upstream of the lower-level components), which in turn receives power from a higher-level component represented by a node at level 3, which receives power from a higher-level component represented by a node at level 4. In some embodiments, each component of the power system receives power initially distributed by a component corresponding to a node at level 5 (e.g., the highest level of the power distribution hierarchy 500) (e.g., node 502, the root node). Node 502 can receive power from a utility power source (e.g., a local power utility system).

[0107] As depicted, the power distribution hierarchy 500 includes node 502 at level 5. In some embodiments, node 502 can represent Figure 2the uninterruptible power supply of one or more UPSs 202. Components corresponding to node 502 can distribute / supply power to components corresponding to node 504 at level 4. Components (e.g., lower-level components) that receive power from higher-level components (components represented by nodes in the power distribution hierarchy 500 that are at a higher level than the level of the nodes representing the lower-level components) can be considered subordinate to the higher-level components. In some embodiments, node 304 can represent a component subordinate to the UPS represented by node 502, such as Figure 2 one of the intermediate PDUs 204 (e.g., distribution panel). The component represented by node 504 can be configured to distribute / supply power to the components represented by nodes 506 and 508 respectively.

[0108] Nodes 506 and 508 at level 3 can each represent Figure 2 corresponding components (e.g., corresponding one or more row PDUs). The component corresponding to node 506 (e.g., row PDU 206, busbar) can distribute / supply power to the component corresponding to node 510 (e.g., Figure 2 rack PDU 208A) and the component corresponding to node 512 (e.g., Figure 2 rack PDU 208N). The component corresponding to node 510 (e.g., rack PDU 208A) can distribute / supply power to the components corresponding to nodes 514 and 516 at level 1 (representing Figure 2 servers 212A and 212B respectively), and these components can be monitored / managed by components of servers 212A and 212B such as power controllers 216A and 216B. The components corresponding to nodes 514 and 516 can be arranged in the same rack.

[0109] The component corresponding to node 512 (e.g., rack PDU 208A) can distribute / supply power to the component corresponding to node 518 at level 1 (e.g., Figure 2 server 214C including power controller 216C). Nodes 514, 516, and 518 at level 1 can be arranged in the same row (e.g., Figure 2 row 210) and / or associated with the same row.

[0110] Returning to node 508 at level 3, node 508 (e.g., a different row PDU, such as a remote power board) can distribute / supply power to the component corresponding to node 510 (e.g., Figure 2 rack PDU 208A) and the component corresponding to node 512 (e.g., Figure 2The rack PDU 208 distributes / supplies power. Components corresponding to node 520 (e.g., the rack PDU) can distribute / supply power to components corresponding to nodes 522 and 526 at level 1 (e.g., corresponding to respective servers including respective power controllers). Components corresponding to nodes 524 and 526 can be arranged in the same rack. Components corresponding to node 522 (e.g., another rack PDU) can distribute / supply power to components corresponding to nodes 528 and 530 at level 1 (e.g., corresponding to respective servers including respective power controllers). Components corresponding to nodes 528 and 530 can be arranged in the same rack.

[0111] The specific number of components (e.g., corresponding to level 1 nodes) that receive power distributed from a higher-level component (e.g., corresponding to a node at level 2) may be different from Figure 5 the number shown. The specific number of levels within the power distribution hierarchy may vary depending on the specific arrangement of components used in a given data center. It is contemplated that each non-root level (e.g., in the Figure 5 example, levels 1 - 4) may include a different number of nodes, which represent a different number of components than the number of nodes depicted in each non-root level of Figure 5 . The nodes can be arranged in a configuration different from but similar to the configuration depicted in Figure 5 .

[0112] Return Figure 4 , the power management service 402 (e.g., the constraint identification manager 408) can calculate aggregated / cumulative consumption data to identify aggregated / cumulative consumption data for one or more devices at a higher level of the power distribution hierarchy 500. By way of example, the power management service 402 (e.g., the constraint identification manager 408) can utilize consumption data provided by one or more rack PDUs and calculate the aggregated / cumulative consumption data for the row of devices. For example, the consumption data corresponds to the devices represented by the level 1 nodes of Figure 5 , and these devices share a common set of nodes up to level 3 of the hierarchy. In this case, the common level 3 nodes represent row PDUs, such as busbars. In some embodiments, the consumption data indicating the power consumption of one or more rack PDUs (e.g., an example of a component corresponding to level 2 of the power distribution hierarchy 500) can be included in the consumption data provided to the power management service 402. The consumption data provided by the rack PDUs can be provided according to any suitable frequency, periodicity, or schedule, or in response to a request transmitted by the power management service 402. The power management service 402 can obtain data related to Figure 3Consumption data associated with any suitable number (one or more) of devices (such as (one or more) hosts 324 and / or racks of (one or more) devices corresponding to one or more racks (e.g., rack 319)) can be aggregated or calculated for any suitable number of components represented by any suitable nodes and / or levels of the power distribution hierarchy 500.

[0113] The power management service 402 (e.g., the constraint identification manager 408) can be configured to calculate one or more power caps (e.g., the power cap values for one or more servers to which power is distributed by the higher-level device (e.g., the row-level device in this example)) at least in part based on the power values (e.g., budgeted power amounts) assigned to the higher-level components. The higher-level components can correspond to any suitable level (e.g., levels 2 - 5) of the power distribution hierarchy 500 other than the lowest level (e.g., level 1). For example, the power management service 402 can calculate the power cap values for servers to which power is distributed by a given row device (e.g., a bus bar). These calculations can be based on consumption data provided by the rack PDU to which power is distributed by the row device. In some embodiments, the power management service 402 can store the consumption data for subsequent use. The power management service 402 can utilize historical consumption data when calculating these power cap values. In some embodiments, the power management service 402 can obtain, utilize, and / or train one or more machine learning models to identify specific power cap values for one or more hosts 324 (e.g., components corresponding to level 1 of the power distribution hierarchy) from historical consumption data. These techniques will be discussed in more detail with respect to Figure 12 be discussed in more detail.

[0114] The power management service 402 (e.g., the enforcement manager 410) can calculate the timing value of a timer (e.g., a timer that can be initiated and managed by (one or more) PDUs (such as rack PDU 320)). The timing value can be calculated based on at least any suitable combination of the following: the change in the power consumption rate of the higher-level component, the direction of change (increase / decrease) in the power consumption of the higher-level component, the power spike tolerance of the higher-level component and / or the entire system 300, or the enforcement time associated with the lower-level device (e.g., the time when each of (one or more) devices (e.g., (one or more) hosts 324) is instructed to enforce the power cap value and the time when each of (one or more) devices (e.g., (one or more) hosts 324) will actively enforce the cap (e.g., the time to first constrain / throttle the power consumption at that device, or at least determine whether to constrain / throttle)).

[0115] The power management service 402 (e.g., enforcement manager 410) can transmit the calculated timing value to the (one or more) PDUs 402 at any appropriate time. In some embodiments, the power management service 450 can initially determine the power ceiling for a given higher-level component (e.g., row component) without considering the consumption that occurs relative to other components at the same level (e.g., other row components). In some embodiments, when the timer is initialized or elapses, or at any appropriate time, the power management service 402 can process the consumption data corresponding to other components at the same level to determine whether it is more desirable to cap the power of one or more downstream components of these components at the same level. In some embodiments, the power management service 402 can utilize the priorities associated with the (one or more) devices and / or the workloads running on the (one or more) devices to determine the power ceiling value (e.g., at least partially based on Figure 3 the priorities identified by the host / instance data 316). The power management service 402 can be configured to preferentially cap the power of the (one or more) low-priority devices / (one or more) workloads while leaving the (one or more) devices / (one or more) workloads with higher priority values unconstrained. In some embodiments, the power management service 402 can be configured to preferentially cap the power of a group of the highest-consuming devices (e.g., across one row, multiple rows, etc.) while leaving the lower-consuming devices to operate unconstrained. In some embodiments, if the priority associated with a device and / or workload is high (or higher than the priorities associated with other devices and / or workloads), then even if the device is included in the group of the highest-consuming devices, the specific consumption of the device can be left unconstrained. Thus, the power management service 402 can be configured to prioritize the priorities of the devices / workloads over the power consumption at that particular device.

[0116] In some embodiments, the power ceiling values can be initially determined by the power management service 402 (e.g., by the constraint identification manager) for a given row. These power ceiling values can be provided to the power controller of the rack PDU (e.g., BMC / ILOM 320, or another agent or component of the rack PDU 320), which in turn can distribute the power ceiling values to the power controllers (e.g., BMC / ILOM 328) at the host(s) 324 for storage. The power management service 402 (e.g., the constraint identification manager 408) can process the consumption data of other row devices on the same row to determine the power ceiling values for the devices corresponding to different rows. This can be advantageous because the devices managed by another row may not consume their budgeted power, leaving some unutilized power at the devices at that row level. In some embodiments, the power management service 402 (e.g., the constraint identification manager 408) can be configured to determine whether it is more advantageous to cap the power of the devices of one row while allowing at least some devices of another row to operate without constraint. The benefits of determining a set of power ceiling values can be at least partially based on minimizing the estimated impact (e.g., the number of devices to be power-capped), minimizing the priority values associated with the devices to be power-capped, maximizing the number of devices associated with a particular high-priority value that will not be power-capped, etc. The power management service 402 can utilize a predefined protocol (e.g., a set of rules) to determine whether enforcing the power ceiling values it has sent to one PDU is more or less advantageous / beneficial than the different power ceiling values it has identified based on processing the consumption data of multiple PDUs associated with one or more other rows.

[0117] The power management service 402 (e.g., the enforcement manager 410) can be configured to formulate the enforcement of a set of power ceiling values determined to be the most advantageous (based on a predefined set of rules) to ensure that one or more circuit breakers within the data center do not trip and / or the total power threshold is enforced. If the current / total power consumption level exceeds the current total power threshold, then the power management service 402 can be configured to perform or trigger an execution operation to bring the current / total power consumption level to a value below the current total power threshold.

[0118] In some embodiments, the monitoring, identification, and / or constraint based on the power ceiling values can be determined directly by the power management service 402 based on power data 310, account data 312, environmental data 314, host / instance data 316, operation data 318, etc.

[0119] In some embodiments, the power management service 402 may be configured to identify and / or enforce a power ceiling value based at least in part on instructions and / or constraints provided by the VEPO orchestration service 302 and / or other applicable data. As a non-limiting example, the VEPO orchestration service 302 may filter aggregated data of any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318 based on a reduction action associated with a level (e.g., "reduce the power ceiling of all low-priority hosts by 10%") to determine the estimated impact of implementing the action, including the number and / or subset of resources (e.g., hosts, instances, workloads, customers) to which the action would be applied if the action were implemented. For example, the VEPO orchestration service 302 may identify the number and / or identifiers corresponding to all low-priority hosts from the aggregated data of any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the identifier and / or instruction (10% reduction in power ceiling) may be provided to the power management service 402, which receives it via the input / output processing module 404 and is implemented by the enforcement manager 410. As another non-limiting example, any suitable portion of the metadata associated with a reduction action (e.g., "reduce the power ceiling of all low-priority hosts by 10%") may be provided to the power management service 402, and the constraint identification manager 408 may be configured to identify specific hosts / instances / workloads from the metadata associated with the reduction action. In other words, any suitable combination of the VEPO orchestration service 302 and / or the power management service 402 (e.g., the power management service 304) may be configured to identify the estimated impact of a given action. In some embodiments, the power management service 402 (e.g., the constraint identification manager 408) may be configured to identify an estimated power reduction value corresponding to the individual or total power consumption of one or more devices (e.g., (one or more) hosts 324), assuming hosts / instances / workloads / customers that are expected to be affected if the action (or level) were implemented.

[0120] If it is determined that a previously provided set of power caps to a power controller (e.g., BMC / ILOM 322 and / or BMC / ILOM 322) is most beneficial, then the power management service 402 can transmit data to the power controller to cause the power controller to transmit an indicator to the (one or more) devices to start power capping based on the previously distributed power caps. Alternatively, if the power management service 402 determines that a new set of power caps is more (or most) beneficial from a power management perspective, then it can transmit data to the power controller (e.g., BMC / ILOM 322) to cancel the timer. In some embodiments, canceling the timer may cause the previously distributed power caps to be deleted by an instruction from the power controller (e.g., transmitted in response to canceling the timer), or these power caps may default to timing out (e.g., according to a predefined time period). The power management service 402 can transmit the new power caps to a suitable PDU (which may or may not include the same PDU to which the previous set of power caps was transmitted) and indicate that the corresponding devices should immediately enforce these power caps. These PDUs can transmit these power caps along with instructions to the receiving devices to immediately start power capping operations (e.g., including monitoring the consumption related to the power cap value, determining whether the cap is based on the power cap value and the current consumption of the device, and capping or unconstraining the operation of the device based on the determination).

[0121] In some embodiments, if the timer expires at the original PDU (e.g., the time period corresponding to the timing value has passed) and no cancellation has been received from the power management system 402, then the power controller (e.g., BMC / ILOM 322) can automatically (e.g., via BMC / ILOM 328) instruct the (one or more) devices to start power capping operations using the stored power cap values. This technique ensures fail-safe when the power management service 402 fails to instruct the PDU or cancel the timer for any reason.

[0122] The techniques described above enable more lower-level devices to operate unconstrained by minimizing the number and / or frequency of capping the power consumption at these devices. Additionally, the peaks of downstream devices in a given device hierarchy can be allowed to exceed the budgeted power of the device while still ensuring that the maximum power capacity of the device (a value higher than the budgeted value) is not exceeded. This allows the consumption of lower-level devices to cause the consumption levels at higher-level devices to operate within a power buffer that is not used in conventional systems. The described systems and techniques allow for more efficient utilization of the power distribution components of system 400 and reduce waste while ensuring that power failures are avoided.

[0123] In some embodiments, the power management service 402 may select a set of power ceiling values determined to be most advantageous (e.g., based on any suitable combination such as at least in part on the maximum expected reduction and / or minimum estimated impact, etc.), or may receive from the VEPO orchestration service 302 and / or implement a specific set of power ceiling values provided by the power management service 402 (previously identified by the power management service 402 and provided to the VEPO orchestration service 302). In some embodiments, the identified specific power ceiling values may be identified at least in part based on maximizing power consumption reduction, identifying a reduction associated with a sufficient power ceiling, which reduction associated with the sufficient power ceiling is identified alone or in combination with other operations corresponding to reduction actions or levels associated with reducing the total power consumption value of the (one or more) devices below the current total power threshold.

[0124] Figure 6 is a flow diagram illustrating an example method 600 for managing excessive power consumption according to at least one embodiment. Method 600 may include more or fewer operations than those depicted or described with respect to Figure 6 These operations may be performed in any suitable order. Any suitable portion of the operations described in connection with the power management service 606 may be performed additionally or alternatively by Figure 3 the VEPO orchestration service 302 and / or the computing service 306.

[0125] Method 600 may begin at 610, where the PDU 604 may receive and / or obtain consumption data from the (one or more) devices 602 (e.g., by request). The (one or more) devices 602 may be Figure 3 examples of the (one or more) hosts 324 (e.g., multiple servers), and may be arranged within a common server rack (e.g., rack 319). The PDU 604 is Figure 2 an example of the rack PDU 208A and / or Figure 3 the rack PDU 320. Each of the (one or more) devices 602 may be a device to which the PDU 604 distributes power. The (one or more) PDUs 608 may be examples of other rack PDUs associated with the same row as the PDU 604, and thus from the same row PDU as the PDU 604 (e.g., Figure 2 the row PDU 206, corresponding to Figure 5The node 506) receives power. The consumption received from the PDU(s) 608 may correspond to the power consumption of the device(s) 612. In some embodiments, at least some portions of the PDU(s) 608 and the device(s) 612 correspond to a different row than the row corresponding to the device(s) 602. In some embodiments, any suitable number of consumption data instances received by the PDU 604 (referred to as "device consumption data") may be associated with a single device among the device(s) 602. The same may be true for the device consumption data received by the PDU(s) 608. In some embodiments, the PDU 604 and the PDU(s) 608 may aggregate and / or perform calculations based on the received device consumption data instances to generate rack consumption data corresponding to each respective PDU. For example, the PDU 605 may generate rack consumption data from the device consumption data provided by the device(s) 602. The rack consumption data may include the device consumption data instances received by the PDU 604 at 610. Similarly, the rack consumption data provided by the PDU(s) 608 may include the device consumption instances received from the device(s) 612. The rack consumption data generated by the PDU 604 and / or the PDU(s) 608 may be generated at least in part based on the power consumption of the PDU 604 and / or the PDU(s) 608, respectively.

[0126] At 614, the power management service 606 ( Figure 4 the power management service 402 of Figure 3 an example of the power management service 304 of Figure 2 an example of the PDU(s) 208 of Figure 3 may receive and / or obtain rack consumption data from any suitable combination of the PDU 604 and the PDU(s) 608 (e.g., via a request), but not necessarily simultaneously. In some embodiments, the power management service 606 may continuously receive / obtain rack consumption data from the PDU 604 and the PDU(s) 608. The PDU 604 and the PDU(s) 608 may likewise receive and / or obtain device consumption data from the device(s) 602 and 612 (e.g., from the corresponding power controllers of these devices, each power controller being Figure 3 an example of the BMC / ILOM 328 of

[0127] At 616, the power management service 606 may at least in part be based on another PDU (e.g., Figure 5The maximum and / or budgeted power associated with row-level devices not depicted herein, such as busbars) and consumption data received from PDU 604 and / or (one or more) PDU 608 to determine whether to calculate the power capping value for (one or more) devices 602. In some embodiments, the power management service 606 may access from the stored data the maximum and / or budgeted power associated with, for example, a row-level PDU (e.g., Figure 2 the row PDU 206 not depicted herein), or the power management service 606 may receive the maximum and / or budgeted amount from a PDU (row-level PDU) associated with the maximum and / or budgeted power at any appropriate time. The power management service 606 may aggregate the rack consumption data from PDU 604 and / or (one or more) PDU 608 to determine the cumulative power consumption value for row-level devices (e.g., the total power consumption of devices downstream of the row-level device within the previous time window, the change in the consumption rate from the perspective of the row-level device, the direction of the change in the consumption rate from the perspective of the row-level device, the time period required to initiate power capping at one or more devices 602 and / or 612, etc.). In some embodiments, the power management service 606 may generate additional cumulative power consumption data from at least a portion of the cumulative power consumption data values and / or using historical device consumption data (e.g., cumulative or individual consumption data). For example, the power management service 606 may calculate any appropriate combination of the following at least in part based on historical consumption data (e.g., historical device consumption data corresponding to (one or more) devices 602 and / or 512 and / or historical rack consumption data): the change in the consumption rate from the perspective of the row-level device, the direction of the change in the consumption rate from the perspective of the row-level device, etc.

[0128] Alternatively, in some embodiments, the power management service 606 may receive at 616 any appropriate data and / or instructions from the VEPO orchestration service 402 for identifying any appropriate portion of the reduction action (e.g., power capping only for low-priority hosts) and / or any appropriate constraints and / or applicable constraints for any appropriate devices. The power management service 606 may identify a subset of (one or more) hosts for which to calculate the power capping (e.g., in this example, only from the low-priority host candidates). For example, the operation of power capping only for low-priority hosts may cause the power management service 606 to identify multiple power caps for a subset of these low-priority hosts from all low-priority hosts (e.g., a subset of (one or more) hosts 324).

[0129] If the cumulative power consumption (e.g., the power consumption corresponding to the device(s) 602 and 612) exceeds the budgeted power amount associated with the row-level PDU, then the power management service 606 may determine that a power ceiling value should be calculated. If the cumulative power consumption does not exceed the budgeted amount associated with the row-level PDU, then the power management service 606 may determine that a power ceiling should not be calculated, and method 600 may end. Alternatively, since the cumulative power consumption of the downstream devices (e.g., the device(s) 602 and 612) exceeds the budgeted amount associated with the row-level PDU, the power management service may determine that a power ceiling should be calculated and may proceed to 618.

[0130] In some embodiments, the power management service 606 may additionally or alternatively determine that the power ceiling value should be calculated at 616 based at least in part on providing historical consumption data (e.g., device consumption data and / or rack consumption data corresponding to the device(s) 602 and / or 612) as input data to one or more machine learning models. Any suitable supervised or unsupervised machine learning algorithm may be used to train the machine learning model(s) to identify, from the historical consumption data provided as input, the likelihood that the consumption corresponding to the device associated with the historical consumption data will exceed the budgeted amount (e.g., the budgeted power amount allocated to the row-level device). The machine learning model(s) may be trained using a corresponding training data set that includes instances of historical consumption data. These machine learning model(s) training is discussed in more detail. Using the above techniques, the power management service 506 may identify that the power ceiling value should be calculated based on the cumulative device / rack consumption level exceeding the budgeted power associated with the row-level device and / or based on determining from the output of the machine learning model(s) that the cumulative device / rack consumption level may exceed the budgeted power associated with the row-level device. Determining that the device / rack consumption level may exceed the budgeted power of the row-level device may be determined based on receiving an output from the machine learning model(s) indicating a likelihood (e.g., likely / unlikely), or by comparing an output value (e.g., percentage, confidence value, etc.) indicating the likelihood of exceeding the budgeted power of the row-level device with a predefined threshold. An output value indicating a likelihood exceeding the predefined threshold may cause the power management service 606 to determine that power capping is reasonable and that a power ceiling value should be calculated. Figure 12

[0131] ​At 618, power management service 606 may calculate a power ceiling value for any suitable number of devices 602 and / or devices 612 (one or more). For example, power management service 606 may utilize device consumption data and / or rack consumption data corresponding to each of devices 602 and 612 (one or more). In some embodiments, power management service 606 may determine the difference between the cumulative power consumption of a row of devices (e.g., devices 602 (one or more) and any devices 612 (one or more) in the same row) (calculated by aggregating device and / or rack consumption data corresponding to devices 602 (one or more) and devices 612 (one or more)) and the budget power associated with the row-level devices. This difference may be used to identify the power consumption amount to be constrained at the devices associated with the row. Power management service 606 may determine a power ceiling value for any suitable combination of devices 602 (one or more) and devices 612 in the same row based on identifying a power ceiling value that, if enforced by the devices, would result in power consumption being reduced to a value less than the budget power associated with the row-level devices.

[0132] In some embodiments, power management service 606 may determine a specific power ceiling value for any suitable combination of devices 602 and 612 (one or more) corresponding to the same row based at least in part on consumption data and / or priority values associated with the devices and / or the workload associated with the power consumption of the devices. In some embodiments, these priority values may be provided as part of the device consumption data provided by the devices associated with the consumption data, or the priority values may be determined at least in part based on device type, workload type, etc. obtained from the consumption data or any other suitable source (e.g., separate data accessible to power management service 606). Using priorities associated with the devices or workloads respectively, power management service 606 may calculate the power ceiling value in a manner that favors power-capped devices associated with lower-priority workloads over power-capped devices associated with higher-priority workloads. In some embodiments, power management service 606 may calculate the power ceiling value based at least in part on power-capped devices that consume at a higher rate over other power-capped devices that consume at a lower rate (e.g., a group of highest-consuming devices). In some embodiments, power management service 606 may calculate the power ceiling value based at least in part on a combination of multiple factors, including the consumption value of each device and the priority associated with the device or the workload being executed by the device. Power management service 606 may identify power ceiling values for devices that avoid fully capping high-consuming devices and / or high-priority devices, or that cap these devices to a lesser extent compared to devices that consume less power and / or are associated with lower priorities.

[0133] At 620, the power management service 620 can calculate a timing value corresponding to the duration of a timer initialized and managed by a PDU (e.g., PDU 604). The timing value can be calculated based at least in part on any suitable combination of the rate of change of power consumption from the perspective of the row-level device, the direction of change of power consumption from the perspective of the row-level device, or the time at which power capping is performed at each device related to the power ceiling calculated at 618. For example, the power management service 620 can determine the timing value based on determining a relatively large increase in power consumption from the perspective of the row-level device, and the timing value is greater than the timing value for a relatively small increase in power consumption from the perspective of the row-level device. Thus, the more the power consumption rate increases, the smaller the timing value (corresponding to a shorter timer), while the smaller the increase in the power consumption rate, the larger the timing value may be (corresponding to a longer timer). Similarly, when the time at which power capping is performed at each device related to the power ceiling is relatively short with respect to the calculation of the first timing value, the calculated first timing value may be less than the calculated second timing value. Thus, the faster the device expecting power capping can enforce the ceiling, the shorter the timing value may be. In some embodiments, no timer is calculated for the power ceiling determined for the data provided or indicated via the VEPO orchestration service 302.

[0134] At 622, the calculated power ceiling value calculated at 618 and / or the timing value calculated at 620 can be transmitted to the PDU 604. Although not shown, the corresponding power ceiling value and / or timing value of any (one or more) devices 612 in the same row can be transmitted at 620. Although not shown, in some embodiments, the transmission at 622 can be based at least in part on an instruction received from the VEPO orchestration service 302 to implement / realize the power ceiling value calculated at 618.

[0135] At 624, when the timing value is provided at 622, the PDU 604 can use the timing value to initialize a timer having a duration corresponding to the timing value. As described above, in the case where the power management service 606 is instructed by the VEPO orchestration service 302, the timing value may not be provided at 622 and may not be stored at 624.

[0136] At 626, the PDU 604 can transmit the power ceiling value (if received at 622) to the (one or more) devices 602. At 628, the (one or more) devices 602 can store the received power ceiling value. In some embodiments, the power ceiling value provided at 626 can include an indicator indicating that enforcement does not start, or the indicator may not be provided, and the (one or more) devices 602 can default to avoiding starting power capping.

[0137] At 630, the power management service 606 can perform a higher level of analysis on device / rack consumption data received from any one or more devices 612 corresponding to a row different from the row corresponding to the one or more devices 602. As part of this process, the power management service 606 can identify unused power associated with other row-level devices. If there is unused power, then the power management service 606 can calculate a new set of power ceiling values at least in part based on consumption data corresponding to at least one other row of devices. In some embodiments, any suitable power ceiling values identified by the power management service 606 can be provided to the VEPO orchestration service 302 at any suitable time. As described above, this new set of power ceiling values may be beneficial because devices managed by another row-level device may not consume their budgeted power, leaving unused power at that row-level device. In some embodiments, the power management service 606 can be configured to determine whether it may be more advantageous to cap the devices of another row while allowing at least some of the one or more devices 602 (and possibly some of the one or more devices 612 corresponding to the same row as the one or more devices 602) to operate without constraint. The benefits of each power capping method can be calculated at least in part based on minimizing the number of devices to be power capped, minimizing the priority values associated with the devices to be power capped, maximizing the number of devices associated with a particular high-priority value that will not be power capped, etc. The power management service 606 can utilize a predefined scheme or set of rules to determine whether enforcing the power ceiling values it has determined for a row (e.g., corresponding to the power ceiling values transmitted at 622) is more or less advantageous than different power capping values identified based on processing consumption data from multiple rows. In some embodiments, the benefits of each set of power ceiling values identified by the power management service 606 can be accessed, and the most advantageous set of power ceilings can be selected by the power management service 606 (e.g., according to a set of rules, according to user input selecting a particular response level, etc.).

[0138] The power management service 606 can be configured to enforce a set of power ceiling values that are determined to be more (or most) favorable (or a set of power ceiling values indicated by the VEPO orchestration service 302) to ensure that one or more circuit breakers within the data center do not trip and / or the total power threshold is enforced. If it is determined that a previously provided set of power ceilings to the PDU 604 is less favorable than the set of power ceiling values calculated at 630, then the power management service 606 can immediately transmit data to the PDU 604 to cause the PDU 604 to transmit an indicator to the device(s) 602 to immediately begin power capping based on the previously distributed power ceilings. Alternatively, the power management service 606 can take no further action, which can allow the timer at the PDU 604 to expire. Any appropriate action performed by the power management service 606 based on the determination or identification made by the power management service 606 can alternatively be triggered by an instruction from the VEPO orchestration service 302.

[0139] If the power management service 606 determines that a new set of power ceilings is more (or most) favorable from a power management perspective, then it can transmit data to the power controller (e.g., Figure 3 the BMC / ILOM 322 of ) to cancel the timer. The previously distributed power ceilings can be deleted by an instruction from the power controller (e.g., the BMC / ILOM 322) (e.g., transmitted in response to canceling the timer), or those power ceilings can default to timeout based on a predefined time period. The power management service 606 can transmit the new power ceilings to a suitable PDU (which can include or exclude the same PDU to which the previous set of power ceilings was transmitted), indicating that the power ceilings will be immediately enforced by the corresponding devices. These PDUs can transmit these power ceilings along with an instruction to the receiving devices to immediately begin the power capping operation (e.g., including monitoring the consumption with respect to the power ceiling values, determining whether to cap based on the power ceiling values and the current consumption of the device, and based on the determination, operating the capping device or leaving the operation of the device unconstrained).

[0140] If, from a power management perspective, the power limit value calculated at 630 is determined to be more favorable than the power limit value calculated at 618 and is ultimately stored at (one or more) devices 602 at 628, then method 600 can proceed to 632, where the power limit value calculated at 630 can be transmitted to (one or more) PDUs 608 (e.g., any (one or more) PDUs 608 that manage devices related to the power limit value). In some embodiments, the power management service 606 can provide an indication that the power limit value transmitted at 632 will be immediately enforced. The operation described at 632 can be triggered by the power management service 606 based on a determination / identification made by the power management service 606 or in response to receiving an instruction from the VEPO orchestration service 302.

[0141] In response to receiving the power limit value and the indication at 632, the (one or more) PDUs 608 that distribute power to the devices related to these power limit values can transmit the power limit value and the indication to the devices related to these power limit values. At 636, the receiving devices of the (one or more) devices 612 can perform a power capping operation without delay based on the received indication. This includes 1) determining whether to limit / constrain the operation based on the current consumption of the device compared to the power limit value provided to the given device, and 2) limiting / constraining or avoiding limiting / constraining the power consumption of the device.

[0142] In some embodiments, at least partially based on identifying that the power limit value calculated at 630 is more (or most) favorable than the power limit value calculated at 618, the power management service 606 can transmit data to cancel the timer and / or the power limit value transmitted at 622. In some embodiments, the PDU 604 can be configured to cancel the timer at 640. In some embodiments, the PDU 604 can transmit at 642 such that the (one or more) devices 602 delete the data of the power limit value from the memory at 644.

[0143] At 646, in the case where a more favorable set of power ceiling values (as described above) is not found or the power management service 606 does not transmit a cancellation (as described in connection with 638 above), the PDU 604 may identify that the timer initiated at 624 has expired (e.g., the duration corresponding to the timing value provided at 622 has passed). Accordingly, the PDU 604 may transmit an indication to the device(s) 602 at 648 to instruct the device(s) 602 to start power capping based on the power ceiling value stored at the device(s) 602 at 628. Receiving the indication at 648 may cause the device(s) 602 to start power capping at 650 based on the power ceiling value stored at 628. In some embodiments, the VEPO orchestration service 302 may be configured to control the operations of 646 - 650, and the power management service 606 cannot perform the operations of 646 - 650 unless the VEPO orchestration service 302 instructs it to do so.

[0144] In some embodiments, if the PDU 604 identifies that the power consumption of the device(s) 602 has decreased, then the PDU 604 may be configured to cancel the timer at any appropriate time after the timer is initiated at 624. In some embodiments, this may cause the PDU 604 to transmit data to the device(s) 602 to cause the device(s) 602 to discard the previously stored power ceiling value from local memory.

[0145] In some embodiments, the power ceiling value enforced at any appropriate device may time out or may be replaced by a power ceiling value calculated by the power management service 606 at any appropriate time. In some embodiments, the power management service 606 may transmit a cancellation or replacement of the power ceiling value to any appropriate device via its corresponding rack PDU. If a cancellation of the power ceiling value is received, then the device may delete the previously stored power ceiling value, thereby allowing the device to resume operation without constraint. If a replacement power ceiling value is received, then the device may store the new power ceiling value and enforce the new power ceiling value immediately or when instructed by its rack PDU. In some embodiments, being instructed by its PDU to enforce a new power ceiling value may cause the device's device (e.g., the power controller 446) to replace the previously used power ceiling value for power capping with the new power ceiling value that the device is instructed to enforce.

[0146] Any suitable operation of method 600 can be continuously performed at any suitable time to manage over - consumption (e.g., the case where the consumption of a server corresponding to a row - level device exceeds the budgeted power of the row - level device), enabling more efficient use of previously unused power while avoiding power outages due to circuit breaker tripping. It should be recognized that the power management service 606 can perform similar operations for any suitable level of the power distribution hierarchy 500. Although an example of monitoring power consumption corresponding to a row - level device (e.g., a device represented by a level 3 node of the power distribution hierarchy 500) has been provided, similar operations can be performed for higher - level devices at any suitable level (e.g., components corresponding to any level from level 2 to level 5 of the power distribution hierarchy 500). As described above, any suitable function / operation of the power management service 606 can be triggered at least in part based on the determination / identification performed by the power management service 606, or any suitable function / operation of the power management service 606 can be triggered via an instruction from the VEPO orchestration service 302.

[0147] Figure 7 Illustrates an example architecture of a VEPO orchestration service 702 ( Figure 3 an example of the VEPO orchestration service 302) according to at least one embodiment, which is configured to orchestrate the management of the power consumption of a host according to a total power threshold. The VEPO orchestration service 702 can include an input / output processing manager 704, which can be configured to receive and / or transmit any suitable data between the VEPO orchestration service 702 and Figure 3 any other suitable device. As shown, the power management service 402 can include a demand manager 710, an impact identification manager 706, and an enforcement manager 708, but more or fewer computing components or sub - routines can be similarly used.

[0148] The demand manager 710 can be configured to monitor and / or modify the total power threshold associated with a physical environment (e.g., Figure 1 the data center 102, room 110A, row 108A, etc.). Monitoring can include monitoring Figure 3Any suitable combination of power data 310, environmental data 314, host / instance data 316, and / or operational data 318. Modifying the total power threshold can be performed, and the amount by which the total power threshold is modified can be determined at least in part based on a plurality of trigger events. These trigger events can include, but are not limited to: 1) receiving a request from a government entity (e.g., a local power authority) to reduce power indefinitely or for a period of time; 2) determining current and / or predicted environmental conditions (e.g., current ambient or external temperature, predicted ambient or external temperature); 3) determining the current or predicted total power consumption corresponding to the devices in the physical environment; 4) determining the current and / or predicted operating conditions of one or more temperature control units in the physical environment. The amount by which the total power threshold can or will be modified can be determined at least in part based on a plurality of factors, including but not limited to: the difference between the ambient temperature and the external temperature, the power consumption attributed to or associated with a particular component (e.g., a particular temperature control unit), the amount or percentage specified in a request received from a government entity, or the difference between the current or predicted total power consumption value and the current or predicted total power threshold.

[0149] The impact identification manager 706 can be configured to generate (either directly or by invoking the functionality of another service or process such as the compute service 306 and / or the power management service 304) one or more instances of an estimated impact corresponding to a response level and / or a reduction action corresponding to a response level. The VEPO orchestration service 702 can obtain configuration data 712 that specifies a plurality of VEPO levels and their corresponding set of one or more reduction actions.

[0150] Figure 8 is Table 800 according to at least one embodiment, which illustrates a set of example response levels and the corresponding set of reduction actions for each response level. The data in Table 800 can be provided differently, such as by mapping or being embedded within the program code of the VEPO orchestration service 702. Each response level in the set of response levels can be associated with a corresponding set of reduction actions that can be performed on a plurality of resources (e.g., hosts, instances, workloads, etc.) in the data center, and each corresponding set of reduction actions is associated with a different reduction in the total power consumption of the data center. In some embodiments, the plurality of response levels indicate increasing severity and / or which reduction actions to perform to reduce the total power consumption of the data center.

[0151] By way of example, Table 800 includes four VEPO response levels, but any suitable number of response levels can be used. The VEPO response level 1 can be associated with one reduction action. Specifically, the reduction action specifies that all vacant and / or idle hosts and hypervisors that can be evacuated (e.g., via migration) will be shut down (e.g., powered off).

[0152] In some embodiments, the VEPO response level 1 may include stopping or prohibiting the start or assignment of instances and / or workloads to affected hosts. In some embodiments, the VEPO response level 1 may include reduction actions, which include migrating instances and / or workloads from one host to another host, or from one instance to another instance (in the case of workload migration).

[0153] The VEPO response level 2 may be associated with multiple actions (such as any / all of the reduction actions discussed above in connection with the VEPO response level 1) and one or more additional reduction actions. For example, an additional reduction action corresponding to shutting down the hypervisor hosting the instances and / or workloads associated with customers of category 1 (e.g., free-tier customers, free-trial customers, etc.).

[0154] The VEPO response level 3 may be associated with multiple actions (such as any / all of the reduction actions associated with the VEPO response level 1 and / or 2) and one or more additional reduction actions. By way of example, the VEPO response level 3 may include additional reduction actions corresponding to shutting down the hypervisors for category 2 bare-metal instances (e.g., pay-as-you-go and / or enterprise BM) and hosting category 3 virtual machines (e.g., VMs associated with pay-as-you-go and enterprise categories) (e.g., after evacuating higher-priority VMs). In some embodiments, the additional reduction actions may specify that BMs and / or VMs associated with customers and / or cloud services of priority level 1 (e.g., critical customers and cloud provider services) should be excluded from consideration.

[0155] The VEPO response level 4 may be associated with multiple actions (such as any / all of the reduction actions associated with the VEPO response levels 1, 2, and / or 3) and one or more additional reduction actions. By way of example, the VEPO response level 4 may include additional reduction actions corresponding to shutting down all compute hosts in a customer enclave, except those that remain in a critical state and / or are considered critical for recovery.

[0156] Figure 8 The number of response levels and / or specific reduction actions depicted herein is not intended to limit the scope of the present disclosure. Any suitable number of response levels may be employed, with each response level corresponding to any suitable number of reduction actions. The reduction actions may include or exclude hosts, instances, workloads, and / or customers based on any suitable attributes associated therewith. For example, based on priority, category, status, and / or based on future recovery requirements. In some embodiments, the hosts, instances, and workloads considered candidates for applying these reduction actions may be based on a limited scope described in more detail Figure 9 in more detail.

[0157] AlthoughFigure 8 The response levels depicted in Figure 8 include some overlapping reduction actions, but this is not a requirement. In some embodiments, the reduction actions for each level can include a lesser or greater degree of overlap, including no overlap, resulting in completely unique actions associated with each level.

[0158] Figure 9 FIG. 900 is a schematic diagram illustrating an example scope of components affected by one or more response levels of a VEPO orchestration service (e.g., Figure 7 the VEPO orchestration service 702 of Figure 7 ) for orchestrating power consumption constraints, migrations, pauses, and / or power-off tasks. The schematic diagram illustrates a user domain 902 and a cloud service domain 904. The user domain 902 can include computing instances operating within a customer footprint (e.g., footprint 908) that run workloads (or any suitable workloads) corresponding to one or more services 906. The instances and / or workloads corresponding to one or more services 906 can be associated with one or more users 910.

[0159] In contrast, the cloud service domain 904 can include computing instances that run workloads corresponding to one or more services 912 (e.g., control and / or data plane services, such as those discussed in connection with Figures 14 - 18 service leasing) and / or resources (such as object storage devices, virtual cloud networking, etc.).

[0160] In some embodiments, although one or more hosts 324 can host any suitable combination of instances and / or workloads corresponding to one or more services 906 associated with a user (e.g., a customer) and / or one or more services 912 and / or resources associated with a cloud provider, only the hosts, instances, and / or workloads (e.g., computing resources) corresponding to the customer domain can be considered candidates to which reduction actions can be applied. In some embodiments, reduction actions can be completely (at least for some period of time) avoided for services and / or resources associated with the cloud provider because these services and / or resources are necessary for later recovery.

[0161] Now returning Figure 7 , in some embodiments, the functionality described in connection with the power management service 402 can be performed by one or more virtual machines implemented in a hosted computing environment (e.g., on one or more hosts 324, corresponding to Figure 1 any suitable number of servers 104 of Figure 1 ). The hosted computing environment can include one or more quickly provisioned and released computing resources, which can include computing, networking, and / or storage devices. The hosted computing environment can also be referred to as a cloud computing environment. Below, with respect to Figures 14 - 18Provide and discuss in more detail multiple example cloud computing environments.

[0162] The VEPO orchestration service 702 can be configured to manage power consumption corresponding to hosts / instances / workloads of the physical environment through functions provided by associated functions or other systems and / or services. Managing the power consumption value can include power capping, pausing workloads, instances, hosts, shutting down workloads / instances / hosts, migrating an instance from one host to another, migrating a workload from one instance and / or host to another instance and / or host, etc.

[0163] The impact identification manager 706 can be configured to utilize configuration data 712 corresponding to specifications of multiple response levels and their corresponding mitigation actions (e.g., Figure 8 table 800) to determine the estimated impact of applying a mitigation action of a given level to hosts / instances / workloads (collectively referred to as "resources") of the physical environment. The estimated impact of applying a given action and / or level can depend on the current conditions of the runtime resources. In some embodiments, identifying the estimated impact of a given mitigation action can include identifying a set of resources and / or customers to which the action would apply if implemented. Thus, the estimated impact is intended to refer to the scope or applicability of a given action or level. Identifying the scope and / or applicability of a given action or level can include identifying the specific resources and / or customers that would be affected by implementing the action or level and / or identifying any appropriate attributes associated with the applicable resources and / or customers. As a simple example, the impact identification manager 706 can identify that if the action of shutting down all idle hosts is taken, then X idle hosts among the (one or more) hosts 324 would be affected, and these (one or more) hosts are associated with a particular customer or a particular number of customers, and / or the (one or more) hosts and / or customers are associated with other attributes (such as category (e.g., free tier), priority (e.g., high priority), etc.). The mitigation actions can provide increasingly severe actions and / or increasingly broad impacts. As an example of increasingly severe actions, a lower level can specify an action of power capping a particular host associated with a particular attribute, while a higher level can specify completely shutting down the host. Examples of increasingly broad impacts can include a lower level that affects a small number of hosts (e.g., 2, 5, 10) corresponding to a single low-priority customer, while a higher level may affect a larger number of hosts (e.g., 100) corresponding to the same or more customers, and these hosts are associated with a wider range of priorities, etc. Thus, a broader impact refers to the quantity, scope, or range of applicability of a given action to hosts, instances, workloads, or customers.

[0164] The impact identification manager 706 can be configured to estimate the impact that one or more levels may experience and / or estimate a power reduction, at least in part, based on configuration data specifying levels and corresponding actions and any suitable combination of power data 310, account data 312, host / instance data, etc. The estimated impact can be identified for one or more response levels, one or more reduction actions corresponding to a given level, etc. (e.g., which (one or more) hosts, (one or more) instances, (one or more) workloads, (one or more) customers may be affected, which attributes are associated with the affected (one or more) hosts, (one or more) instances, (one or more) workloads, and / or customers, how many (one or more) hosts / (one or more) instances / (one or more) workloads / (one or more) customers are affected, etc.). In some embodiments, the estimated impact of a given action can be aggregated with the estimated impact of all actions for a given level to identify the estimated impact of the given level.

[0165] In some embodiments, the impact identification manager 706 can be used to estimate the impact of each level and / or estimate a power reduction. In some embodiments, determining the estimated power reduction can be an iterative process, including identifying applicable resources (e.g., hosts, instances, and / or workloads), determining the current power consumption of the applicable resources, estimating the reduction for each response if the action is implemented, and aggregating the individual estimated reductions of the individual responses to identify the total estimated power reduction corresponding to a given action. This process can be repeated for each action at a given level, and the final estimated power consumption reductions for each action can be aggregated to determine the total estimated power consumption reduction for the given level. The functionality of the impact identification manager 706 can be invoked by the enforcement manager 708.

[0166] In some embodiments, the demand manager 710 can invoke the functionality of the enforcement manager 708, at least in part, based on monitoring or determining a need to change the total power threshold. Monitoring can include monitoring Figure 3Any suitable combination of power data 310, environmental data 314, host / instance data 316, and / or operational data 318. The total power threshold can be modified, and the amount by which the total power threshold can be modified can be determined at least in part based on a plurality of trigger events. These trigger events can include, but are not limited to: 1) receiving a request from a government entity (e.g., a local power authority) to curtail power indefinitely or for a period of time; 2) determining current and / or predicted environmental conditions (e.g., current environmental or external temperature, predicted environmental or external temperature); 3) determining current or predicted total power consumption corresponding to devices in the physical environment; 4) determining current and / or predicted operating conditions of one or more temperature control units in the physical environment. The amount by which the total power threshold can or will be modified can be determined at least in part based on a plurality of factors, including but not limited to: the difference between the environmental temperature and the external temperature, the power consumption attributed to or associated with a particular component (e.g., a particular temperature control unit), the amount or percentage specified in a request received from a government entity, or the difference between the current or predicted total power consumption value and the current or predicted total power threshold.

[0167] In some embodiments, after determining the amount of change in the total power threshold, the demand manager 710 can invoke the functionality of the enforcement manager 708 to determine which response level is appropriate for a given change. As a non-limiting example, the enforcement manager 708 can incrementally invoke the functionality of the impact identification manager 706 (e.g., via a function call or otherwise) to determine the impact associated with a given level (e.g., Figure 8The estimated impact and / or estimated power reduction corresponding to the VEPO response level 1). The resulting estimated impact and / or estimated power reduction calculated by the impact identification manager can be returned to the enforcement manager 708. In some embodiments, the enforcement manager 708 can obtain and / or implement a predefined scenario to determine the appropriateness of selecting one response level over another. For example, the environment manager 708 can be configured to compare the estimated power reduction at a given level with the current total power threshold identified by the demand manager 710 to determine whether the estimated power consumption at the given level (e.g., VEPO response level 1) is sufficient (e.g., exceeds the change required to bring the current total power consumption (e.g., identified and provided by the power management service 402) down to below the total power threshold identified by the demand manager 710). If sufficient, then in some embodiments, the enforcement manager 708 can evaluate the estimated impact according to the predefined scenario to determine whether the estimated impact is sufficient and / or acceptable. As a non-limiting example, even if the estimated power reduction is sufficient to bring the current total power consumption below the current total power threshold, if the estimated impact exceeds the conditions of the predefined scenario (e.g., the number of affected hosts exceeds the threshold identified in the predefined scenario), then the enforcement manager 708 can reject the response level. If the estimated power consumption and / or estimated impact of the response level fails to meet the requirements / conditions of the predefined scenario, then the enforcement manager 708 can be configured to call the impact identification manager 706 to determine the estimated impact and / or estimated power reduction corresponding to a higher VEPO response level (e.g., VEPO response level 2).

[0168] The estimation of impact and / or power reduction can be invoked by the enforcement manager 708 and determined by the impact identification manager 706 one level at a time until the enforcement manager 708 finds the first level that meets the requirements / conditions of the predefined scenario regarding the estimated impact and / or estimated power reduction. The level that is determined to be the lowest level that meets all the impact conditions of the predefined scenario (e.g., the level that appears higher in Table 800) can be referred to as the "least impactful resource level". The level that is determined to be the lowest level that meets all the power reduction conditions of the predefined scenario and / or complies with (e.g., is below) the current total power threshold (e.g., the level that appears higher in Table 800) can be referred to as the "least reduced resource level". The level that is determined to be the lowest level among the response levels for which all the conditions regarding the estimated impact and estimated power reduction are met can be referred to as the "first sufficient response level". The enforcement manager 708 can be configured to select the response level, depending on the predefined scenario, that is the first to be identified as the least impactful, the least reduced, or the first sufficient.

[0169] In some embodiments, the enforcement manager 708 may cause the impact identification manager 706 to determine the estimated impact and / or estimated power reduction for each level (or some subset, such as the top five, the next five, etc.). The enforcement manager 708 may be configured to select a particular response level at least in part based on comparing the estimated impact and / or estimated power reduction for each level to determine the response level with the least impact (e.g., having a relatively minimal impact), the least reduction (e.g., resulting in the smallest amount of power reduction sufficient to drop the current total power consumption below the current total power threshold), or the least reduction and the least impact (e.g., at least in part based on a weighted algorithm).

[0170] The estimated impact and / or estimated reduction calculated by the impact identification manager 706 (possibly by invoking Figure 3 the functions of the computer service 306 and / or the power management service 304), and / or any appropriate data on which these estimates depend (such as current values and / or predicted values) may be presented via the interface 11 discussed in more detail below.

[0171] As discussed herein, any appropriate determination and / or identification by the demand manager 710, the impact identification manager 706, and / or the enforcement manager 708 (or any appropriate service or subroutine invoked thereby) may depend on current data, historical data, and / or predicted data. As a non-limiting example, the demand manager 710 may determine the amount of change needed and / or the amount by which the current total power threshold needs to be modified based on Figure 1 the current operating condition of the (one or more) temperature control units 112 or the predicted operating condition of these units over a period of time in the future. The predicted condition may be identified at least in part based on historical data (e.g., indicating a partial or complete failure or shutdown of one or more parts of the temperature control unit) and / or based on the output provided by a machine learning model (e.g., a model trained in the manner described in conjunction with Figure 12 ). In some embodiments, the amount of change in the total power threshold determined by the demand manager 710 may be based on a predefined formula and / or table. For example, a formula and / or table may be obtained from which the capacity associated with the temperature control unit may be determined and / or calculated. The table (if used) may indicate the power consumption attributed to or associated with the temperature control unit. Thus, a failure of the temperature control unit (e.g., partial and / or complete failure) may be associated with all or a proportional share of the power consumption attributed to or associated with the temperature control unit (e.g., the power consumption in the form of heat that the temperature control unit is configured to dispose of). Thus, the demand manager 710 may determine that a complete failure of the temperature control unit requires reducing the total power threshold by an amount equal to the total power consumption attributed to or associated with the temperature control unit specified in the table.

[0172] In some embodiments, the enforcement manager 708 may be configured to present, recommend, or automatically select a given response level from a possible set of response levels based at least in part on any suitable combination of an estimated impact and / or an estimated power reduction that may be experienced if a reduction action corresponding to a given level is implemented (e.g., effected). Thus, in some embodiments, the VEPO orchestration service 702 may recommend and / or select a particular level based on any suitable combination of: 1) a determined level that, if implemented, would likely result in a sufficient but not excessive reduction in power consumption at the resources of the physical environment (e.g., the minimum power consumption reduction sufficient to bring the current power consumption value below the current total power threshold), 2) a determined level that includes a least restrictive set of actions, or 3) a determined level or action having the least impact (e.g., a set of affected resources and / or customers with the fewest resources / customers likely to be affected, having a priority or category indicating the lowest degree of collective importance, etc.). Determining the least restrictive level or action, or the level or action having the least impact, may be determined without regard to the estimated power consumption reduction that may be experienced by implementing the level and / or action, or determining the least restrictive level / action and / or the level / action having the least impact may be determined from one or more levels that, if implemented under given current conditions, are estimated to result in a sufficient reduction in power consumption to bring the power consumption value below an aggregated value of the current total power threshold. In some embodiments, the enforcement manager 708 may cause any suitable data corresponding to the current, predicted, or estimated power consumption and / or impact to be presented on Figure 11 the interface. In some embodiments, the selection of a particular level to be effected may be driven by user input provided on a user interface. Thus, in some embodiments, the enforcement manager 708 may refrain from performing operations to effect a reduction action for a level until the user has identified and / or confirmed, via user input provided on the user interface, the level to be effected.

[0173] In some embodiments, performing these operations may include instructing the power management service 402 and / or the computing service 306 to perform operations. For example, the enforcement manager 708 may provide the power management service 402 with identifiers of affected resources (e.g., hosts, instances, workloads, etc.) and estimated power reduction and / or reduction action data specifying the actions to be taken (e.g., resources specific to power capping, power capping all low-priority hosts / instances / workloads, power capping a specific or all resources meeting a set of specific attributes by a specific amount (e.g., 10%, at least x amount, etc.)). As described above, this may be configured to enact / implement corresponding power caps as directed by the enforcement manager 708. Similarly, the computing service 306 may be provided with identifiers of affected resources (e.g., hosts, instances, workloads, etc.) and reduction action data specifying the actions for the computing service 306 to take (e.g., shutting down all idle hosts, shutting down these specific hosts, pausing all low-priority workloads, migrating and / or compressing all low-priority workloads to maximize the number of idle / vacant hosts, etc.). As described above, this may be configured to enact / implement corresponding power caps as directed by the enforcement manager 708.

[0174] The demand manager 710 may also be configured to identify that a previous trigger that led to the achievement of a response level no longer occurs and may modify the total power threshold based on the identification that the trigger condition no longer occurs. For example, the demand manager 710 (e.g., via monitoring the operation data 318) may identify that a temperature control unit that previously failed is no longer operating. Accordingly, the demand manager 710 may modify (e.g., increase) the total power threshold by the amount attributed to the temperature control unit when it was fully operational. In accordance with this change, the enforcement manager 706 may be configured to perform any appropriate operations to reverse, to the extent possible, any reduction actions taken previously in response to any response level selected in response to the original trigger condition. If power caps were used, then the enforcement manager 706 may instruct the power management service 402 to remove those previously applied power caps. If migrations of workloads and / or instances were performed, then the enforcement manager 706 may instruct the computing service 306 to attempt to migrate those resources back to the original hosts and / or instances hosting the instance and / or workload, respectively. If resources were paused and / or shut down, then the enforcement manager 706 may instruct the computing service 306 to perform operations to resume the instance and / or workload and / or send a signal to the shut-down host to start. In this way, the enforcement manager 706 may perform any appropriate operations to recover from reduction actions taken based on a previously selected given response level.

[0175] Figure 10 is a block diagram illustrating an example use case of applying multiple response levels based on the current conditions of a set of hosts according to at least one embodiment.Figure 10 The example provided in Figure 8 discusses that the VEPO response levels and corresponding actions are Figure 7 the current configuration data used by the VEPO orchestration service 702 of

[0176] · Servers 1004A and 1004C are vacant.

[0177] · Server 1004D is hosting resources (e.g., instances) associated with customers at category level 1

[0178] (e.g., instances).

[0179] · Server 1004I is hosting BM instances associated with customers associated with priority level 2.

[0180] · Server 1004J is hosting the BM of customers associated with priority level 3.

[0181] · Server 1004K is hosting the VMs of customers associated with priority level 1.

[0182] · Server 1004B is hosting one or more instances that are considered critical for recovery. In some embodiments, these instances may be associated with the service(s) 912 provided by the cloud provider.

[0183] As described above, the VEPO orchestration service 702 can identify the estimated impact and / or estimated power reduction corresponding to each response level step by step or in parallel. The VEPO orchestration service 702 can identify the responses applicable to a given level (e.g., the reduction actions corresponding to a given response level) (e.g., through the functions provided by the power management service 304 and / or the computing service 306). For example, in the ongoing example, the VEPO orchestration service 702 (through the power management service 304) can identify that servers 1004A and 1004C are a set of resources for the redundant action application of VEPO response level 1 based on (e.g., from non-cloud-provider-based resources, such as resources associated with the user territory (such as user territory 902)) and identify that VEPO response level 1 is estimated to potentially affect 2 servers, namely server 1004A and server 1004C. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which the estimated power reduction at a given level (if formulated / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 10% reduction will have no impact on the customers. The idle BM capacity is reduced to 0%, the idle CM capacity is severely reduced, etc.). Any appropriate part of this information can be presented on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0184] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources (e.g., from non-cloud-provider-based resources such as resources associated with a user's premises (such as user premises 902)) on which server 1004D is hosting redundant action applications at the VEPO response level 3, also identify that the VEPO response level 2, including scaled actions at the VEPO response level 1, is estimated to potentially affect servers 1004A and 1004C. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at the VEPO response level 2 (if enacted / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 20% reduction has a minimal impact on the customer. Up to 400 low-priority VMs are affected, etc.). Any appropriate portion of this information can be presented on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0185] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources (e.g., from non-cloud-provider-based resources such as resources associated with a user's premises (such as user premises 902)) on which the BM hosted by server 1004I and the VMs hosted by server 1004J are resources for redundant action applications at the VEPO response level 3, also identify that the VEPO response level 3, including scaled actions at the VEPO response levels 1 and 2, is estimated to potentially affect servers 1004A, 1004C, and 1004D. Resources hosted by server 1004K and / or server 1004K can be excluded based on meeting exclusions associated with the evaluated response level. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which an estimated power reduction at the VEPO response level 3 (if enacted / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a 40% reduction has a moderate impact. Up to 200 category 1 or 2 BMs / VMs are affected, etc.). Any appropriate portion of this information can be displayed on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0186] The VEPO orchestration service 702 (via the power management service 304) can, in addition to identifying resources hosted by the server 1004I, such as BM and VM hosted by the server 1004J, which are redundant action applications at the VEPO response level 4 based on the identification (e.g., from non-cloud-provider-based resources, such as resources associated with a user territory (such as the user territory 902)), also identify that the VEPO response 4 estimate may affect all servers other than the server 1004B (e.g., a group of servers including the servers 1004A and 1004C - 1004L) based on the reduced actions including VEPO response levels 1 - 3. In some embodiments, the VEPO orchestration service 702 (e.g., the impact identification manager 706) can determine the extent to which the estimated power reduction at the VEPO response level 4 (if formulated / implemented) will affect the total power consumption of the resources in the physical environment (e.g., a reduction of 80% has a serious impact, regional downtime, etc.). Any appropriate portion of this information can be displayed on the user interface 1100, where similar data associated with other VEPO response levels may or may not be presented.

[0187] Figure 11 is a schematic diagram of an example user interface 1100 according to at least one embodiment. The user interface 1100 can include any appropriate number and type of graphical interface elements (e.g., dropdown boxes 1102 - 1108) to respectively select and / or view VEPO data corresponding to a given region, availability domain (AD), building, and / or room. Other interface elements, such as radio buttons and / or edit boxes, can be used. The VEPO data presented on the user interface 1100 can correspond to the region, AD, building, and room selected via the graphical interface elements 1102 - 1108 (also referred to as "options" 1102 - 1108). As depicted, the VEPO data 1109 is displayed according to the values selected via the options 1102 - 1108.

[0188] As discussed above, any appropriate information used by the VEPO orchestration service 702, the power management service 304, and / or the computing service 306 can be presented via the user interface 1100. As a non-limiting example, any appropriate combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318 can be presented in the user interface 1100.

[0189] The specific content of the VEPO data 1109 displayed on the user interface 1100 may vary, but as depicted, the VEPO data 1109 includes VEPO level data (e.g., presented in column 1110 associated with Figure 8The identifier corresponding to the VEPO response level), the current power consumption value (e.g., the current power consumption value presented in column 1112), the estimated reduction associated with each VEPO level (e.g., the estimated VEPO reduction presented in column 1114), and the estimated impact associated with each VEPO level (e.g., the estimated VEPO impact presented in column 1116).

[0190] The current total power threshold can be displayed at 1118 (e.g., the current total power threshold determined by Figure 7 the demand engine 710). The current power reduction required to meet the current total power threshold can be presented at 1120. This value (e.g., indicating a need to reduce by 5 kW) can be calculated at least in part based on identifying the difference between the current total power threshold (e.g., 11 kW) and the current power consumption presented at 1112 (e.g., 16 kW) (e.g., by the demand engine 710).

[0191] The user interface 1100 can be configured to present estimated impact data corresponding to each VEPO response level (referred to herein as "VEPO level" for brevity) within column 1116. In some embodiments, a region corresponding to a row of column 1116 can be selected to present an expanded view of the estimated impact data corresponding to a particular VEPO level. Similarly, a region corresponding to a row of column 1114 can be selected to present an expanded view of the estimated VEPO reduction corresponding to a particular VEPO level.

[0192] The user interface 1100 can include an indicator 1122 that indicates the recommended VEPO response level (as depicted, the system recommends VEPO level 2). As discussed above, the recommended VEPO level can be identified by the power orchestration service 702 based on the above factors. In some embodiments, the recommended VEPO level (e.g., VEPO level 2) can be selected based on identifying that VEPO level 2 is associated with the minimum estimated VEPO reduction amount to meet the required reduction indicated at 1120. In some embodiments, the VEPO level 2 can be recommended additionally or alternatively at least in part based on identifying that the estimated impact of applying VEPO level 2 provides the minimum impact relative to the estimated impact of VEPO level 104.

[0193] In some embodiments, a region corresponding to each row of VEPO data 1109 can be selected to select a given VEPO response level. Once selected, an additional menu or options can be provided to the user for indicating that the system should continue with the selected VEPO response according to the reduction action associated with that level.

[0194] Figure 12The figure illustrates a flow depicting an example method 1200 for training one or more machine learning models according to at least one embodiment. In some embodiments, any suitable machine learning algorithm (e.g., supervised, unsupervised, etc.) and any suitable number of training data sets (e.g., training data 1208) can be used (e.g., by the Figure 3 power management service 304 of Figure 3 the power orchestration service 302 of Figure 3 the computing service 306, or different devices or systems) to train the model(s) 1202. A supervised machine learning algorithm refers to a machine learning task that includes learning an inference function that maps an input to an output based on a labeled training data set of known example input / output pairs. An unsupervised machine learning algorithm refers to a set of algorithms for analyzing and clustering unlabeled data sets (e.g., unlabeled data 1210). These algorithms are configured to identify patterns or data groupings without human intervention. In some embodiments, any suitable number of the model(s) 1202 can be trained during the training phase 1204.

[0195] According to at least one embodiment, at least one of the model(s) can be trained to identify the likelihood or confidence that the total power consumption of a downstream device (e.g., the Figure 3 (one or more) hosts 324 of Figure 2 ) will exceed a budget threshold corresponding to an upstream device (e.g., a row-level device such as the Figure 3 (one or more) hosts 324 of Figure 1 (one or more) servers 104, etc.) through which power is distributed to any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. The training data 1208 used to train one or more of the model(s) 1202 can include any suitable combination of individual and / or total power consumption values (current or historical) of power data 310 corresponding to one or more hosts (e.g., the Figure 3 (one or more) hosts 324 of Figure 1 (one or more) servers 104, etc.). In some embodiments, the training data 1208 used to train these specific model(s) 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the model(s) can be trained to identify a prediction corresponding to the total power consumption of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 can include current or historical power caps applied to the corresponding hosts, and the output provided by such model(s) 1208 can identify the predicted power cap values of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104.

[0196] According to at least one embodiment, at least one of the (one or more) models can be trained to identify a predicted workload for any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. Training data 1208 for training one or more of the (one or more) models 1202 to identify the predicted workload can include any suitable combination of current and / or historical data of host / instance data 316 corresponding to one or more hosts (e.g., Figure 3 the (one or more) hosts 324 of Figure 1 the (one or more) servers 104, etc.). In some embodiments, the training data 1208 for training these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more models can be trained to identify the likelihood or confidence of assigning a predicted set of workloads to any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104.

[0197] According to at least one embodiment, at least one of the (one or more) models can be trained to identify a predicted partial or complete failure of one or more temperature control units. Training data 1208 for training one or more of the (one or more) models 1202 to identify such failures can include current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current or past external temperatures occurring relative to a region outside the physical environment (e.g., the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, and any suitable combination of current and / or historical failures experienced by one or more temperature control units (e.g., Figure 1 the (one or more) temperature control systems 112 of ). In some embodiments, the training data 1208 for training these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more models can be trained to identify the likelihood or confidence that a specific failure (partial or complete) of the (one or more) temperature control systems 112 will occur during a future time period. In some embodiments, one or more models can be trained to identify the degree of failure of the (one or more) temperature control systems 112 (e.g., 80% failure, 100% failure, etc.).

[0198] According to at least one embodiment, at least one of the (one or more) models can be trained to provide an estimated total power reduction required due to a partial and / or complete failure of one or more temperature control units (e.g., the (one or more) temperature control systems 112). The training data 1208 used to train one or more of the (one or more) models 1202 to identify the required estimated total power reduction can include any suitable combination of the following: current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current and / or historical external temperatures that currently or previously occurred relative to a region outside the physical environment (e.g., in the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, current and / or historical failures experienced by one or more temperature control units (e.g., Figure 1 the (one or more) temperature control systems 112), current and / or historical total power thresholds, and / or current and / or historical power consumption levels of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 used to train these (one or more) specific models 1202 can include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, one or more of the (one or more) models can be trained to identify the likelihood or confidence that an estimated total power reduction will be required during a future time period.

[0199] According to at least one embodiment, at least one of the (one or more) models can be trained to predict future temperatures (e.g., ambient temperature and / or external temperature). The training data 1208 used to train one or more of the (one or more) models 1202 to identify future temperatures can include any suitable combination of the following: current and / or historical data of environmental data 314 and / or operation data 318 corresponding to the physical location where the (one or more) hosts 324 and / or the (one or more) servers 104 are located, current and / or historical external temperatures that currently or previously occurred relative to a region outside the physical environment (e.g., in the geographical region where the physical environment is located), current and / or historical environmental locations associated with the physical location, weather forecasts corresponding to a future time period, one or more temperature control units (e.g., Figure 1The current and / or historical faults experienced by the (one or more) temperature control systems 112, the current and / or historical total power thresholds, and / or the current and / or historical power consumption levels of any suitable combination of the (one or more) hosts 324 and / or the (one or more) servers 104. In some embodiments, the training data 1208 for training these (one or more) specific models 1202 may include any suitable combination of power data 310, account data 312, environmental data 314, host / instance data 316, and / or operation data 318. In some embodiments, the (one or more) models may be trained to identify the likelihood or confidence of experiencing a predicted temperature during a future time period.

[0200] Generally, the (one or more) models 1202 may include any suitable number of models. The (one or more) models 1202 may be individually trained to identify the likelihood or confidence or accuracy of the output provided by the (one or more) models 1202 from the training data 1208 discussed above. In some embodiments, the (one or more) models 1202 may be configured to determine values corresponding to the examples provided above, and the likelihood and / or confidence values may indicate the likelihood or confidence that the output provided by the (one or more) models is accurate.

[0201] Generally, the (one or more) models 1202 may be trained during the training phase 1204 using supervised learning algorithms and labeled data 1206 to identify the output described in the examples above. The likelihood value may be a binary indicator, a percentage, a confidence value, etc., which indicates the degree of likelihood. The likelihood value may be a binary indicator that indicates whether a particular budget amount is likely or unlikely to be breached, or the likelihood value may indicate the likelihood and / or confidence that a predicted output will occur in the future. The labeled data 1206 may be any suitable portion of the potential training data (e.g., training data 1208) that can be used to train the (one or more) models to produce the output described above. The labeled data 1206 may include any suitable number of examples of current and / or historical data corresponding to any suitable number of devices in the physical environment (e.g., data center 102, the (one or more) servers 104, the (one or more) temperature control systems 112, Figure 2 PDUs, etc.). In some embodiments, the labeled data 1206 may include labels identifying known likelihoods and / or actual values. Using the labeled data 1206, a model (e.g., an inference function) that maps inputs (e.g., one or more instances of training data) to outputs (e.g., predicted values or the likelihood / confidence values that the predicted value or output will occur during a future time period) can be learned. In some embodiments, Figure 3Any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306, or a separate service or system, can train any suitable combination of the (one or more) models 1202. In some embodiments, any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the computing service 306 can obtain a pre-trained version of the (one or more) models 1202.

[0202] (One or more) models 1202 and the various types of models discussed above can include any suitable number of models that are trained using unsupervised learning techniques to identify likelihoods / confidences and / or predictions corresponding to the examples provided above. Unsupervised machine learning algorithms are configured to learn patterns from unlabeled data. In some embodiments, the training phase 1204 can utilize unsupervised machine learning algorithms to generate one or more of the (one or more) models 1202. For example, the training data 1208 can include unlabeled data 1210 (e.g., any suitable combination of current and / or historical values corresponding to power data 310, account data 12, environmental data 314, host / instance data 316, operational data 318, etc.). The unlabeled data 1210 can be used with an unsupervised learning algorithm to divide the entries of the unlabeled data 1210 into groups. The unsupervised learning algorithm can be configured to group similar entries together into common groups. Examples of unsupervised learning algorithms can include clustering methods such as k-means clustering, DBScan, etc. In some embodiments, the unlabeled data 1210 can be clustered with the labeled data 1206 such that the unlabeled instances of a given group can be assigned the same label as the other labeled instances within that group.

[0203] In some embodiments, any suitable portion of the training data 1208 can be used to train the model(s) 1202 during the training phase 1204. For example, 70% of the labeled data 1206 and / or the unlabeled data 1210 can be used to train the model(s) 1202. Once trained, or at any suitable time, the model(s) 1202 can be evaluated to assess its quality (e.g., the accuracy of the output(s) 1212 relative to the labels corresponding to the labeled data 1206). By way of example, a portion of the examples of the labeled data 1206 and / or the unlabeled data 1210 can be used as input to the model(s) 1202 to generate the output(s) 1212. By way of example, an example of the labeled data 1206 can be provided as input, and the corresponding output (e.g., the output(s) 1212) can be compared to the label known to be associated with that example. If a portion of the output (e.g., a label) matches the example label, then that portion of the output can be considered accurate. Any suitable number of labeled examples can be used, and the number of accurate labels can be compared to the total number of examples provided (and / or the total number of previously identified labels) to determine an accuracy value for the given model, which quantifies how accurate the model is. For example, if 90 out of 100 input examples generate output labels that match the previously known example labels, the model being evaluated can be determined to be 90% accurate.

[0204] In some embodiments, when the model(s) 1202 is used for subsequent inputs, the subsequent output(s) generated by the model(s) 1202 can be added to the corresponding input and used to retrain and / or update the model(s) 1202 at 1216. In some embodiments, examples may not be used to retrain or update the model until the feedback process 1214 is performed. In the feedback process 1214, an example (e.g., an example including one or more historical consumption data instances corresponding to one or more devices and / or racks) and the corresponding output generated by one of the model(s) 1202 for the example are presented to the user, and the user identifies whether the generated output (e.g., a quantity and / or a likelihood confidence value) is correct for the given example.

[0205] Figure 12 The training process depicted (e.g., method 1200) can be performed any suitable number of times at any suitable interval and / or according to any suitable schedule such that the accuracy of the model(s) 1202 improves over time.

[0206] In some embodiments, any suitable number and / or combination of (one or more) models 1202 can be used to determine the output. In some embodiments, the power management service 402 can utilize any suitable combination of outputs provided by (one or more) models 1202 to determine whether the budget threshold of a given component (e.g., row-level PDU) is likely to be breached (and / or likely to be breached by a certain amount). Thus, in some embodiments, the power management service 402 can utilize models trained using any suitable combination of supervised and unsupervised learning algorithms.

[0207] Figure 13 is a block diagram illustrating an example method 1300 for managing total power consumption based on a total power threshold according to at least one embodiment. Method 1300 can be performed by Figure 3 one or more components of the power orchestration system 300 or in combination with Figure 4 and Figure 7 its sub-components discussed. By way of example. Method 1300 can be performed at least in part by Figure 3 any suitable combination of the VEPO orchestration service 302, the power management service 304, and / or the compute service 306 of Figure 13 The operations of method 1300 can be performed in any suitable order. Method 1300 can include more or fewer operations than

[0208] Method 1300 can begin at 1302, where the current value of the total power threshold for a plurality of hosts can be identified (e.g., by Figure 7 the demand manager 710 of

[0209] At 1304, the current value of the total power threshold can be used during a first time period (e.g., by Figure 7 the enforcement manager 708 of Figure 3 utilizing the power management service 304 and / or the compute service 306 of

[0210] At 1306, a trigger event indicating a need to modify the total power threshold can be detected (e.g., by Figure 7 the enforcement manager 708, by the demand manager 710, etc.). In some embodiments, detecting the trigger event can include monitoring a temperature control system associated with the plurality of hosts (e.g., Figure 1The temperature control system 112A) and determine that the temperature control system is operating at a reduced capacity. As another example, detecting a trigger event may include at least one of the following: 1) receiving a curtailment request from a power authority from the utility grid, the curtailment request indicating the degree to which power consumption in the computer system is to be curtailed, 2) identifying a degradation or failure of the temperature control system of a physical environment including multiple hosts, or 3) identifying an increase in external temperature. In some embodiments, detecting a trigger event may include at least partially based on an external temperature associated with the physical environment in which the multiple hosts are located (e.g., by Figure 7 The enforcement manager 708) predict a failure of the temperature control system associated with the multiple hosts. In some embodiments, predicting a failure of the temperature control system associated with the multiple hosts is at least partially based on identifying that the external temperature exceeds a temperature threshold associated with the physical environment in which the multiple hosts are located. In some embodiments, predicting a failure of the temperature control system associated with the multiple hosts is also at least partially based on identifying that a redundant temperature control system corresponding to the temperature control system is associated with a second failure.

[0211] In some embodiments, the trigger event is associated with monitoring an external temperature outside the physical environment in which the multiple hosts are located. In some embodiments, the difference between the current ambient temperature of the physical environment in which the multiple hosts are located and the external temperature outside the physical environment may be determined from environmental data (e.g., Figure 3 The environmental data 314). In some embodiments, the environment includes an association between the difference and a corresponding power consumption value. In some embodiments, a new value of the total power threshold is determined at least partially based on the corresponding power consumption value associated with the difference between the current ambient temperature and the external temperature.

[0212] At 1308, a new value of the total power threshold may be determined (e.g., by the demand manager 710) based on the trigger event. In some embodiments, the new value may be determined by identifying a table specifying corresponding power consumption values applicable to each of the multiple temperature control systems associated with the multiple hosts. In some embodiments, the multiple temperature control systems include a temperature control system operating at a reduced capacity. In some embodiments, the new value may be determined by identifying from the table the power consumption value corresponding to the temperature control system operating at a reduced capacity. In some embodiments, a new value of the total power threshold is identified at least partially based on the current power consumption value being less than the power consumption amount associated with the temperature control system operating at a reduced capacity.

[0213] As another example, a machine learning model may be obtained (e.g., Figure 12One or more of the (one or more) models 1202 in (e.g., by the demand manager 710) determine a new value. In some embodiments, the machine learning model was previously trained to identify a corresponding power consumption value applicable to a corresponding temperature control system from input data including an identifier of the corresponding temperature control system. The new value can be determined by providing, as input, the corresponding identifier (or any suitable input data) associated with the temperature control system operating at a reduced capacity to the machine learning model. The new value can be determined by receiving, as output of the machine learning model, the power consumption value applicable to the temperature control system. In some embodiments, a new total power consumption is determined at least in part based on the power consumption value applicable to the temperature control system. In some embodiments, the machine learning model is trained at least in part based on a training dataset including historical power consumption values and historical metadata associated with one or more temperature control systems.

[0214] At 1310, during a second time period (e.g., by Figure 7 the enforcement manager 708) the new value of the total power threshold can be used to manage the total power consumption of multiple hosts. In some embodiments, using the new value of the total power threshold to manage the total power consumption of multiple hosts includes at least one of the following: 1) enforcing a power ceiling on a host; 2) migrating a workload from a host; or 3) shutting down or pausing a host. In some embodiments, using the new value of the total power threshold to manage the total power consumption of multiple hosts during a second time period includes at least one of the following: 1) determining a difference between a current value of the total power consumption of multiple hosts during the second time period and a current value of the total power threshold of multiple hosts during the second time period, 2) determining that a first value of power reduction resulting from a first response level is greater than the difference, or 3) selecting the first response level based at least on determining that the first value of power reduction resulting from the first response level is greater than the difference.

[0215] It should be appreciated that Figure 13 the values and / or trigger events can (e.g., by the enforcement manager 708, the demand manager 710, etc.) be identified and / or detected at least in part based on a machine learning model (e.g., Figure 12 one or more of the (one or more) models 1202) that has previously been trained using a supervised learning algorithm and a training dataset including any suitable combination of the following: historical external temperature corresponding to multiple hosts, historical ambient temperature, historical total power threshold, and historical current power consumption values.

[0216] Although Figure 13Although not shown in the figure, in some embodiments, detecting a trigger event may include detecting a fault in a temperature control system associated with multiple hosts. In some embodiments, it may be detected that the temperature control system is operable. An updated value of the total power threshold may be determined at least in part based on determining that the temperature control system is operable. During a third time period, the updated value of the total power threshold may be used to manage the total power consumption of the multiple hosts. In some embodiments, one or more recovery actions may be performed at least in part based on the updated value of the total power threshold. The one or more recovery actions may include at least one of the following: 1) stopping power capping of a host among the multiple hosts, 2) migrating a workload back to the host, or 3) restarting a previously shut down or paused host.

[0217] Although Figure 13 Although not shown in the figure, in some embodiments, method 1300 may include any suitable combination of the following: 1) identifying, e.g., by enforcement manager 708, multiple response levels that specify the applicability of a set of corresponding reduction actions to multiple hosts, and 2) selecting, e.g., by enforcement manager 708, using user input, etc., a first response level among the multiple response levels at least in part based on a first power reduction estimate of applying a first set of reduction actions to a first host among the multiple hosts. In some embodiments, the first response level among the multiple response levels specifies the applicability of the first set of reduction actions to the multiple hosts such that a first reduction action among the first set of reduction actions is applicable to the first host among the multiple hosts.

[0218] Figures 14 - 18 depicts multiple example environments that may be hosted by Figure 3 one or more of the hosts 324. Figures 14 - 18 The environment depicted in the figure depicts a cloud computing, multi-tenant environment. As described above, cloud computing and / or multi-tenant environments and other environments may benefit from leveraging the power management techniques disclosed herein. These techniques enable a data center including components that implement an environment described in conjunction with Figures 1 - 4 and Figure 7 etc. to utilize the power resources of the data center more efficiently than conventional techniques that result in a large amount of unused power.

[0219] IaaS Infrastructure Example

[0220] Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also provide various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Thus, since these services may be policy-driven, IaaS users can be able to implement policies to drive load balancing to maintain the availability and performance of applications.

[0221] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create a virtual machine (VM), install an operating system (OS) on each VM, deploy middleware such as a database, create buckets for workloads and backups, and even install enterprise software into that VM. Then, the customer can use the provider's services to perform various functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, etc.

[0222] In most cases, the cloud computing model will require the involvement of a cloud provider. The cloud provider can be but is not necessarily a third-party service that specifically provides (e.g., provisions, rents, sells) IaaS. An entity may also choose to deploy a private cloud and thus become its own infrastructure service provider.

[0223] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It can also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider and is below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Thus, the customer can be responsible for handling the (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc. that can be launched on demand).

[0224] In some examples, IaaS provisioning can refer to obtaining computers or virtual hosts for use and even installing the required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0225] In some cases, there are two different challenges in IaaS provisioning. First, there is an initial challenge in provisioning the initial set of infrastructure before anything is running. Second, once everything has been provisioned, there is a challenge in evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two challenges can be addressed by enabling the configuration of the infrastructure to be defined in a declarative manner. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, a workflow for creating and / or managing the different components described in the configuration files can be generated.

[0226] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs) (e.g., a potentially on-demand pool of configurable and / or shareable computing resources), also known as the core network. In some examples, there can also be one or more inbound / outbound traffic group rules that are provisioned to define how inbound and / or outbound traffic will be set up for the network, and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., can also be provisioned. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve incrementally.

[0227] In some cases, continuous deployment techniques can be employed to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the techniques described can enable infrastructure management within these environments. In some examples, a service team can write code that is intended to be deployed to one or more but typically many different production environments (e.g., across various different geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0228] Figure 14 FIG. 1400 is a block diagram illustrating an example schema of an IaaS architecture according to at least one embodiment. A service operator 1402 can be communicatively coupled to a secure host lease 1404, which can include a virtual cloud network (VCN) 1406 and a secure host subnet 1408. In some examples, the service operator 1402 can use one or more client computing devices, which can be portable handheld devices (e.g., cellular phones, a computing tablet, personal digital assistant (PDA)), or wearable device (e.g., Google Head-Mounted Display), running software such as Microsoft Windows and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and enabling Internet, email, Short Message Service (SMS), or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, for example, including personal computers and / or laptop computers running various versions of Microsoft Apple and / or Linux operating systems. The client computing device can be a workstation computer running various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems such as, for example, Google Chrome OS). Alternatively, or additionally, the client computing device can be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a gesture input device) and / or a personal messaging device, which can communicate via a network that can access VCN 1406 and / or the Internet.

[0229] VCN 1406 can include a Local Peer Gateway (LPG) 1410, which can be communicatively coupled to a Secure Shell (SSH) VCN 1412 via the LPG 1410 included in the SSH VCN 1412. The SSH VCN 1412 can include an SSH subnet 1414, and the SSH VCN 1412 can be communicatively coupled to a control plane VCN 1416 via the LPG 1410 included in the control plane VCN 1416. Moreover, the SSH VCN 1412 can be communicatively coupled to a data plane VCN 1418 via the LPG 1410. The control plane VCN 1416 and the data plane VCN 1418 can be included in a service lease 1419 that can be owned and / or operated by an IaaS provider.

[0230] The control plane VCN 1416 may include a control plane demilitarized zone (DMZ) layer 1420 that serves as a perimeter network (e.g., a portion of the corporate network between the corporate intranet and the external network). Servers based on the DMZ can assume limited liability and help control vulnerabilities. Additionally, the DMZ layer 1420 may include one or more load balancer (LB) subnets 1422, a control plane application layer 1424 that may include one or more application subnets 1426, and a control plane data layer 1428 that may include one or more database (DB) subnets 1430 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). The one or more LB subnets 1422 included in the control plane DMZ layer 1420 may be communicatively coupled to the one or more application subnets 1426 included in the control plane application layer 1424 and the Internet gateway 1434 that may be included in the control plane VCN 1416, and the one or more application subnets 1426 may be communicatively coupled to the one or more DB subnets 1430 included in the control plane data layer 1428, as well as the service gateway 1436 and the network address translation (NAT) gateway 1438. The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0231] The control plane VCN 1416 may include a data plane mirror application layer 1440, which may include one or more application subnets 1426. The one or more application subnets 1426 included in the data plane mirror application layer 1440 may include virtual network interface controllers (VNICs) 1442 that may execute compute instances 1444. The compute instances 1444 may communicatively couple the one or more application subnets 1426 of the data plane mirror application layer 1440 to the one or more application subnets 1426 that may be included in the data plane application layer 1446.

[0232] The data plane VCN 1418 may include a data plane application layer 1446, a data plane DMZ layer 1448, and a data plane data layer 1450. The data plane DMZ layer 1448 may include one or more LB subnets 1422, which may be communicatively coupled to the one or more application subnets 1426 of the data plane application layer 1446 and the Internet gateway 1434 of the data plane VCN 1418. The one or more application subnets 1426 may be communicatively coupled to the service gateway 1436 of the data plane VCN 1418 and the NAT gateway 1438 of the data plane VCN 1418. The data plane data layer 1450 may also include one or more DB subnets 1430 that may be communicatively coupled to the one or more application subnets 1426 of the data plane application layer 1446.

[0233] The Internet gateway 1434 for the control plane VCN 1416 and the data plane VCN 1418 can be communicatively coupled to the metadata management service 1452, and the metadata management service 1452 can be communicatively coupled to the public Internet 1454. The public Internet 1454 can be communicatively coupled to the NAT gateway 1438 for the control plane VCN 1416 and the data plane VCN 1418. The service gateway 1436 for the control plane VCN 1416 and the data plane VCN 1418 can be communicatively coupled to the cloud service 1456.

[0234] In some examples, the service gateway 1436 for the control plane VCN 1416 or the data plane VCN 1418 can make an application programming interface (API) call to the cloud service 1456 without going through the public Internet 1454. The API call from the service gateway 1436 to the cloud service 1456 can be one-way: the service gateway 1436 can make an API call to the cloud service 1456, and the cloud service 1456 can send the requested data to the service gateway 1436. However, the cloud service 1456 may not initiate an API call to the service gateway 1436.

[0235] In some examples, the secure host lease 1404 can be directly connected to the service lease 1419, which would otherwise be isolated. The secure host subnet 1408 can communicate with the SSH subnet 1414 via the LPG 1410, and the LPG 1410 can enable two-way communication on otherwise isolated systems. Connecting the secure host subnet 1408 to the SSH subnet 1414 can enable the secure host subnet 1408 to access other entities within the service lease 1419.

[0236] The control plane VCN 1416 can allow users of the service lease 1419 to set or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1416 can be deployed or otherwise used in the data plane VCN 1418. In some examples, the control plane VCN 1416 can be isolated from the data plane VCN 1418, and the data plane mirror application layer 1440 of the control plane VCN 1416 can communicate with the data plane application layer 1446 of the data plane VCN 1418 via the VNIC 1442, and the VNIC 1442 can be included in the data plane mirror application layer 1440 and the data plane application layer 1446.

[0237] In some examples, a user or customer of the system can make requests, such as create, read, update, or delete (CRUD) operations, via a public internet 1454 that can transmit requests to a metadata management service 1452. The metadata management service 1452 can transmit the requests to a control plane VCN 1416 via an internet gateway 1434. The requests can be received by one or more LB subnets 1422 included in a control plane DMZ layer 1420. The one or more LB subnets 1422 can determine that the requests are valid, and in response to that determination, the one or more LB subnets 1422 can transmit the requests to one or more application subnets 1426 included in a control plane application layer 1424. If the requests are verified and a call to the public internet 1454 is required, then the call to the public internet 1454 can be transmitted to a NAT gateway 1438 that can make calls to the public internet 1454. Metadata that the requests may expect to be stored can be stored in one or more DB subnets 1430.

[0238] In some examples, a data plane mirror application layer 1440 can facilitate direct communication between a control plane VCN 1416 and a data plane VCN 1418. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources included in the data plane VCN 1418. Via a VNIC 1442, the control plane VCN 1416 can communicate directly with resources included in the data plane VCN 1418 and thereby can perform configuration changes, updates, or other appropriate modifications.

[0239] In some embodiments, a control plane VCN 1416 and a data plane VCN 1418 can be included in a service tenancy 1419. In such a case, a user or customer of the system may not own or operate the control plane VCN 1416 or the data plane VCN 1418. Instead, an IaaS provider can own or operate the control plane VCN 1416 and the data plane VCN 1418, both of which can be included in the service tenancy 1419. This embodiment can enable isolation of networks that may prevent a user or customer from interacting with resources of other users or other customers. Moreover, this embodiment can allow a user or customer of the system to privately store databases without relying on the public internet 1454, which may not have a desired threat prevention level, for storage.

[0240] In other embodiments, the (one or more) LB subnets 1422 included in the control plane VCN 1416 can be configured to receive signals from the service gateway 1436. In this embodiment, the control plane VCN 1416 and the data plane VCN 1418 can be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1454. A customer of the IaaS provider may desire this embodiment because the (one or more) databases used by the customer can be controlled by the IaaS provider and can be stored on the service lease 1419, which may be isolated from the public Internet 1454.

[0241] Figure 15 is a block diagram 1400 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1502 (e.g., Figure 14 the service operator 1402) can be communicatively coupled to a secure host lease 1504 (e.g., Figure 14 the secure host lease 1404), which can include a virtual cloud network (VCN) 1506 (e.g., Figure 14 the VCN 1406) and a secure host subnet 1508 (e.g., Figure 14 the secure host subnet 1408). The VCN 1506 can include a local peer gateway (LPG) 1510 (e.g., Figure 14 the LPG 1410), which can be communicatively coupled to a Secure Shell (SSH) VCN 1512 (e.g., Figure 14 the SSH VCN 1412) via the LPG 1410 included in the SSH VCN 1512. The SSH VCN 1512 can include an SSH subnet 1514 (e.g., Figure 14 the SSH subnet 1414), and the SSH VCN 1512 can be communicatively coupled to a control plane VCN 1516 (e.g., Figure 14 the control plane VCN 1416) via the LPG 1510 included in the control plane VCN 1516. The control plane VCN 1516 can be included in a service lease 1519 (e.g., Figure 14 the service lease 1419), and a data plane VCN 1524 (e.g., Figure 14 the data plane VCN 1418) can be included in a customer lease 1521 that may be owned or operated by a user or customer of the system.

[0242] The control plane VCN 1516 can include a control plane DMZ layer 1520 (e.g., Figure 14 the control plane DMZ layer 1420), which can include the (one or more) LB subnets 1522 (e.g.,Figure 14 one or more LB subnets 1422), and may include one or more application subnets 1526 (e.g., Figure 14 one or more application subnets 1426) of the control plane application layer 1524 (e.g., Figure 14 control plane application layer 1424), and may include one or more database (DB) subnets 1530 (e.g., similar to Figure 14 one or more DB subnets 1430) of the control plane data layer 1528 (e.g., Figure 14 control plane data layer 1428). One or more LB subnets 1522 included in the control plane DMZ layer 1520 may be communicatively coupled to one or more application subnets 1526 included in the control plane application layer 1524 and an Internet gateway 1534 that may be included in the control plane VCN 1516 (e.g., Figure 14 Internet gateway 1434), and one or more application subnets 1526 may be communicatively coupled to one or more DB subnets 1530 included in the control plane data layer 1528, as well as a service gateway 1536 (e.g., Figure 14 service gateway 1436) and a network address translation (NAT) gateway 1538 (e.g., Figure 14 NAT gateway 1438). The control plane VCN 1516 may include a service gateway 1536 and a NAT gateway 1538.

[0243] The control plane VCN 1516 may include a data plane mirror application layer 1540 that may include one or more application subnets 1526 (e.g., Figure 14 data plane mirror application layer 1440). One or more application subnets 1526 included in the data plane mirror application layer 1540 may include a virtual network interface controller (VNIC) 1542 that may execute a compute instance 1544 (e.g., similar to Figure 14 compute instance 1444) of 1442. The compute instance 1544 may facilitate communication between one or more application subnets 1526 of the data plane mirror application layer 1540 and one or more application subnets 1526 that may be included in the data plane application layer 1546 (e.g., Figure 14 data plane application layer 1446) via the VNIC 1542 included in the data plane mirror application layer 1540 and the VNIC 1542 included in the data plane application layer 1546.

[0244] The Internet gateway 1534 included in the control plane VCN 1516 may be communicatively coupled to a metadata management service 1552 (e.g.,Figure 14 a metadata management service 1452), and the metadata management service 1552 can be communicatively coupled to a public Internet 1554 (e.g., Figure 14 the public Internet 1454). The public Internet 1554 can be communicatively coupled to a NAT gateway 1538 included in the control plane VCN 1516. A service gateway 1536 included in the control plane VCN 1416 can be communicatively coupled to a cloud service 1556 (e.g., Figure 14 the cloud service 1456).

[0245] In some examples, the data plane VCN 1518 can be included in a customer tenancy 1521. In this case, the IaaS provider can provide a control plane VCN 1516 for each customer, and the IaaS provider can set up a unique compute instance 1544 included in a service tenancy 1519 for each customer. Each compute instance 1544 can permit communication between the control plane VCN 1516 included in the service tenancy 1519 and the data plane VCN 1518 included in the customer tenancy 1521. The compute instance 1544 can permit resources provisioned in the control plane VCN 1516 included in the service tenancy 1519 to be deployed or otherwise used in the data plane VCN 1518 included in the customer tenancy 1521.

[0246] In other examples, a customer of the IaaS provider can have a database that exists in the customer tenancy 1521. In this example, the control plane VCN 1516 can include a data plane mirror application layer 1540, which can include one or more application subnets 1526. The data plane mirror application layer 1540 can reside in the data plane VCN 1518, but the data plane mirror application layer 1540 may not be in the data plane VCN 1518. That is, the data plane mirror application layer 1540 can access the customer tenancy 1521, but the data plane mirror application layer 1540 may not exist in the data plane VCN 1518 or be owned or operated by the customer of the IaaS provider. The data plane mirror application layer 1540 can be configured to make calls to the data plane VCN 1518, but may not be configured to make calls to any entity included in the control plane VCN 1516. The customer may expect to deploy or otherwise use resources provisioned in the control plane VCN 1516 in the data plane VCN 1518, and the data plane mirror application layer 1540 can facilitate the customer's desired deployment or other use of the resources.

[0247] In some embodiments, a customer of an IaaS provider can apply filters to a data plane VCN 1518. In this embodiment, the customer can determine what the data plane VCN 1518 can access, and the customer can restrict access from the data plane VCN 1518 to the public Internet 1554. The IaaS provider may not be able to apply filters or otherwise control access of the data plane VCN 1518 to any external network or database. Applying filters and controls by the customer to the data plane VCN 1518 included in the customer lease 1521 can help isolate the data plane VCN 1518 from other customers and the public Internet 1554.

[0248] In some embodiments, a cloud service 1556 can be invoked by a service gateway 1536 to access services that may not be present on the public Internet 1554, the control plane VCN 1516, or the data plane VCN 1518. The connection between the cloud service 1556 and the control plane VCN 1516 or the data plane VCN 1518 may not be real-time or continuous. The cloud service 1556 can exist on a different network owned or operated by the IaaS provider. The cloud service 1556 can be configured to receive calls from the service gateway 1536 and can be configured not to receive calls from the public Internet 1554. Some cloud services 1556 can be isolated from other cloud services 1556, and the control plane VCN 1516 can be isolated from cloud services 1556 that may not be in the same region as the control plane VCN 1516. For example, the control plane VCN 1516 can be located in "Region 1", and the cloud service "Deployment 14" can be located in Region 1 and "Region 2". If the service gateway 1536 included in the control plane VCN 1516 located in Region 1 makes a call to Deployment 14, then the call can be transmitted to Deployment 14 in Region 1. In this example, the control plane VCN 1516 or Deployment 14 in Region 1 may not be communicatively coupled or otherwise communicate with Deployment 14 in Region 2.

[0249] Figure 16 is a block diagram 1500 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1602 (e.g., Figure 14 the service operator 1402) can be communicatively coupled to a secure host lease 1604 (e.g., Figure 14 the secure host lease 1404), which can include a virtual cloud network (VCN) 1606 (e.g., Figure 14 the VCN 1406) and a secure host subnet 1608 (e.g., Figure 14 the secure host subnet 1408). The VCN 1606 can include an LPG 1610 (e.g.,Figure 14 1410), which may be communicatively coupled to the SSH VCN 1612 via the LPG 1610 contained in the SSH VCN 1612 (e.g., Figure 14 SSH VCN 1412). SSH VCN 1612 may include SSH subnet 1614 (e.g., Figure 14 SSH subnet 1414 of the control plane VCN 1616), and SSH VCN 1612 can be communicatively coupled to control plane VCN 1616 via LPG 1610 contained in control plane VCN 1616 (e.g., Figure 14 The control plane VCN 1416 of FIG. 1414 is coupled to the data plane VCN 1618 via the LPG 1610 included in the data plane VCN 1618 (e.g., Figure 14 The control plane VCN 1616 and the data plane VCN 1618 may be included in a service lease 1619 (e.g., Figure 14 Service lease 1419).

[0250] The control plane VCN 1616 may include a load balancer (LB) subnet 1622 (e.g., Figure 14 The control plane DMZ layer 1620 (e.g., Figure 14 control plane DMZ layer 1426), may include (one or more) application subnets 1626 (e.g., similar to Figure 14 (one or more) application subnets 1426) of the control plane application layer 1624 (e.g., Figure 14 A control plane application layer 1424), a control plane data layer 1628 (e.g., Figure 14 1428 of the control plane data layer 1428). The LB subnet(s) 1622 contained in the control plane DMZ layer 1620 may be communicatively coupled to the application subnet(s) 1626 contained in the control plane application layer 1624 and the Internet gateway 1634 (e.g., Figure 14 1434), and the application subnet(s) 1626 may be communicatively coupled to the DB subnet(s) 1630 and the service gateway 1636 (e.g., Figure 14 ) and a network address translation (NAT) gateway 1638 (e.g., Figure 14 The control plane VCN 1616 may include a service gateway 1636 and a NAT gateway 1638.

[0251] The data plane VCN 1618 may include a data plane application layer 1646 (e.g., Figure 14 the data plane application layer 1446 of), a data plane DMZ layer 1648 (e.g., Figure 14 the data plane DMZ layer 1448 of), and a data plane data layer 1650 (e.g., Figure 14 the data plane data layer 1450 of). The data plane DMZ layer 1648 may include one or more trusted application subnets 1660 and one or more untrusted application subnets 1662 that may be communicatively coupled to the data plane application layer 1646 and one or more LB subnets 1622 of the Internet gateway 1634 included in the data plane VCN 1618. One or more trusted application subnets 1660 may be communicatively coupled to a service gateway 1636 included in the data plane VCN 1618, a NAT gateway 1638 included in the data plane VCN 1618, and one or more DB subnets 1630 included in the data plane data layer 1650. One or more untrusted application subnets 1662 may be communicatively coupled to a service gateway 1636 included in the data plane VCN 1618 and one or more DB subnets 1630 included in the data plane data layer 1650. The data plane data layer 1650 may include one or more DB subnets 1630 that may be communicatively coupled to a service gateway 1636 included in the data plane VCN 1618.

[0252] One or more untrusted application subnets 1662 may include one or more primary VNICs 1664(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1666(1)-(N). Each tenant VM 1666(1)-(N) may be communicatively coupled to a corresponding application subnet 1667(1)-(N) that may be included in a corresponding container egress VCN 1668(1)-(N), and the corresponding container egress VCNs 1668(1)-(N) may be included in corresponding customer tenancies 1670(1)-(N). Corresponding secondary VNICs 1672(1)-(N) may facilitate communication between one or more untrusted application subnets 1662 included in the data plane VCN 1618 and application subnets included in the container egress VCNs 1668(1)-(N). Each container egress VCN 1668(1)-(N) may include a NAT gateway 1638 that may be communicatively coupled to a public Internet 1654 (e.g., Figure 14 the public Internet 1454 of).

[0253] An Internet gateway 1634 that is included in the control plane VCN 1616 and that is included in the data plane VCN 1618 can be communicatively coupled to a metadata management service 1652 (e.g., Figure 14 the metadata management system 1452), and the metadata management service 1652 can be communicatively coupled to the public Internet 1654. The public Internet 1654 can be communicatively coupled to a NAT gateway 1638 that is included in the control plane VCN 1616 and that is included in the data plane VCN 1618. A service gateway 1636 that is included in the control plane VCN 1616 and that is included in the data plane VCN 1618 can be communicatively coupled to a cloud service 1656.

[0254] In some embodiments, the data plane VCN 1618 can be integrated with a customer lease 1670. In some cases, such as when it may be desirable to support during code execution, this integration may be useful or desirable for customers of an IaaS provider. A customer may provide code to run that may be disruptive, may communicate with other customer resources, or may otherwise cause undesired effects. In response thereto, the IaaS provider can determine whether to run the code given to the IaaS provider by the customer.

[0255] In some examples, a customer of an IaaS provider can grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1646. The code to run the functionality can be executed in VMs 1666(1)-(N), and the code can be not configured to run anywhere else on the data plane VCN 1618. Each VM 1666(1)-(N) can be connected to a customer lease 1670. The corresponding containers 1671(1)-(N) included in the VMs 1666(1)-(N) can be configured to run the code. In such a case, there can be double isolation (e.g., the containers 1671(1)-(N) run the code, where the containers 1671(1)-(N) may be at least included in the VMs 1666(1)-(N) included in one or more untrusted application subnets 1662), which can help prevent incorrect or otherwise undesired code from corrupting the IaaS provider's network or corrupting the networks of different customers. The containers 1671(1)-(N) can be communicatively coupled to the customer lease 1670 and can be configured to transmit or receive data from the customer lease 1670. The containers 1671(1)-(N) can be not configured to transmit or receive data from any other entity in the data plane VCN 1618. After the code execution is complete, the IaaS provider can terminate or otherwise dispose of the containers 1671(1)-(N).

[0256] In some embodiments, the (one or more) trusted application subnets 1660 may run code that may be owned or operated by an IaaS provider. In this embodiment, the (one or more) trusted application subnets 1660 may be communicatively coupled to the (one or more) DB subnets 1630 and configured to perform CRUD operations in the (one or more) DB subnets 1630. The (one or more) untrusted application subnets 1662 may be communicatively coupled to the (one or more) DB subnets 1630, but in this embodiment, the (one or more) untrusted application subnets may be configured to perform read operations in the (one or more) DB subnets 1630. Containers 1671(1)-(N) that may be included in each customer's VM 1666(1)-(N) and may run code from the customer may not be communicatively coupled to the (one or more) DB subnets 1630.

[0257] In other embodiments, the control plane VCN 1616 and the data plane VCN 1618 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1616 and the data plane VCN 1618. However, communication may occur indirectly through at least one method. The LPG 1610 may be established by the IaaS provider, which may facilitate communication between the control plane VCN 1616 and the data plane VCN 1618. In another example, the control plane VCN 1616 or the data plane VCN 1618 may invoke cloud services 1656 via the service gateway 1636. For example, an invocation of cloud services 1656 from the control plane VCN 1616 may include a request for a service that may communicate with the data plane VCN 1618.

[0258] Figure 17 is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. A service operator 1702 (e.g., Figure 14 the service operator 1402) may be communicatively coupled to a secure host lease 1704 (e.g., Figure 14 the secure host lease 1404), which may include a virtual cloud network (VCN) 1706 (e.g., Figure 14 the VCN 1406) and a secure host subnet 1708 (e.g., Figure 14 the secure host subnet 1408). The VCN 1706 may include an LPG 1710 (e.g., Figure 14 the LPG 1410), which may be via an SSH VCN 1712 (e.g., Figure 14The LPG 1710 in the SSH VCN 1412 is communicatively coupled to the SSH VCN 1712. The SSH VCN 1712 may include an SSH subnet 1714 (e.g., Figure 14 the SSH subnet 1414), and the SSH VCN 1712 may be communicatively coupled to the control plane VCN 1716 via the LPG 1710 included in the control plane VCN 1716 (e.g., Figure 14 the control plane VCN 1416) and coupled to the data plane VCN 1718 via the LPG 1710 included in the data plane VCN 1718 (e.g., Figure 14 the data plane 1418). The control plane VCN 1716 and the data plane VCN 1718 may be included in a service tenancy 1719 (e.g., Figure 14 the service tenancy 1419).

[0259] The control plane VCN 1716 may include a control plane DMZ layer 1720 that may include one or more LB subnets 1722 (e.g., Figure 14 one or more LB subnets 1422), a control plane application layer 1724 that may include one or more application subnets 1726 (e.g., Figure 14 the control plane DMZ layer 1420), a control plane application layer 1724 that may include one or more application subnets 1726 (e.g., Figure 14 one or more application subnets 1426), and a control plane data layer 1728 that may include one or more DB subnets 1730 (e.g., Figure 14 the control plane application layer 1424), a control plane data layer 1728 that may include one or more DB subnets 1730 (e.g., Figure 16 one or more DB subnets 1630). One or more LB subnets 1722 included in the control plane DMZ layer 1720 may be communicatively coupled to one or more application subnets 1726 included in the control plane application layer 1712 and an Internet gateway 1734 that may be included in the control plane VCN 1716 (e.g., Figure 14 the control plane data layer 1428). One or more application subnets 1726 may be communicatively coupled to one or more DB subnets 1730 included in the control plane data layer 1728, a service gateway 1736 (e.g., Figure 14 the Internet gateway 1434), and a network address translation (NAT) gateway 1738 (e.g., Figure 14 the service gateway), and a network address translation (NAT) gateway 1738 (e.g., Figure 14 the NAT gateway 1438). The control plane VCN 1716 may include a service gateway 1736 and a NAT gateway 1738.

[0260] The data plane VCN 1718 may include a data plane application layer 1746 (e.g., Figure 14 's data plane application layer 1446), a data plane DMZ layer 1748 (e.g., Figure 14 's data plane DMZ layer 1448)), and a data plane data layer 1750 (e.g., Figure 14 's data plane data layer 1450). The data plane DMZ layer 1748 may include one or more trusted application subnets 1760 (e.g., Figure 16 's one or more trusted application subnets 1660) and one or more untrusted application subnets 1762 (e.g., Figure 16 's one or more untrusted application subnets 1662) that are communicatively coupled to the data plane application layer 1746, and one or more LB subnets 1722 of the Internet gateway 1734 included in the data plane VCN 1718. The one or more trusted application subnets 1760 may be communicatively coupled to a service gateway 1736 included in the data plane VCN 1718, a NAT gateway 1738 included in the data plane VCN 1718, and one or more DB subnets 1730 included in the data plane data layer 1750. The one or more untrusted application subnets 1762 may be communicatively coupled to a service gateway 1736 included in the data plane VCN 1718 and one or more DB subnets 1730 included in the data plane data layer 1750. The data plane data layer 1750 may include one or more DB subnets 1730 that are communicatively coupled to a service gateway 1736 included in the data plane VCN 1718.

[0261] The one or more untrusted application subnets 1762 may include primary VNICs 1764(1)-(N) that are communicatively coupled to tenant virtual machines (VMs) 1766(1)-(N) residing within the one or more untrusted application subnets 1762. Each tenant VM 1766(1)-(N) may run code in a corresponding container 1767(1)-(N) and is communicatively coupled to an application subnet 1726 in the data plane application layer 1746 that may be included in a container egress VCN 1768. Corresponding secondary VNICs 1772(1)-(N) may facilitate communication between the one or more untrusted application subnets 1762 included in the data plane VCN 1718 and the application subnet included in the container egress VCN 1768. The container egress VCN may include a NAT gateway 1738 that is communicatively coupled to a public Internet 1754 (e.g., Figure 14 's public Internet 1454).

[0262] The Internet gateway 1734 included in the control plane VCN 1716 and the Internet gateway 1734 included in the data plane VCN 1718 can be communicatively coupled to a metadata management service 1752 (e.g., Figure 14 the metadata management system 1452), and the metadata management service 1752 can be communicatively coupled to the public Internet 1754. The public Internet 1754 can be communicatively coupled to the NAT gateway 1738 included in the control plane VCN 1716 and included in the data plane VCN 1718. The service gateway 1736 included in the control plane VCN 1716 and included in the data plane VCN 1718 can be communicatively coupled to the cloud service 1756.

[0263] In some examples, Figure 17 the pattern shown in the architecture of the block diagram 1600 of Figure 16 can be considered an exception to the pattern shown in the architecture of the block diagram 1500 of

[0264] and may be desirable for customers of the IaaS provider if the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). The customer can access in real time the respective containers 1767(1)-(N) included in each customer's VMs 1766(1)-(N). The containers 1767(1)-(N) can be configured to make calls to the respective secondary VNICs 1772(1)-(N) included in the (one or more) application subnets 1726 of the data plane application layer 1746, and the data plane application layer 1746 can be included in the container egress VCN 1768. The secondary VNICs 1772(1)-(N) can transmit the calls to the NAT gateway 1738, and the NAT gateway 1738 can transmit the calls to the public Internet 1754. In this example, the containers 1767(1)-(N) that can be accessed by the customer in real time can be isolated from the control plane VCN 1716 and can be isolated from other entities included in the data plane VCN 1718. The containers 1767(1)-(N) can also be isolated from resources from other customers.In other examples, a customer can use containers 1767(1)-(N) to invoke cloud service 1756. In this example, the customer can run code in containers 1767(1)-(N) that requests services from cloud service 1756. Containers 1767(1)-(N) can transmit the request to secondary VNICs 1772(1)-(N), which can transmit the request to a NAT gateway that can transmit the request to public Internet 1754. Public Internet 1754 can transmit the request via Internet gateway 1734 to one or more LB subnets 1722 included in control plane VCN 1716. In response to determining that the request is valid, the one or more LB subnets can transmit the request to one or more application subnets 1726, which can transmit the request to cloud service 1756 via service gateway 1736.

[0265] It should be appreciated that the IaaS architectures 1400, 1500, 1600, 1700 depicted in the figures may have other components than those depicted. Additionally, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that may incorporate the embodiments of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0266] In certain embodiments, the IaaS systems described herein may include application suite, middleware, and database service offerings that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0267] Figure 18 An example computer system 1800 in which various embodiments may be implemented is illustrated. System 1800 may be used to implement any of the computer systems described above. As shown, computer system 1800 includes a processing unit 1804 that communicates with a plurality of peripheral subsystems via a bus subsystem 1802. These peripheral subsystems may include a processing acceleration unit 1806, an I / O subsystem 1808, a storage subsystem 1818, and a communication subsystem 1824. Storage subsystem 1818 includes a tangible computer-readable storage medium 1822 and system memory 1810.

[0268] The bus subsystem 1802 provides a mechanism for enabling the various components and subsystems of the computer system 1800 to communicate with each other as intended. Although the bus subsystem 1802 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1802 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which can be implemented as Mezzanine buses manufactured to the IEEE P1386.1 standard.

[0269] The processing unit 1804, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1800. One or more processors may be included in the processing unit 1804. These processors may include single-core or multi-core processors. In certain embodiments, the processing unit 1804 may be implemented as one or more independent processing units 1832 and / or 1834, where each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1804 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0270] In various embodiments, the processing unit 1804 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in the (one or more) processors 1804 and / or the storage subsystem 1818. Through appropriate programming, the (one or more) processors 1804 can provide the various functions described above. The computer system 1800 may additionally include a processing acceleration unit 1806, which may include a digital signal processor (DSP), a dedicated processor, and the like.

[0271] The I / O subsystem 1808 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keyboard, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as Microsoft A motion sensor that enables a user to control and interact with an input device such as a Microsoft 360 game controller using gestures and voice commands through a natural user interface. The user interface input device can also include an eye gesture recognition device, such as one that detects eye activity from the user (e.g., "blinking" when taking a photo and / or making a menu selection) and converts the eye gesture into an input to the input device (e.g., Google ) Google blink detector. Additionally, the user interface input device can include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., Navigator) via voice commands.

[0272] The user interface input device can also include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, game pad, and graphics tablet, as well as audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye tracking devices. Additionally, the user interface input device can include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. The user interface input device can also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0273] The user interface output device can include a display subsystem, indicator lights, or a non-visual display such as an audio output device, etc. The display subsystem can be a cathode ray tube (CRT), a flat panel device such as one using a liquid crystal display (LCD) or a plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1800 to a user or other computer. For example, the user interface output device can include, but is not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

[0274] The computer system 1800 can include a storage subsystem 1818 that provides a tangible non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software can include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of the processing unit 1804, provide the above functionality. The storage subsystem 1818 can also provide a repository for storing data used in accordance with this disclosure.

[0275] As Figure 18 depicted in the example of

[0276]

[0276] , the storage subsystem 1818 can include various components, including system memory 1810, computer-readable storage medium 1822, and computer-readable storage medium reader 1820. The system memory 1810 can store program instructions that can be loaded and executed by the processing unit 1804. The system memory 1810 can also store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs can be loaded into the system memory 1810, including but not limited to client applications, web browsers, middle-tier applications, relational database management systems (RDBMSs), virtual machines, containers, etc.

[0276] The system memory 1810 can also store an operating system 1816. Examples of the operating system 1816 can include various versions of Microsoft Apple and / or Linux operating systems, various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems (such as iOS, Phone, OS, OS, and OS operating systems). In some embodiments where the computer system 1800 executes one or more virtual machines, the virtual machines along with their guest operating systems (GOSs) can be loaded into the system memory 1810 and executed by one or more processors or cores of the processing unit 1804.

[0277] The system memory 1810 can be configured differently depending on the type of the computer system 1800. For example, the system memory 1810 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, the system memory 1810 can include a basic input / output system (BIOS) that contains basic routines that help transfer information between components within the computer system 1800, such as during startup.

[0278] The computer-readable storage medium 1822 can represent a remote, local, fixed, and / or removable storage device and a storage medium for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1804 of the computer system 1800) for use by the computer system 1800.

[0279] The computer-readable storage medium 1822 can include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for the storage and / or transmission of information. This can include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape cartridges, tapes, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0280] For example, the computer-readable storage medium 1822 can include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a disk drive that reads from or writes to a removable non-volatile disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk (such as a CD ROM, DVD, and disc or other optical media). The computer-readable storage medium 1822 can include, but is not limited to, drives, flash cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD discs, digital audio tapes, and so on. The computer-readable storage medium 1822 can also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), SSDs based on volatile memory (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1800.

[0281] Machine-readable instructions executable by one or more processors or cores of processing unit 1804 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, which include volatile memory storage devices and / or non-volatile storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0282] Communication subsystem 1824 provides an interface to other computer systems and networks. Communication subsystem 1824 serves as an interface for receiving data from other systems and sending data from computer system 1800 to other systems. For example, communication subsystem 1824 may enable computer system 1800 to connect to one or more devices via the Internet. In some embodiments, communication subsystem 1824 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), advanced data network technologies such as WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, as an addition or alternative to the wireless interface, communication subsystem 1824 may provide a wired network connection (e.g., Ethernet).

[0283] In some embodiments, communication subsystem 1824 may also receive input communications in the form of structured and / or unstructured data feeds 1826, event streams 1828, event updates 1830, etc., on behalf of one or more users who may use computer system 1800.

[0284] By way of example, communication subsystem 1824 may be configured to receive data feeds 1826 from users of social networks and / or other communication services in real time, such as feeds, updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0285] In addition, the communication subsystem 1824 can also be configured to receive data in the form of a continuous data stream, which can include an event stream 1828 and / or event updates 1830 of real-time events that can be essentially continuous or unbounded without a clear termination. Examples of applications that produce continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and so on.

[0286] The communication subsystem 1824 can also be configured to output structured and / or unstructured data feeds 1826, event streams 1828, event updates 1830, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1800.

[0287] The computer system 1800 can be one of various types, including handheld portable devices (e.g., cellular phones, computing tablets, PDAs), wearable devices (e.g., Glass head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing system.

[0288] Due to the ever-changing nature of computers and networks, the description of the computer system 1800 depicted in the figure is merely to serve as a specific example. Many other configurations with more or fewer components than those depicted in the figure are possible. For example, custom hardware can also be used and / or specific elements can be implemented in hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices such as network input / output devices can also be employed. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will recognize other ways and / or methods to implement various embodiments.

[0289] Embodiments can be implemented by using a computer program product, including computer programs / instructions that, when executed by a processor, cause the processor to perform any method described in this disclosure.

[0290] Although specific embodiments have been described, various modifications, variations, alternative constructs, and equivalent forms are also included within the scope of this disclosure. Embodiments are not limited to operating within certain specific data processing environments, but can operate freely within multiple data processing environments. Additionally, although embodiments have been described using a specific series of transactions and steps, those skilled in the art should appreciate that the scope of this disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above embodiments can be used alone or in combination.

[0291] In addition, while embodiments have been described using a specific combination of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the present disclosure. Embodiments can be implemented using only hardware, or only software, or using a combination thereof. The various processes described herein can be implemented in the same processor or in different processors in any combination. Accordingly, where a component or service is described as being configured to perform certain operations, such a configuration can be accomplished by, for example, designing electronic circuitry to perform the operations, programming a programmable electronic circuit such as a microprocessor to perform the operations, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes can use different techniques, or the same pair of processes can use different techniques at different times.

[0292] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it will be apparent that additions, subtractions, deletions, and other modifications and changes can be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalent forms are within the scope of the following claims.

[0293] In the context of describing the disclosed embodiments (notably in the context of the following claims), the terms "a", "an", "the", and similar references are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms "comprising", "having", "including", and "containing" are to be construed as open-ended terms (i.e., meaning "including but not limited to"). The term "connected" shall be construed as being at least partially incorporated in, attached to, or joined together, even if there is something in between. Unless otherwise indicated herein, the recitation of a range of values herein is merely intended to be a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated into the specification as if it were individually recited herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples, or exemplary language (e.g., "such as") provided herein is merely intended to better illuminate the embodiments and does not pose a limitation on the scope of the present disclosure, unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the present disclosure.

[0294] Disjunctive language, such as the phrase "at least one of X, Y, or Z", unless otherwise expressly specified, is intended to be understood in the context of generally being used to represent items, terms, etc., and can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language generally is not intended to and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0295] This disclosure describes preferred embodiments of the present disclosure, including the known best mode for carrying out the present disclosure. Variations of these preferred embodiments may become apparent to those of ordinary skill in the art after reading the foregoing description. Those of ordinary skill in the art should be able to appropriately adopt such variations and practice the present disclosure in a manner different from that specifically described herein. Accordingly, the present disclosure includes all modifications and equivalent forms of the subject matter recited in the appended claims as permitted by applicable law. Moreover, unless otherwise indicated herein, the present disclosure includes any combination of the above elements in all possible variations thereof.

[0296] Example embodiments of the present disclosure can be described in accordance with the following clauses.

[0297] Clause 1. A method is disclosed. The method can include a computer system identifying a current value of a total power threshold for a plurality of hosts. The method can include the computer system using the current value of the total power threshold to manage the total power consumption of the plurality of hosts during a first time period. The method can include the computer system detecting a trigger event indicating a need to modify the total power threshold. The method can include the computer system determining a new value of the total power threshold based on the trigger event. The method can include the computer system using the new value of the total power threshold to manage the total power consumption of the plurality of hosts during a second time period.

[0298] Clause 2. The method of clause 1, wherein detecting the trigger event includes: 1) the computer system monitoring a temperature control system associated with the plurality of hosts, and 2) the computer system determining that the temperature control system is operating at a reduced capacity.

[0299] Clause 3. The method of clause 2, wherein determining the new value of the total power threshold includes: 1) the computer system identifying a table specifying respective power consumption values applicable to each of a plurality of temperature control systems associated with the plurality of hosts, the plurality of temperature control systems including the temperature control system operating at a reduced capacity, and 2) the computer system identifying from the table the power consumption value corresponding to the temperature control system operating at a reduced capacity. In some embodiments, the new value of the total power threshold is identified at least in part based on the current total power consumption value being less than the power consumption amount associated with the temperature control system operating at a reduced capacity.

[0300] Clause 4. The method of Clause 2, wherein determining a new value of the total power threshold includes: 1) obtaining, by a computer system, a machine learning model that has been previously trained to identify, from input data including an identifier of a corresponding temperature control system, a corresponding power consumption value applicable to the corresponding temperature control system; 2) providing, by the computer system, a corresponding identifier associated with the temperature control system operating at a reduced capacity as an input to the machine learning model; 3) receiving, by the computer system, a power consumption value applicable to the temperature control system as an output from the machine learning model, wherein the new value of the total power consumption is determined at least in part based on the power consumption value applicable to the temperature control system.

[0301] Clause 5. The method of Clause 4, wherein the machine learning model is trained at least in part based on a training data set including historical power consumption values and historical metadata associated with one or more temperature control systems.

[0302] Clause 6. The method of any one of Clauses 1 - 5, wherein detecting a trigger event includes at least one of the following: 1) receiving, by the computer system, a curtailment request from a power authority from the utility grid, the curtailment request indicating the extent to which power consumption in the computer system is to be curtailed; 2) identifying, by the computer system, a degradation or failure of a temperature control system including the physical environment of the plurality of hosts; 3) identifying, by the computer system, an increase in external temperature.

[0303] Clause 7. The method of any one of Clauses 1 - 6, wherein detecting a trigger event includes predicting, by the computer system, a failure of a temperature control system associated with the plurality of hosts at least in part based on an external temperature associated with the physical environment in which the plurality of hosts are located.

[0304] Clause 8. The method of Clause 7, wherein predicting a failure of a temperature control system associated with the plurality of hosts is at least in part based on identifying that the external temperature exceeds a temperature threshold associated with the physical environment in which the plurality of hosts are located.

[0305] Clause 9. The method of Clause 8, wherein predicting a failure of a temperature control system associated with the plurality of hosts is also at least in part based on identifying that a redundant temperature control system corresponding to the temperature control system is associated with a second failure.

[0306] Clause 10. The method of any one of Clauses 1 - 9, wherein using the new value of the total power threshold to manage the total power consumption of the plurality of hosts includes at least one of the following: 1) enforcing a power ceiling on one or more hosts; 2) migrating workloads from the hosts; or 3) shutting down or pausing the hosts.

[0307] Clause 11. A computer system includes a memory configured to store instructions and one or more processors configured to execute the instructions to perform at least operations. The operations may include identifying a current value of a total power threshold for a plurality of hosts. The operations may include using the current value of the total power threshold to manage the total power consumption of the plurality of hosts during a first time period. The operations may include detecting a trigger event indicating a need to modify the total power threshold. The operations may include determining a new value of the total power threshold based on the trigger event. The operations may include using the new value of the total power threshold to manage the total power consumption of the plurality of hosts during a second time period.

[0308] Clause 12. The computer system of Clause 11, wherein executing the instructions further causes the one or more processors to identify a plurality of response levels that specify a set of applicability of reduction actions to the plurality of hosts. In some embodiments, a first response level of the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts such that a first reduction action of the first set of reduction actions is applicable to a first host of the plurality of hosts. In some embodiments, executing the instructions further causes the one or more processors to select the first response level of the plurality of response levels at least in part based on a first power reduction estimate of applying the first set of reduction actions to the first host of the plurality of hosts.

[0309] Clause 13. The computer system of Clause 12, wherein executing the instructions further causes the one or more processors to determine that a total power reduction estimated for the first response level is greater than a difference between a current value of the total power consumption and the new value of the total power threshold. In some embodiments, the total power reduction estimated for the first response level is determined at least in part based on the first power reduction estimate of applying the first set of reduction actions to the first host of the plurality of hosts.

[0310] Clause 14. Using the new value of the total power threshold to manage the total power consumption of the plurality of hosts during the second time period in the computer system of Clause 12 includes: 1) determining a difference between a current value of the total power consumption of the plurality of hosts during the second time period and a current value of the total power threshold of the plurality of hosts during the second time period, 2) determining that a first value of power reduction caused by the first response level is greater than the difference, and 3) selecting the first response level at least based on determining that the first value of power reduction caused by the first response level is greater than the difference.

[0311] Clause 15. A computer system of any one of Clauses 11-14, wherein a detected trigger event is associated with an external temperature outside the physical environment in which the plurality of hosts are located, and wherein execution of the instructions further causes the one or more processors to determine, from environmental data, a difference between a current ambient temperature of the physical environment in which the plurality of hosts are located and the external temperature outside the physical environment, wherein the environmental data includes an association between the difference and a corresponding power consumption value. In some embodiments, a new value of the total power threshold is determined at least in part based on the corresponding power consumption value associated with the difference between the current ambient temperature and the external temperature.

[0312] Clause 16. The computer system of Clause 15, wherein execution of the instructions further causes the one or more processors to: 1) obtain a machine learning model that has been previously trained to identify, from input data including a corresponding external temperature outside the physical environment in which the plurality of hosts are located, an output indicating an amount by which total power consumption should be reduced, 2) provide the current external temperature outside the physical environment in which the plurality of hosts are located as an input to the machine learning model, and 3) receive, from the machine learning model by the computer system, the reduction amount, wherein a new value of the total power threshold is determined at least in part based on the reduction amount received from the machine learning model.

[0313] Clause 17. The computer system of Clause 16, wherein the machine learning model has been previously trained using a supervised learning algorithm and a training data set that includes historical external temperatures, historical ambient temperatures, historical total power thresholds, and historical current power consumption values corresponding to the plurality of hosts.

[0314] Clause 18. A computer system of any one of Clauses 11-17, wherein detection of the trigger event is associated with detection of a failure of a temperature control system associated with the plurality of hosts, and wherein execution of the instructions further causes the one or more processors to: 1) detect that the temperature control system is operable, 2) determine an updated value of the total power threshold at least in part based on determining that the temperature control system is operable, and 3) manage the total power consumption of the plurality of hosts using the updated value of the total power threshold during a third time period.

[0315] Clause 19. The computer system of Clause 18, wherein execution of the instructions further causes the one or more processors to perform one or more recovery actions at least in part based on the updated value of the total power threshold.

[0316] Clause 20. The computer system of Clause 19, wherein the one or more recovery actions include at least one of the following: 1) stopping power capping for one or more of the plurality of hosts, 2) migrating a workload back to a host, or 3) restarting a host that was previously shut down or paused.

[0317] Clause 21. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the method of any one of claims 1-20.

[0318] All references cited herein, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and set forth in full herein.

[0319] In the foregoing specification, aspects of the disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects disclosed above may be used singly or in combination. Additionally, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A method, comprising: The computer system identifies the current value of the total power threshold for multiple hosts; During a first time period, the computer system manages the total power consumption of the multiple hosts using the current value of the total power threshold; The computer system detects a trigger event indicating a need to modify the total power threshold; The computer system determines a new value of the total power threshold based on the trigger event; And During a second time period, the computer system manages the total power consumption of the multiple hosts using the new value of the total power threshold.

2. The method according to claim 1, wherein detecting a trigger event comprises: The computer system monitors a temperature control system associated with the multiple hosts; And The computer system determines that the temperature control system is operating at a reduced capacity.

3. The method according to claim 2, wherein determining a new value of the total power threshold comprises: The computer system identifies a table specifying corresponding power consumption values applicable to each of a plurality of temperature control systems associated with the multiple hosts, the plurality of temperature control systems including the temperature control system operating at a reduced capacity; And The computer system identifies, from the table, the power consumption value corresponding to the temperature control system operating at a reduced capacity; Wherein the new value of the total power threshold is identified at least in part based on the current total power consumption value being less than the power consumption associated with the temperature control system operating at a reduced capacity.

4. The method according to claim 2, wherein determining a new value of the total power threshold comprises: The computer system obtains a machine learning model that was previously trained to identify, from input data including an identifier of a corresponding temperature control system, a corresponding power consumption value applicable to the corresponding temperature control system; The computer system provides the corresponding identifier associated with the temperature control system operating at a reduced capacity as input to the machine learning model; And The computer system receives, as output from the machine learning model, a power consumption value applicable to the temperature control system, wherein the new value of the total power consumption is determined at least in part based on the power consumption value applicable to the temperature control system.

5. The method according to claim 4, wherein the machine learning model is trained at least in part based on a training data set comprising historical power consumption values and historical metadata associated with one or more temperature control systems.

6. The method according to any one of claims 1-5, wherein detecting a trigger event comprises at least one of the following: Receiving, by the computer system, a curtailment request from a power authority from a utility grid, the curtailment request indicating the degree to which power consumption in the computer system is to be curtailed; Identifying, by the computer system, a degradation or failure of a temperature control system of a physical environment comprising the plurality of hosts; or Identifying, by the computer system, an increase in external temperature.

7. The method according to any one of claims 1-6, wherein detecting a trigger event comprises: The computer system predicts a failure of a temperature control system associated with the multiple hosts at least in part based on an external temperature associated with the physical environment in which the multiple hosts are located.

8. The method according to claim 7, wherein predicting a failure of a temperature control system associated with the plurality of hosts is at least in part based on identifying that an external temperature exceeds a temperature threshold associated with the physical environment in which the plurality of hosts are located.

9. The method according to claim 8, wherein predicting a failure of the temperature control system associated with the plurality of hosts is also at least partially based on identifying a second failure associated with a redundant temperature control system corresponding to the temperature control system.

10. The method according to any one of claims 1-9, wherein managing the total power consumption of the plurality of hosts using a new value of the total power threshold comprises at least one of the following: 1) enforcing a power ceiling on one or more hosts; 2) migrating workloads from a host; or 3) shutting down or suspending a host.

11. A computer system, comprising: A memory configured to store instructions; And One or more processors configured to execute the instructions to at least: Identify the current value of the total power threshold for multiple hosts; During a first time period, manage the total power consumption of the multiple hosts using the current value of the total power threshold; Detect a trigger event indicating a need to modify the total power threshold; Determine a new value of the total power threshold based on the trigger event; And During a second time period, manage the total power consumption of the multiple hosts using the new value of the total power threshold.

12. The computer system according to claim 11, wherein executing the instructions further causes the one or more processors to: Identify a plurality of response levels specifying the applicability of a corresponding set of reduction actions to the plurality of hosts; Wherein a first response level among the plurality of response levels specifies the applicability of a first set of reduction actions to the plurality of hosts, such that a first reduction action in the first set of reduction actions is applicable to a first host among the plurality of hosts; and Select the first response level among the plurality of response levels at least partially based on a first power reduction estimate of applying the first set of reduction actions to the first host among the plurality of hosts.

13. The computer system according to claim 12, wherein executing the instructions further causes the one or more processors to: Determine that a total power reduction estimated for the first response level is greater than a difference between a current value of the total power consumption and a new value of the total power threshold; Wherein the total power reduction estimated for the first response level is determined at least partially based on a first power reduction estimate of applying the first set of reduction actions to the first host among the plurality of hosts.

14. The computer system according to claim 12, wherein managing the total power consumption of the plurality of hosts using a new value of the total power threshold during a second time period comprises: Determine the difference between the current value of the total power consumption of the multiple hosts during the second time period and the current value of the total power threshold of the multiple hosts during the second time period; Determine that a first value of power reduction caused by a first response level is greater than the difference; And Select the first response level at least based on determining that the first value of power reduction caused by the first response level is greater than the difference.

15. The computer system according to any one of claims 11-14, wherein the detected trigger event is associated with an external temperature outside the physical environment in which the plurality of hosts are located, and wherein executing the instructions further causes the one or more processors to: Determine a difference between a current ambient temperature of the physical environment in which the plurality of hosts are located and an external temperature outside the physical environment from environmental data, wherein the environmental data includes an association between the difference and a corresponding power consumption value; Wherein a new value of the total power threshold is determined at least in part based on the corresponding power consumption value associated with the difference between the current ambient temperature and the external temperature.

16. The computer system according to claim 15, wherein executing the instructions further causes the one or more processors to: Obtain a machine learning model that has been previously trained to identify an output indicating an amount by which the total power consumption should be reduced from input data including a corresponding external temperature outside the physical environment in which the plurality of hosts are located; Provide the current external temperature outside the physical environment in which the multiple hosts are located as input to the machine learning model; And The computer system receives a reduction amount from the machine learning model, where a new value of the total power threshold is determined at least in part based on the reduction amount received from the machine learning model.

17. The computer system according to claim 16, wherein the machine learning model has been previously trained using a supervised learning algorithm and a training data set that includes historical external temperatures, historical ambient temperatures, historical total power thresholds, and historical current power consumption values corresponding to the plurality of hosts.

18. The computer system according to any one of claims 11-17, wherein detecting the trigger event is associated with detecting a failure of a temperature control system associated with the plurality of hosts, and wherein executing the instructions further causes the one or more processors to: Detect that the temperature control system is operable; Determine an updated value of the total power threshold at least in part based on determining that the temperature control system is operable; and During a third time period, manage the total power consumption of the plurality of hosts using the updated value of the total power threshold.

19. The computer system according to claim 18, wherein executing the instructions further causes the one or more processors to perform one or more recovery actions at least in part based on the updated value of the total power threshold.

20. The computer system according to claim 19, wherein the one or more recovery actions include at least one of the following: 1) stopping power capping for one or more of the plurality of hosts, 2) migrating the workload back to the host, or 3) restarting a previously shut down or paused host.

21. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-20.

Images