Data center level power management with reactive power capping

Abstract

The disclosed technology relates to managing power within a power distribution system. Power consumption corresponding to devices (e.g., servers) that receive power from upstream devices (e.g., busbars) may be monitored (e.g., by a service) to determine when the power consumption corresponding to those devices breaks through (or approaches) a budget threshold corresponding to the amount of power dispensed to the upstream devices. If the budget threshold is broken through or is likely to be broken through, the service may initiate an operation to allocate a power cap for the device and start a timer. Although allocated, power capping may be ignored by devices until they are indicated to perform power capping (e.g., when a timer expires). This allows the power consumption of the device to exceed a budget power associated with the upstream device at least prior to expiration of the timer while avoiding a power-off event.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 409,469, filed on September 23, 2022, entitled “DATACENTER LEVEL POWER MANAGEMENTWITH REACTIVE POWER CAPPING,” and U.S. Patent Application No. 18 / 202,712, filed on May 26, 2023, entitled “DATACENTER LEVEL POWER MANAGEMENT WITH REACTIVE POWER CAPPING,” the disclosures of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to techniques for managing power consumption in a data center. Power capping may be applied at any suitable level of a power distribution hierarchy. Power capping values ​​may be identified and assigned. A timer may be set based on a number of factors. While the timer is running, identification of a more favorable power capping solution may be attempted. If found, the more favorable power capping solution may be implemented. However, if the timer expires and / or a more favorable power capping solution has not been found, the previously assigned power capping values ​​may be implemented on servers in the data center. Background Art

[0004] Data centers are configured with a power infrastructure that provides a variety of safety features based on hierarchical power distribution. Power is provided from the local electric utility and distributed to various components of the data center, including power distribution units (PDUs) (e.g., transformers, switchboards, bus channels, rack PDUs, etc.) and power-consuming devices (e.g., servers, network devices, etc.) according to this distribution hierarchy. This ensures that the power consumed by all downstream devices adheres to the power limits of each upstream device. When demand reaches a peak and the power consumption of downstream devices exceeds the power limits of upstream devices, circuit breakers associated with higher-level devices may trip, causing significant interference to at least the downstream devices, resulting in reduced processing capacity and poor user experience within the data center. In the worst-case scenario, when workloads are redistributed in an attempt to recover from the initial power outage, tripped circuit breakers may trigger cascading power failures to other devices in the same or additional data centers.

[0005] Due to the disruptive and costly nature of exceeding these power limits and the potentially far-reaching impact, previous systems over-provisioned data center components, causing each power distribution device to be allocated power that exceeds the expected maximum power consumption of all downstream devices. Downstream devices are assigned power caps to constrain and throttle operations at each device to ensure that downstream devices comply with this expected maximum power consumption. These power caps are applied to avoid or correct even momentary deviations that exceed the assigned power limits. This ensures that even in the worst-case scenario where the power consumption of all downstream devices peaks at the same time, the expected maximum power draw at the upstream device will not be exceeded. These techniques are intended to reduce the risk of circuit breaker tripping and loss of computing resources. However, these methods result in wasteful power utilization, causing the power allocated to upstream devices to remain underutilized. In addition, power caps constrain the operations performed at downstream devices and can be visible to customers, which in turn leads to a frustrating user experience. Therefore, it is desirable to utilize power management techniques that minimize the frequency of constrained operations at downstream devices while maintaining a high degree of safety with respect to avoiding power failures. Summary of the invention

[0006] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and descriptions are not intended to be limiting.

[0007] Some embodiments may include a method. The method may include identifying, by a power management service, a plurality of components of an electric power system arranged according to a power distribution hierarchy, the power distribution hierarchy including a plurality of nodes organized according to respective levels of a plurality of levels. In some embodiments, a node in a plurality of nodes of the power distribution hierarchy represents a corresponding component in a plurality of components. A subset of nodes of a first level in a plurality of levels may descend from a specific node of a second level in a plurality of levels that is higher than the first level. A group of lower-level components represented by a subset of nodes of a first level in a plurality of components may receive power allocated by a higher-level component represented by a specific node of a second level in a plurality of components. The method may include monitoring, by a power management service, the power consumption of the group of lower-level components represented by a subset of nodes of a first level. The method may include determining, by a power management service, at least in part based on monitoring, that the power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component. The method may include, in response to determining that the power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component, transmitting a power cap value of the lower-level component. In some embodiments, transmission of the power cap value causes the lower level component to store the power cap value in memory while allowing corresponding power consumption of the lower level component to exceed the power cap value until a time period corresponding to the timing value expires.

[0008] Some embodiments may include a second method. The second method may include identifying, by a power management service, a plurality of components of a power system arranged according to a power distribution hierarchy, the power distribution hierarchy including a plurality of nodes organized according to respective levels of a plurality of levels. In some embodiments, a node in a plurality of nodes of the power distribution hierarchy represents a corresponding component in a plurality of components. A subset of nodes of a first level in a plurality of levels may descend from a specific node of a second level in a plurality of levels that is higher than the first level. A group of lower-level components represented by a subset of nodes of a first level in a plurality of components may receive power allocated by a higher-level component represented by a specific node of a second level in a plurality of components. The second method may include monitoring, by a power management service, power consumption of the group of lower-level components represented by a subset of nodes of a first level. The second method may include determining, by a power management service, at least in part based on monitoring, that power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component. The second method may include, in response to determining that power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component, starting a timer corresponding to a timing value. In some embodiments, expiration of the timer indicates expiration of a time period corresponding to the timing value. In some embodiments, a lower-level component in the group of lower-level components stores a power cap value in a memory during the time period. In some embodiments, enforcement of power capping values ​​for lower level components is delayed until a timer expires.

[0009] Systems, devices, and computer media are disclosed, each of which may include one or more memories on which instructions corresponding to the methods disclosed herein may be stored. The instructions may be executed by one or more processors of the disclosed systems and devices to perform the methods disclosed herein. One or more computer programs may be configured to perform specific operations or actions corresponding to the methods by including instructions that, when executed by a data processing device, cause the device to perform an action. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An example environment housing a variety of components in accordance with at least one embodiment is depicted.

[0011] Figure 2 A simplified diagram of an exemplary power distribution infrastructure including various components of a data center is shown in accordance with at least one embodiment.

[0012] Figure 3 FIG. 1 illustrates a method for performing a Figure 2 The arrangement of components corresponds to an example power distribution hierarchy.

[0013] Figure 4An example power management system for managing power allocation and consumption of components across a data center is illustrated in accordance with at least one embodiment.

[0014] Figure 5 is a flow chart illustrating an example method for managing excessive power consumption in accordance with at least one embodiment.

[0015] Figure 6 A flow chart depicting an example method for training a machine learning model to determine a likelihood that an aggregate power consumption of a downstream device will exceed a budget threshold corresponding to an upstream device is illustrated in accordance with at least one embodiment;

[0016] Figure 7 is a block diagram illustrating an example method for managing power according to at least one embodiment;

[0017] Figure 8 is a block diagram illustrating another example method for managing power in accordance with at least one embodiment;

[0018] Fig. 9 is a block diagram illustrating one mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0019] Fig.10 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment

[0020] Fig.11 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment

[0021] Fig.12 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0022] Fig.13 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION

[0023] In the following description, various embodiments will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that the embodiments can be practiced without the specific details. In addition, well-known features may be omitted or simplified in order not to obscure the embodiments being described.

[0024] The present disclosure relates to managing power distribution systems, such as power distribution infrastructure within a data center. More specifically, techniques are described for enabling components within a data center to safely consume power at a rate closer to the maximum capacity / supply of the data center. Traditionally, at least some of the power capacity of a data center is unutilized due to oversupply / over-rationing of power to various distribution units (e.g., transformers, switchboards, bus channels, rack PDUs, etc.) within the power distribution infrastructure. A "distribution unit" (also referred to as a "distribution component") refers to any suitable device, component, or structure configured to distribute power to other devices or zones. "Oversupply of power," also referred to as "over-rationing," refers to a method of power distribution in which power is supplied / rationed to a distribution unit based on a worst-case power consumption scenario, in which each power-consuming device in the data center is expected to operate at peak capacity. In this method, each distribution unit is allocated enough power to handle the peak consumption of each device downstream, plus some buffering. The power allocated to a distribution unit may be referred to as its "budget power" or "rationed power."

[0025] It is desirable to balance the supply of power and the consumption of power within a data center. When the power consumed exceeds the available supply, the balance can be restored by increasing the power supply or reducing the rate of consumption. If this balance is not maintained and a component attempts to consume more than the available supply, a circuit breaker may trip, disconnecting the component from the supply. The disconnected component may cause an interruption in the operations performed by the components of the data center. For example, if a tripped circuit breaker disconnects a server from the power supply of the data center, a website hosted on the server in the data center will crash. Therefore, various techniques have been used to avoid tripping circuit breakers.

[0026] The balance between power supply and power consumption in a data center can be managed by maintaining a balance between available supply and demand and / or consumption. Scaling the power supply in a data center may sometimes not be feasible because power is typically statically allocated in long-term contracts with power utilities. Although the supply may be statically allocated, the power consumption in a data center may vary, sometimes significantly. As a simple example, when the number of threads executed by a server's processor increases, the power consumption of the server may increase. The ambient temperature within the data center may affect the power consumed by the data center cooling system. When the cooling system works harder, it consumes more power because it works to reduce the ambient temperature experienced in the data center. The demand caused by some components in the data center (such as uninterruptible power supplies, power distribution units, cooling systems, or bus channels) may be difficult to adjust. However, some components (such as servers or networking components) can be more easily constrained.

[0027] Many traditional power management methods utilize power capping to constrain operations at power-consuming devices (e.g., servers, network devices, etc.) within a data center. When power capping is utilized, power capping limits can be used to constrain the power consumed by the devices. Power capping limits are used to constrain (e.g., throttle) operations at servers to ensure that the power consumption of the servers does not exceed the power capping limits. Using power capping ensures that the allocated power limits of upstream devices are not breached, so that the over-provisioned power at each upstream device remains unused. These methods waste precious power and limit the density of power-consuming devices that can be utilized within a data center.

[0028] Technological impact

[0029] An efficient power infrastructure within a data center is necessary to increase the provider's profit margins, manage scarce power resources, and make the services provided by the data center more environmentally friendly. A data center that includes components that host a multi-tenant environment (e.g., a public cloud) may experience higher than average consumption because not all of the cloud's leases are used at the same time. In order to push power consumption closer to the statically allocated capacity to improve the efficiency and resource utilization of the data center, the data center provider can add servers and / or leases so that the power consumption across all power-consuming devices is closer to the allocated power capacity of the data center. However, in some cases, narrowing the gap between the allocated power capacity and power consumption increases the risk of circuit breaker tripping and losing the ability to utilize computing resources. The technology described herein minimizes the frequency of operation constraints at downstream devices, enabling these devices to utilize previously unused power while maintaining a high degree of security with respect to avoiding power failures.

[0030] Figure 1 An example environment (e.g., environment 100) including a variety of components according to at least one embodiment is depicted. Environment 100 may include a data center 102, which may include a dedicated space that hosts any suitable number of servers, such as servers 104A-104P (collectively referred to as "servers 104"), and infrastructure for hosting those servers, such as networking hardware, cooling systems, and storage devices. Networking hardware (not depicted here) of data center 100 may enable remote users to interact with the servers over a network (e.g., the Internet). Any suitable number of servers 104 (e.g., 10, 14, 21, 42, etc.) may be held in various racks, such as racks 106A-106H (collectively referred to as "racks 106"). Racks 106 may include a frame or housing into which a corresponding group of servers is placed and / or mounted.

[0031] Various subsets of racks 106 may be organized into groups referred to as “rows” (e.g., rows 108A-108D, collectively referred to as “rows 108”). In some implementations, rows 108 may include any suitable number of racks (e.g., 5, 8, 10, up to 10, etc.) that are co-located (e.g., within a threshold distance of each other). In other implementations, rows may be organizational units, and racks having a given row may be placed in different locations (not necessarily within a threshold distance of each other). As an example, rows 108 may be located in rooms (e.g., room 110A, room 110N, etc.). A room (e.g., room 110A) may be a partition or physical enclosure of a building in which any suitable number of racks 106 are placed. In other embodiments, a room may be an organizational unit, and rooms may be located in different physical locations, or multiple rooms may be located in a single partition of a building.

[0032] Figure 2 A schematic diagram is shown that includes various components (e.g., Figure 1 1 and 1 . A simplified diagram of an exemplary power distribution infrastructure 200 of a data center 102 (including components of a data center 102 of FIG. 1 ). The power distribution infrastructure 200 may be connected to a utility power source (not shown), and power may be initially received at one or more uninterruptible power supplies (UPS(s) 202). In some embodiments, power may be received at the UPS(s) from the utility via an on-site substation (not shown) that is configured to establish appropriate voltage levels for distribution of power through the data center. The UPS(s) 202 may individually include dedicated batteries or generators that provide emergency power if the input power source fails. The UPS(s) 202 may monitor the input power and provide backup power if a drop in input power is detected.

[0033] The power distribution infrastructure 200 may include any suitable number of intermediate power distribution units (PDUs) (e.g., intermediate PDUs 204) connected to and receiving power / electricity from the UPSs 202. Any suitable number of intermediate PDUs 204 may be provided between the UPSs (of the UPSs 202) and any suitable number of row PDUs (e.g., row PDUs 206). The power distribution units (e.g., intermediate PDUs 204, row PDUs 206, rack PDUs 208, etc.) may be any suitable devices configured to control and distribute power / electricity. Example power distribution units may include, but are not limited to, main switchboards, switchboards, remote power panels, busbars, power outlets, transformers, etc. Power may be provided from the UPSs 202 to the intermediate PDUs 204. The intermediate PDUs 204 may distribute power to downstream components (e.g., row PDUs 206) of the power distribution infrastructure 200.

[0034] The power distribution infrastructure 200 may include any suitable number of row power distribution units (including row PDUs 206). The row PDUs may include any suitable PDUs (e.g., remote power panels, busbars / busways, etc.) disposed between an intermediate PDU (e.g., a PDU in intermediate PDU(s) 204) and one or more rack PDUs (e.g., rack PDU 208A, rack PDU 208N, collectively referred to as "rack PDU(s) 208"). A "row PDU" refers to a PDU configured to distribute power to one or more row devices (e.g., row 210, including servers 212A-212D, collectively referred to as "servers 212"). As described above, a row (e.g., row 210) may include any suitable number of racks (e.g., racks 214A-214N, collectively referred to as "racks 214") in which servers 212 are located.

[0035] The power distribution infrastructure 200 may include any suitable number of rack power distribution units (including (one or more) rack PDUs 208). The rack PDUs may include row PDUs (e.g., row PDUs 206) and racks corresponding to racks (e.g., racks 214A, 214B, 214C, 214D, 214E, 214F ... Figure 1The rack PDU 208 may be any suitable PDU between one or more servers (e.g., server 212A, server 212B, etc.) within a rack 106 (e.g., rack 106 of FIG. 1 ). A "rack PDU" refers to any suitable PDU configured to distribute power to one or more servers within a rack. A rack (e.g., rack 214A) may include any suitable number of servers 212. In some embodiments, rack PDU(s) 208 may include an intelligent PDU that is additionally configured to monitor, manage, and control consumption at multiple devices (e.g., rack PDU 208A, server 212A, server 212B, etc.).

[0036] Server 212 (each Figure 1 ) may individually include a power controller (power controllers 216A-216D, collectively referred to as "power controllers 216"). A power controller refers to any suitable hardware or software component operating at a device (e.g., a server) that is configured to monitor and / or manage power consumption at the device. The power controllers 216 may individually monitor the power consumption of the respective servers on which they operate. The power controllers 216 may each be configured to enforce power capping restrictions to constrain power consumption at the respective servers. Enforcing power capping restrictions may include any suitable combination of monitoring power consumption at a server, determining whether to constrain (e.g., limit, restrict, etc.) power consumption at a server (e.g., based at least in part on a comparison between the current consumption of the server and a stored power capping limit), and limiting / restricting power consumption at a server (e.g., using processor and memory dynamic voltage and frequency scaling to suppress server power consumption). Enforcing power capping restrictions may be referred to as "power capping."

[0037] Figure 1 Data center 102 in FIG. 10A may include various components shown in power distribution infrastructure 200. For example, room 110A may include one or more busways (each an example of a row PDU 206). A busway (also referred to as a "busway") refers to a conduit of conductive material that can distribute power (e.g., within room 110A). A busway may receive power from the power distribution units of the intermediate PDU(s) 204 and distribute power to the row PDUs 206. Figure 1 row 108A) associated with one or more racks (e.g., Figure 1 Rack 106A, Figure 1 Each power infrastructure component that distributes / provides power to other components also consumes a portion of the power that passes through it. This loss may be caused by heat loss from the power flowing through the component or by power consumed directly by the component (e.g., power consumed by a processor in a rack PDU).

[0038] Figure 3 According to at least one embodiment, Figure 2 The power distribution hierarchy 300 may represent an arrangement of any suitable number of components of a power system, such as Figure 2 The power distribution infrastructure 200 discussed above is related to the power distribution infrastructure components discussed above. The power distribution hierarchy 300 may include any suitable number of nodes organized according to any suitable number of levels. Each level may include one or more nodes. A root level (e.g., level 5) may include a single root node of the power distribution hierarchy 300. Each node of the power distribution hierarchy 300 may represent a corresponding component of the power distribution infrastructure 200. A set of one or more nodes at a given level may descend from a particular node corresponding to a higher level of the power distribution hierarchy 300. A set of lower-level components represented by nodes at a lower level (e.g., level 1) may receive power distributed through higher-level components (e.g., components located upstream of the lower-level components) represented by nodes at level 2, which in turn receive power from higher-level components represented by nodes at level 3, which receive power from higher-level components represented by nodes at level 4. In some embodiments, each component of the power system receives power initially distributed through a component corresponding to a node (e.g., node 302, a root node) at level 5 (e.g., the highest level of power distribution hierarchy 300). Node 302 may receive power from a utility power source (e.g., a local electric utility system).

[0039] As shown, the power distribution hierarchy 300 includes a node 302 at level 5. In some embodiments, the node 302 may represent Figure 2 The component corresponding to node 302 may distribute / supply power to the component corresponding to node 304 at level 4. A component (e.g., a lower-level component) that receives power from a higher-level component (a component represented by a node at a higher level in the power distribution hierarchy 300 than the node representing the lower-level component) may be considered to be subordinate to the higher-level component. In some embodiments, node 304 may represent a component such as a UPS that is subordinate to the UPS represented by node 302. Figure 2 Components (eg, switchboards) of one of the intermediate PDU(s) 204. The component represented by node 304 may be configured to distribute / supply power to components represented by nodes 306 and 308, respectively.

[0040] Level 3 nodes 306 and 308 may each represent Figure 2The component corresponding to node 306 (e.g., row PDU 206, bus) may send a signal to the component corresponding to node 310 (e.g., Figure 2 Rack PDU 208A) and components corresponding to node 312 (e.g., Figure 2 The component corresponding to node 310 (e.g., rack PDU 208A) can distribute / supply power to the components corresponding to nodes 314 and 316 of level 1 (represented respectively by Figure 2 The components corresponding to the level 1 nodes 314 and 316 may be monitored / managed by the components of the servers 212A and 212B (such as the power controllers 216A and 216B). The components corresponding to the nodes 314 and 316 may be arranged in the same rack.

[0041] The component corresponding to node 312 (eg, rack PDU 208A) may provide a signal to the component corresponding to level 1 node 318 (eg, Figure 2 The server 214C including the power controller 216C distributes / supplies power. The nodes 314, 316 and 318 of level 1 may be arranged in the same row (e.g., Figure 2 210 in ) and / or associated therewith.

[0042] Returning to level 3 node 308, node 308 (eg, a different row PDU, a remote power panel, etc.) may provide a signal to a component corresponding to node 310 (eg, Figure 2 Rack PDU 208A) and components corresponding to node 312 (e.g., Figure 2 A component corresponding to node 320 (e.g., a rack PDU) can distribute / supply power to components corresponding to nodes 322 and 326 of level 1 (e.g., corresponding to corresponding servers including corresponding power controllers). Components corresponding to nodes 324 and 326 can be arranged in the same rack. A component corresponding to node 322 (e.g., another rack PDU) can distribute / supply power to components corresponding to nodes 328 and 330 of level 1 (e.g., corresponding to corresponding servers including corresponding power controllers). Components corresponding to nodes 328 and 330 can be arranged in the same rack.

[0043] A specific number of components (e.g., nodes corresponding to level 1) receiving power distributed from higher level components (e.g., nodes corresponding to level 2) may be different from Figure 3The specific number of levels within the power distribution hierarchy may vary depending on the specific arrangement of components used in a given data center. It is envisioned that each non-root level (e.g., Figure 3 In the example, levels 1-4) can include different numbers of nodes, which represent Figure 3 The number of nodes shown in each non-root level of the component varies. The nodes can be arranged in Figure 3 A different but similar configuration is shown in .

[0044] Figure 4 An example power management system 400 (referred to as "system 400") for managing power allocation and consumption across components of a data center in accordance with at least one embodiment is illustrated. Allocating power may refer to the process of allocating a budgeted amount of power (e.g., an expected load / power draw) to any suitable component. System 400 may include (one or more) PDUs 402, (one or more) devices 404, and (one or more) service provider computers 406. (One or more) PDUs 402 may be the same as described above. Figure 2 Examples of any suitable power distribution units discussed in connection therewith (e.g., UPS(s) 202, intermediate PDU(s) 204, row PDU(s) 206, rack PDU(s) 208, etc.) may be used. For example, PDU(s) 402 may be Figure 2 The device(s) 404 may be examples of servers and / or networking equipment to which the PDU(s) 402 distribute power.

[0045] The PDU(s) 402, device(s) 404, and service provider computer(s) 406 may communicate via one or more wired or wireless networks, such as network(s) 808. In some embodiments, network(s) 408 may include any one or combination of a variety of different types of networks, such as a wired network, the Internet, a wireless network, a cellular network, and other private and / or public networks.

[0046] Device(s) 404 may be any suitable type of computing device, such as, but not limited to, a server device, a networking device, or any suitable device within a data center. In some embodiments, PDU(s) 402 and device(s) 404 are arranged in a power distribution hierarchy, such as with Figure 33. In some embodiments, device(s) 404 may correspond to and be represented by a level 1 node of the power distribution hierarchy 300. PDU(s) 402 may correspond to and be represented by a higher level node (eg, a level 2 node) of the power distribution hierarchy.

[0047] Each of the PDU(s) 402, device(s) 404, and service provider computer(s) 406 may include at least one memory (e.g., memory 410, memory 412, and memory 414, respectively) and one or more processing units (e.g., processor(s) 416, processor(s) 418, and processor(s) 420, respectively). The processor(s) 416-420 may each be suitably implemented in hardware, computer executable instructions, firmware, or a combination thereof. The computer executable instructions or firmware implementation of the processor(s) 416-420 may include computer executable or machine executable instructions written in any suitable programming language to perform the various functions described.

[0048] The memories 410-414 may store program instructions that may be loaded and executed on the corresponding processor(s) of a given device, as well as data generated during the execution of these programs. Depending on the configuration and type of the user computing device, the memories 410-414 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). The PDU(s) 402, the device(s) 404, and the service provider computer(s) 406 may also include additional removable storage and / or non-removable storage, including but not limited to magnetic storage, optical disks, and / or tape storage. Disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computing devices. In some embodiments, the memories 410-414 may individually include a variety of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM.

[0049] Turning to the contents of memories 410-414 in more detail, memories 410-414 may include an operating system (e.g., operating system 422, operating system 424, and operating system 426, respectively), one or more data repositories (e.g., data repository(s) 428, data repository(s) 430, and data storage(s) 432, respectively), and one or more applications, modules, or services.

[0050] The PDU(s) 402, the device(s) 404, and the service provider computer(s) 406 may include communication connection(s) (e.g., communication connection(s) 434, communication connection(s) 436, and communication connection(s) 438, respectively) that allow the PDU(s) 402, the device(s) 404, and the service provider computer(s) 406 to communicate with each other via the network(s) 408. The PDU(s) 402, the device(s) 404, and the service provider computer(s) 406 may also include I / O devices (e.g., I / O devices(s) 440, I / O devices(s) 442, and I / O devices(s) 444, respectively), such as a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc. In some embodiments, the service provider computer(s) 406 may be one of the device(s) 404.

[0051] Device(s) 404 (e.g., Figure 1 Server 104, Figure 2 Any suitable combination of servers 214, etc.) may include a corresponding power controller (e.g., power controller 446). Power controller 446 may be Figure 2 An example of a power controller 216 of FIG. 446. The power controller 446 can be any suitable hardware or software component operating at a device (e.g., a server) that is configured to monitor and / or manage power consumption at the device. The power controller 446 can individually monitor the power consumption of the corresponding device on which it operates. The power controller 446 can each be configured to enforce power capping restrictions to constrain power consumption at the corresponding device. Enforcing power capping restrictions can include monitoring power consumption at the device, determining whether to constrain (e.g., limit, restrict, etc.) power consumption at the device (e.g., based at least in part on a comparison between the current consumption of the device and a stored power capping limit), and limiting / restricting power consumption at the device (e.g., using dynamic voltage and / or frequency scaling of (one or more) processors 418 and / or memory 412 to suppress power consumption at the device). Any suitable operations associated with power capping can be implemented by the power controller 446.

[0052] In some embodiments, power controller 446 may communicate with power controller 448 via a direct connection (e.g., via a cable) and / or via network(s) 408. Power controller 446 may provide power consumption data indicating the current power consumption of the device (e.g., cumulative power consumption over a period of time, current power consumption rate, etc.). Power consumption data may be provided to power controller 448 at any suitable frequency, periodically, or according to a predefined schedule or event (e.g., upon breaching a predefined consumption threshold, upon a change in a threshold amount of the consumption rate, upon determining that one or more predefined conditions are satisfied, upon determining thermal properties of the device, etc.).

[0053] In some embodiments, power controller 446 may receive a power cap value (also referred to as a "power cap") from power controller 448. In some embodiments, additional data may be provided with the power cap value. For example, an indicator may be included with the power cap value indicating whether the power cap value is to be applied immediately. In some embodiments, the received power cap may be applied / implemented immediately by default. In other embodiments, the received power cap may not be applied / implemented immediately by default.

[0054] When power capping is applied / implemented, the power controller 446 can monitor power consumption at the device. This can include using a metering device (e.g., an example of (one or more) I / O devices 442) or software to identify / calculate power consumption data for the device (e.g., cumulative power consumption over a time period, current power consumption rate, current change in power consumption rate within a time window, etc.). As part of applying / implementing power capping (also known as "power capping"), the power controller 446 can determine whether to constrain (e.g., limit, restrict, etc.) power consumption at the device (e.g., based at least in part on comparing the current consumption rate of the device to a stored power cap value). When constraining power consumption at the device, the power controller 446 can limit / restrict power consumption at the device (e.g., using dynamic voltage and frequency scaling of (one or more) processors 418 and / or memory 412 to suppress power consumption at the device). In some embodiments, when the current consumption data of the device (e.g., cumulative consumption, consumption rate within a time window, etc.) approaches the power cap value being implemented (e.g., breaks through a threshold value less than the power cap value), the power controller 446 can execute instructions to limit / restrict the power consumption of the device. When constraining / restricting the power consumption at the device (also known as "throttling"), the power controller 446 ensures that the power consumption of the device is maintained below the power consumption indicated by the power cap value.

[0055] In some embodiments, the power controller of the device(s) 404 may be configured to allow the device(s) 404 to operate unconstrained (e.g., not constrained based on power consumption and power capping) until instructed by the power controller 448 to apply / enforce power capping. In some embodiments, the power controller 446 may receive a power cap value that may or may not be later instructed to be enforced, but when received, the power cap value may be stored in the memory 412 and not used for power management at the device. Thus, in some embodiments, the power controller 446 may refrain from initiating power capping operations (e.g., the comparisons and determinations described above) until instructed to do so (e.g., via an indicator provided by the power controller 448). The indication of initiation of enforcement of the power cap may be received with the power cap value, or may be received as a separate communication from the power controller 448.

[0056] According to the power distribution hierarchy 300, each power controller (e.g., power controller 448) of the PDU(s) 402 may be configured to communicate with respect to any suitable number of the device(s) 404 (e.g., Figure 2 212A and 212B). In some embodiments, power controller 448 may be a device installed in a given PDU (e.g., Figure 2 The power controller 448 may be configured to obtain power consumption data from the device(s) 404 corresponding to the device(s) with which it is associated (e.g., the device for which a given PDU operating the remote controller 448 is configured to distribute, manage, or monitor power). The power controller 448 may receive the consumption data according to a predefined frequency, period, or schedule implemented by the power controller 446, and / or the power controller 448 may receive the consumption data in response to requesting the consumption data from the power controller 446. The power controller 448 may be configured to request the consumption data from the power controller 446 according to a predefined frequency, period, or schedule implemented by the power controller 448.

[0057] The power controller 448 may transmit the received consumption data to a power management service 450 implemented by the service provider computer(s) 406 at any suitable time, according to any suitable frequency, period or schedule, or as a result of satisfying one or more predefined conditions (e.g., a rate change in individual / cumulative / collective consumption of the device(s) 404 breaching a threshold). In some embodiments, the power controller 448 may aggregate the consumption data received from the device(s) 404 (e.g., via the power controller 446 of each device), and then transmit the aggregated consumption data to the power management service 450. In some embodiments, the consumption data may be aggregated per device or across all device(s) 404.

[0058] The power controller 448 may receive one or more power cap values ​​corresponding to the device(s) 404 at any suitable time. These power cap values ​​may be calculated by the power management service 450. These calculations may be related to Figure 5 Further detailed discussion. In some embodiments, the power controller 448 may receive a timing value with a power cap value indicating the duration to be used for the timer. The power controller 448 may be configured to generate and start a timer with an associated duration / time period corresponding to the timing value. When the timer indicating that the time period corresponding to the time period has elapsed expires, the power controller 448 may transmit data to one or more devices, the data including an indicator or other suitable data indicating that one or more devices continue to perform power capping using the power cap value previously stored at each device. In some embodiments, the power cap value may be provided with the indicator. In other embodiments, immediately after receiving (one or more) power cap values ​​from the power management service 450, the power cap value may be provided by the power controller 448 to the power controller of the device (e.g., the power controller 446). Therefore, in some embodiments, the power cap value may be transmitted by the power controller 448 to the power controller 446, stored in the memory 412, but not until the power controller 446 receives subsequent data from the power controller 448 indicating that the power controller 446 starts power capping.

[0059] The service provider computer(s) 406 may be arranged in a server cluster or as a server farm and may implement the power management service 450. In some embodiments, the functionality described in connection with the service provider computer(s) 406 (including the functionality described in connection with the power management service 450) is performed by a computer running on a hosted computing environment (e.g., on a server corresponding to a server). Figure 1A hosted computing environment may include one or more rapidly provisioned and released computing resources, which may include computing, networking, and / or storage devices. A hosted computing environment may also be referred to as a cloud computing environment. Figure 10-1 4 provides and discusses a number of example cloud computing environments in greater detail.

[0060] The power management service 450 can be configured to receive consumption data from (one or more) PDUs 402 (e.g., from the power controller 448). As described above, the consumption data can be aggregated or accumulated with respect to one or more devices. As a non-limiting example, the consumption data instance received by the power management service 450 may include cumulative / aggregated consumption data for each device associated with (e.g., managed by) a given PDU. For example, the PDU providing the consumption data may be a rack PDU, and the consumption data provided by the PDU may include cumulative / aggregated and / or individual consumption data values ​​for each device in the rack. In some embodiments, the cumulative / aggregated consumption data values ​​may additionally include the power consumption of the PDU. The power management service 450 can access from this consumption data individual and / or aggregated or accumulated power consumption data for an individual device or all devices to which the instance of the consumption data is associated. The power management service 450 may be configured to perform any suitable operations (e.g., aggregating consumption data, calculating power cap values, calculating timing values, determining budgeted power, identifying whether power capping is required (e.g., when aggregated consumption exceeds / is likely to exceed budgeted power, etc.) based at least in part on the data representing the power distribution hierarchy 300 and / or any suitable configuration data identifying the arrangement of components within the data center (e.g., indicating which devices allocate power to which devices). For example, the configuration data may include data representing the power distribution hierarchy 300.

[0061] The power management service 450 may calculate aggregated / accumulated consumption data to identify aggregated / accumulated consumption data for one or more devices at a higher level of the power distribution hierarchy 300. For example, the power management service 450 may utilize consumption data provided by one or more rack PDUs and calculate aggregated / accumulated consumption data for row devices. For example, a row device corresponding to a group of common nodes sharing up to level 3 of the hierarchy may be a row device. Figure 3The power management service 450 may also include consumption data for devices represented by level 1 nodes of the power distribution hierarchy 300. In this case, the common level 3 nodes represent row PDUs, such as buses. In some embodiments, consumption data indicating power consumption of one or more rack PDUs (e.g., examples corresponding to components at level 2 of the power distribution hierarchy 300) may be included in the consumption data provided to the power management service 450. The consumption data provided by the rack PDUs may be provided according to any suitable frequency, period, or schedule or provided in response to a request transmitted by the power management service 450. The power management service 450 may obtain consumption data associated with any suitable number of device(s) 404 and / or racks of (one or more) devices 404, and may aggregate or calculate any suitable consumption data corresponding to any suitable number of components represented by any suitable node and / or level of the power distribution hierarchy 300.

[0062] The power management service 450 may be configured to calculate one or more power caps (e.g., power cap values ​​for one or more servers to which power is allocated by the higher-level device) based at least in part on the allocated power values ​​(e.g., budgeted power) for higher-level devices (e.g., row-level devices in this example). The higher-level components may correspond to any suitable level (e.g., levels 2-5) of the power distribution hierarchy 300 other than the lowest level (e.g., level 1). For example, the power management service 450 may calculate power cap values ​​for servers to which power is allocated by a given row device (e.g., a bus). These calculations may be based on consumption data provided by the rack PDUs to which power is allocated by the row devices. In some embodiments, the power management service 45 may store the consumption data for subsequent use. The power management service 450 may utilize historical consumption data when calculating these power cap values. In some embodiments, power management service 450 can obtain, utilize, and / or train one or more machine learning models to identify specific power capping values ​​for device(s) 404 (e.g., components corresponding to level 1 of the power distribution hierarchy) from historical consumption data. Figure 7 These techniques are discussed in more detail.

[0063] The power management service 450 can calculate a timing value for a timer (e.g., a timer that can be started and managed by (one or more) PDUs 402). The timing value can be calculated based at least in part on any suitable combination of: the rate of change of power consumption of a higher-level component, the direction of change of power consumption of the higher-level component (increase / decrease), the power peak tolerance of the higher-level component and / or the entire system 400, or the implementation time associated with the lower-level device (e.g., the assumed / known delay between the time when each of the (one or more) devices 404 is instructed to implement the power cap value and the time when each of the (one or more) devices 404 will actually implement the cap (e.g., the first time that power consumption at the device is constrained / throttled or at least a determination is made whether to constrain / throttle)).

[0064] The power management service 450 may transmit the calculated timing value to the PDU(s) 402 at any suitable time. In some embodiments, the power management service 450 may initially determine the power cap for a given higher-level component (e.g., a row component) without regard to consumption occurring with respect to other components of the same level (e.g., other row components). In some embodiments, while the timer is being initialized or elapsed, or at any suitable time, the power management service 450 may process consumption data corresponding to other same-level components to determine whether it is more desirable to power cap downstream components of one or more of those same-level components. In some embodiments, the power management service 450 may utilize priorities associated with the device(s) 404 and / or workloads running on those device(s) to determine the power cap value. The power management service 450 may be configured to prefer power capping of lower priority device(s) / workload(s) while leaving device(s) / workload(s) with higher priority values ​​unconstrained. In some embodiments, the power management service 450 can be configured to prefer power capping for a group of highest consuming devices (e.g., across a row, across a line, etc.) while allowing lower consuming devices to operate unconstrained. In some embodiments, if the priority associated with a device and / or workload is high (or higher than the priority associated with other devices and / or workloads), the specific consumption of the device can be made unconstrained even if the device is included in the group of highest consuming devices. Therefore, the power management service 450 can be configured to prioritize the priority of the device / workload over the power consumption of the particular device.

[0065] In some embodiments, power capping values ​​may be initially determined by the power management service for a given row. Those power capping values ​​may be provided to the power controller 448, which in turn may assign the power capping values ​​to the power controller 446 for storage at (one or more) devices 404. The power management service 450 may process consumption data about other row devices of the same row to determine power capping values ​​corresponding to devices of different rows. This may be advantageous because devices managed by another row may not consume their budgeted power, leaving some amount of unused power at the row-level device. In some embodiments, the power management service 450 may be configured to determine whether it may be more advantageous to cap the power of a row of devices while allowing at least some devices of another row to operate unconstrained. The benefits of determining a set of power capping values ​​may be based at least in part on minimizing the number of devices to be power capped, minimizing the priority values ​​associated with the devices to be power capped, maximizing the number of devices that will not be power capped that are associated with a specific high priority value, etc. The power management service 450 can utilize a predefined protocol (e.g., a set of rules) to determine whether implementation of a power cap value it has sent to one PDU is more favorable / beneficial or less favorable / beneficial than the value of a different power cap it has identified based on processing consumption data from multiple PDUs associated with one or more other rows.

[0066] The power management service 450 may be configured to implement (enact) the implementation of a power cap value group that is determined to be more favorable for ensuring that one or more circuit breakers in the data center are not tripped. In some embodiments, a group of power cap values ​​determined to be the most favorable may be selected. If the group of power caps previously provided to the power controller 448 is not determined to be more favorable, the power management service 450 may transmit data to the power controller 448 so that the power controller 448 transmits an indicator to (one or more) devices 404 (e.g., the power controller 446) to start power capping based on the previously assigned power cap. Alternatively, if the power management service 450 determines that a new group of power caps is more favorable (or most favorable) from a power management perspective, it may transmit data to the power controller 448 to cancel the timer. In some embodiments, canceling the timer may cause the previously assigned power caps to be deleted due to instructions from the power controller 448 (e.g., transmitted in response to canceling the timer), or those power caps may time out by default (e.g., according to a predefined time period). Along with an indication that the power cap is to be immediately enforced by the corresponding device, the power management service 450 can communicate the new power cap to the appropriate PDUs (which can include or exclude the same PDUs to which the previous set of power caps were communicated). These PDUs can communicate those power caps to the receiving devices along with instructions to immediately begin power capping operations (e.g., including monitoring consumption relative to the power cap value, determining whether to cap based on the power cap value and the current consumption of the device, and capping or unconstraining the operation of the device based on the determination).

[0067] In some embodiments, if the timer expires at the original PDU (e.g., a time period corresponding to the timer value elapses), and a cancellation from the power management system 450 has not been received, the power controller 448 can automatically instruct the device(s) 404 to begin power capping operation using the stored power capping value. This technique ensures fail-safe when the power management service 450 is unable to instruct the PDU or cancel the timer for any reason.

[0068] The above techniques enable more lower level devices to operate unconstrained by minimizing the amount and / or frequency at which the power consumption of those devices is capped. In addition, downstream devices of a given device in the hierarchy can be allowed to reach peaks that exceed the device's budgeted power while still ensuring that the device's maximum power capacity (a value above the budgeted value) is not exceeded. This enables the consumption of lower level devices to operate at consumption levels at higher level devices within a power buffer that is not utilized by traditional systems. The systems and techniques allow for more efficient utilization of the power distribution components of the system 400 and reduce waste while ensuring that power failures are avoided.

[0069] Figure 5is a flow chart illustrating an example method 500 for managing excessive power consumption according to at least one embodiment. The method 500 may include Figure 5 The operations shown or described may be more or fewer than the operations shown or described. The operations may be performed in any suitable order.

[0070] Method 500 may begin at 510, where PDU 504 may receive and / or obtain (eg, by requesting) consumption data from device(s) 502. Device(s) 502 may be Figure 4 504 . Figure 4 Examples of PDU(s) 402 (e.g., Figure 2 Each of the device(s) 502 may be a device to which the PDU 504 distributes power. The PDU(s) 508 may be examples of other rack PDUs associated with the same row as the PDU 504, and thus, power may be distributed from the PDUs in the same row as the PDU 504 (e.g., Figure 2 The row PDU 206 corresponds to Figure 3 Node 306 of (one or more) PDUs 504) receives power. The consumption received from (one or more) PDUs 508 may correspond to the power consumption of (one or more) devices 512. In some embodiments, (one or more) PDUs 508 and at least some portions of (one or more) devices 512 correspond to rows that are different from the rows corresponding to (one or more) devices 502. In some embodiments, any suitable number of instances of consumption data (referred to as "device consumption data") received by PDU 504 may be related to a single device in (one or more) devices 502. The same may also be true for the device consumption data received by (one or more) PDUs 508. In some embodiments, PDU 504 and (one or more) PDUs 508 may aggregate the received device consumption data instances and / or perform calculations therefrom to generate rack consumption data corresponding to each respective PDU. For example, PDU 505 may generate rack consumption data from device consumption data provided by (one or more) devices 502. The rack consumption data may include the device consumption data instances received by PDU 504 at 510. Similarly, rack consumption data provided by PDU(s) 508 may include device consumption instances received from device(s) 512. Rack consumption data generated by PDU 504 and / or PDU(s) 508 may be generated based at least in part on power consumption of PDU 504 and / or PDU(s) 508, respectively.

[0071] At 514, the power management service 506 ( Figure 4 An example of a power management service 450) can be provided from PDU 504 and (one or more) PDU 508 (also Figure 4 4 and 508). In some embodiments, power management service 506 may receive and / or obtain (e.g., via request) rack consumption data from PDU 504 and PDU(s) 508 on an ongoing basis. PDU 504 and PDU(s) 508 may similarly receive and / or obtain (e.g., via request) rack consumption data from device(s) 502 and 512, respectively (e.g., from corresponding power controllers of those devices, each of which is a Figure 4 The power controller 446 of the example receives and / or obtains (eg, via request) device consumption data. The PDU(s) 508 may correspond to and be associated with the same row of the PDU 504.

[0072] At 516, the power management service 506 can, based at least in part on a connection with another PDU (e.g., Figure 5 506 may determine whether to calculate a power cap value for device(s) 502 by using the maximum and / or budgeted power associated with row-level devices (not depicted in the diagram, such as buses) and consumption data received from PDU 504 and / or PDU(s) 508. In some embodiments, power management service 506 may access, from stored data, information associated with, for example, row-level PDUs (e.g., Figure 2The power management service 506 may aggregate rack consumption data from PDU 504 and / or PDU(s) 508 to determine cumulative power consumption values ​​for row devices (e.g., total power consumption from devices downstream of the row device over the last time window, changes in consumption rate from the perspective of the row device, directions of changes in consumption rate from the perspective of the row device, time period required to initiate power capping at one or more of the devices 502 and / or 512, etc.). In some embodiments, the power management service 506 may generate additional cumulative power consumption data from at least a portion of the cumulative power consumption data values ​​and / or using historical device consumption data (e.g., cumulative or individual consumption data). For example, the power management service 506 can calculate any suitable combination of the following items based at least in part on historical consumption data (e.g., historical device consumption data and / or historical rack consumption data corresponding to (one or more) devices 502 and / or 512): a change in consumption rate from the perspective of a row-level device, a direction of change in consumption rate from the perspective of a row-level device, etc.

[0073] If the cumulative power consumption (e.g., power consumption corresponding to device(s) 502 and 512) exceeds the budgeted amount of power associated with the row-level PDU, the power management service 506 can determine that a power cap value should be calculated. If the cumulative power consumption does not exceed the budgeted amount associated with the row-level PDU, the power management service 506 can determine that a power cap should not be calculated, and the method 500 can end. Alternatively, because the cumulative power consumption of the downstream devices (e.g., device(s) 502 and 512) exceeds the budgeted amount associated with the row-level PDU, the power management service can determine that a power cap should be calculated, and can proceed to 518.

[0074] In some embodiments, the power management service 506 may additionally or alternatively determine that a power cap value should be calculated at 516 based at least in part on providing historical consumption data (e.g., device consumption data and / or rack consumption data corresponding to (one or more) devices 502 and / or 512) as input data to one or more machine learning models. The (one or more) machine learning models may be trained using any suitable supervised or unsupervised machine learning algorithm to identify, from the historical consumption data provided as input, the likelihood that consumption corresponding to a device associated with the historical consumption data will exceed a budgeted amount (e.g., a budgeted amount of power allocated to row-level devices). The (one or more) machine learning models may be trained using a corresponding training data set that includes instances of historical consumption data. Figure 6 The training of these (one or more) machine learning models is discussed in more detail. Using the above-described techniques, the power management service 506 can identify that a power cap value is to be calculated based on the cumulative device / rack consumption level exceeding the budgeted power associated with the row-level devices and / or based on determining from the output of the (one or more) machine learning models that the cumulative device / rack consumption level is likely to exceed the budgeted power associated with the row-level devices. Determining that the device / rack consumption level is likely to exceed the budgeted power of the row-level devices can be determined based on receiving an output from the (one or more) machine learning models indicating a likelihood (e.g., likely / unlikely), or by comparing an output value (e.g., a percentage, a confidence value, etc.) indicating the likelihood of exceeding the budgeted power of the row-level devices to a predefined threshold. The output value indicating the likelihood of exceeding the predefined threshold can cause the power management service 506 to determine that power capping is warranted and that a power cap value is to be calculated.

[0075] At 518, the power management service 506 may calculate power capping values ​​for any suitable number of device(s) 502 and / or device(s) 512. For example, the power management service 506 may utilize device consumption data and / or rack consumption data corresponding to each of the device(s) 502 and 512. In some embodiments, the power management service 506 may determine the difference between the cumulative power consumption (calculated from aggregating the device and / or rack consumption data corresponding to the device(s) 502 and the device(s) 512 of the same row) of a row of devices (e.g., any of the device(s) 502 and the device(s) 512 of the same row) and the budgeted power associated with the row-level devices. The difference may be used to identify the amount of power consumption to be constrained at the devices associated with the row. The power management service 506 can determine power cap values ​​for any suitable combination of (one or more) devices 502 and / or 512 in the same row of devices, based on identifying that the power cap values, if implemented by the devices, will result in power consumption being reduced to a value that is less than the budgeted power associated with the row-level devices.

[0076] In some embodiments, the power management service 506 may determine a specific power cap value corresponding to any suitable combination of (one or more) devices 502 and 512 in the same row based at least in part on consumption data and / or priority values ​​associated with those devices and / or workloads associated with power consumption of those devices. In some embodiments, these priority values ​​may be provided as part of the device consumption data provided by the device to which the consumption data is related, or the priority values ​​may be determined at least in part based on the device type, workload type, etc. obtained from the consumption data or any other suitable source (e.g., from separate data accessible to the power management service 506). Using the priorities associated with the devices or workloads, respectively, the power management service 506 may calculate the power cap value in a manner that prefers power capping of devices associated with lower priority workloads over power capping of devices with higher priority workloads. In some embodiments, the power management service 506 may calculate the power cap value based at least in part on power capping of devices (e.g., a group of highest consuming devices) that consume at a higher rate over power capping of other devices that consume at a lower rate. In some embodiments, the power management service 506 can calculate a power cap value based at least in part on a combination of factors including a consumption value for each device and a priority associated with the device or a workload being executed by the device. The power management service 506 can identify power cap values ​​for devices that avoid fully capping high-consuming devices and / or high-priority devices or capping those devices to a lesser extent than devices that consume less power and / or are associated with lower priorities.

[0077] At 520, the power management service 520 may calculate a timing value corresponding to the duration of a timer to be initialized and managed by a PDU (e.g., PDU 504). The timing value may be calculated based at least in part on any suitable combination of the rate of change of power consumption from the perspective of the row-level device, the direction of change of power consumption from the perspective of the row-level device, or the time at which power capping is performed at each of the devices to which the power capping is associated calculated at 518. For example, the power management service 520 may determine a timing value based on determining a relatively large increase in power consumption from the perspective of the row-level device, which is greater than the timing value for a smaller increase in power consumption from the perspective of the row-level device. Therefore, the greater the increase in the rate of power consumption, the smaller the timing value (corresponding to a shorter timer), while a smaller increase in the rate of power consumption may result in a larger timing value (corresponding to a longer timer). Similarly, when the time at which power capping is performed at each of the devices to which the power capping is associated is shorter with respect to the calculation of the first timing value, the calculated first timing value may be less than the calculated second timing value. Therefore, the faster the device to which the power capping is envisioned to be capped is capped, the shorter the timing value may be.

[0078] At 522, the calculated power cap value calculated at 518 and / or the timing value calculated at 520 may be transmitted to the PDU 504. Although not shown, the corresponding power cap value and / or timing value for any of the device(s) 512 in the same row may be transmitted at 520.

[0079] At 524 , when the timing value is provided at 522 , the PDU 504 may use the timing value to initialize a timer having a duration corresponding to the timing value.

[0080] At 526, the PDU 504 may transmit the power cap value (if received at 522) to the device(s) 502. The device(s) 502 may store the received power cap value at 528. In some embodiments, the power cap value provided at 526 may include an indicator to not initiate enforcement, or the indicator may not be provided and the device(s) 502 may refrain from initiating power capping by default.

[0081] At 530, the power management service 506 may perform a higher level analysis of the device / rack consumption data received from any of the (one or more) devices 512 corresponding to a row different from the row to which the (one or more) devices 502 correspond. As part of this process, the power management service 506 may identify unutilized power associated with other row-level devices. If unutilized power exists, the power management service 506 may calculate a new set of power capping values ​​based at least in part on the consumption data corresponding to at least one other row of devices. As described above, this may be advantageous because devices managed by another row-level device may not consume their budgeted power, leaving unutilized power on the row-level device. In some embodiments, the power management service 450 may be configured to determine whether it is advantageous to cap the devices of another row while allowing at least some of the (one or more) devices 502 (and potentially some of the (one or more) devices 512 corresponding to the same row of the (one or more) devices 502) to operate unconstrained. The benefit of each power capping approach may be calculated based, at least in part, on minimizing the number of devices to be power capped, minimizing priority values ​​associated with devices to be power capped, maximizing the number of devices associated with a particular high priority value that will not be power capped, etc. The power management service 506 may utilize a predefined scheme or set of rules to determine whether implementation of a power capping value that has been determined for a row (e.g., corresponding to the power capping value communicated at 522) is more or less beneficial than a different power capping value that has been identified based on processing consumption data from multiple rows.

[0082] The power management service 506 may be configured to implement the set of power cap values ​​determined to be more favorable (or most favorable) to ensure that one or more circuit breakers within the data center are not tripped. If it is determined that the set of power caps previously provided to the PDU 504 is less favorable than the set of power cap values ​​calculated at 530, the power management service 450 may immediately send data to the PDU 504 to cause the PDU 504 to transmit an indicator to the device(s) 502 to immediately begin power capping based on the previously assigned power cap. Alternatively, the power management service 506 may take no further action, which may allow a timer at the PDU 504 to elapse.

[0083] Alternatively, if the power management service 450 determines that a new set of power caps is more advantageous (or most advantageous) from a power management perspective, it can transmit data to the power controller 448 to cancel the timer. Previously assigned power caps can be deleted by instructions from the power controller 448 (e.g., transmitted in response to canceling the timer), or those power caps can default to timeout according to a predefined time period. Along with an indication that the power cap will be immediately implemented by the corresponding device, the power management service 450 can transmit the new power cap to the appropriate PDU (which may include or exclude the same PDU to which the previous set of power caps were transmitted). These PDUs can transmit those power caps to the receiving device along with instructions to immediately begin power capping operations (e.g., including monitoring consumption about power cap values, determining whether to cap based on the power cap value and the current consumption of the device, and capping or unconstraining the operation of the device based on the determination).

[0084] If the power cap value calculated at 530 is determined to be more advantageous from a power management perspective than the power cap value calculated at 518 and ultimately stored in the device(s) 502 at 528, the method 500 may proceed to 532, where the power cap value calculated at 530 may be transmitted to the PDU(s) 508 (e.g., any of the PDU(s) 508 that manage the device(s) to which the power cap value is associated). In some embodiments, the power management service 506 may provide an indication that the power cap value transmitted at 532 is to be immediately implemented.

[0085] In response to receiving the power cap value and indication at 532, the PDU(s) 508 that allocate power to the devices to which those power cap values ​​are associated may transmit the power cap value and indication to the devices to which those power cap values ​​are associated. At 536, the receiving device(s) 512 may perform power capping operations without delay based on receiving the indication. This includes 1) determining whether to limit / restrict operation based on the current consumption of a given device when compared to the power cap value provided to the device, and 2) limiting / restricting or refraining from limiting / restricting the power consumption of the device.

[0086] In some embodiments, the power management service 506 may transmit data canceling the timer and / or power cap value transmitted at 522 based at least in part on identifying that the power cap value calculated at 530 is more favorable (or most favorable) than the power cap value calculated at 518. In some embodiments, the PDU 504 may be configured to cancel the timer at 540. In some embodiments, the PDU 504 may transmit data at 542 that causes the device(s) 502 to delete the power cap value from memory at 544.

[0087] At 546, in the event that a more favorable set of power capping values ​​is not found (as described above) or the power management service 506 has not transmitted a cancellation (as described above in connection with 538), the PDU 504 can identify that the timer started at 524 has expired (e.g., a duration corresponding to the timing value provided at 522 has elapsed). Accordingly, the PDU 504 can transmit an indication to the device(s) 502 at 548, indicating that the device(s) 502 begin power capping based on the power capping value stored in the device(s) 502 at 528. Receiving this indication at 548 can cause the device(s) 502 to begin power capping at 550 according to the power capping value stored at 528.

[0088] In some embodiments, if PDU 504 identifies that power consumption of device(s) 502 has decreased, PDU 504 may be configured to cancel the timer at any suitable time after the timer is started at 524. In some embodiments, this may cause PDU 504 to transmit data to device(s) 502 to cause device(s) 502 to discard previously stored power capping values ​​from local memory.

[0089] In some embodiments, a power cap value being implemented at any suitable device may time out or may be replaced by a power cap value calculated by the power management service 506 at any suitable time. In some embodiments, the power management service 506 may transmit the cancellation or replacement of a power cap value to any suitable device via its corresponding rack PDU. If a cancellation of a power cap value is received, the device may delete the previously stored power cap value, allowing the device to resume unconstrained operation. If a replacement power cap value is received, the device may store a new power cap value and implement the new power cap value immediately or when indicated by its rack PDU. In some embodiments, the implementation of a new power cap value indicated by its PDU may cause the device (e.g., the power controller 446) of the device to replace the power cap value previously used for power capping with the new power cap value that the device is instructed to implement.

[0090] Any suitable operations of method 500 may be performed on an ongoing basis at any suitable time to manage excessive consumption (e.g., where consumption of servers corresponding to row-level devices exceeds budgeted power for the row-level devices) to enable more efficient use of previously unutilized power while avoiding power outages due to tripped circuit breakers. It should be understood that power management service 506 may perform similar operations with respect to any suitable level of the power distribution hierarchy 300. Although examples have been provided with respect to monitoring power consumption corresponding to row-level devices (e.g., devices represented by level 3 nodes of the power distribution hierarchy 300), similar operations may be performed with respect to any suitable level of higher-level devices (e.g., components corresponding to any of levels 2-5 of the power distribution hierarchy 300).

[0091] Figure 6 An illustration depicting a method for training a machine learning model to determine a downstream device (e.g., Figure 2 Flow of an example method 600 for determining the likelihood that an aggregate power consumption of device(s) 502 will exceed a budget threshold corresponding to an upstream device (e.g., a row-level device through which power is allocated to device(s) 502).

[0092] In some embodiments, model(s) 602 may be (e.g., by Figure 5 The power management service 506 or a different device) is trained using any suitable machine learning algorithm (e.g., supervised, unsupervised, etc.) and any suitable number of training data sets (e.g., training data 608). Supervised machine learning algorithms refer to machine learning tasks that include learning an inference function that maps inputs to outputs based on a labeled training data set (for which example input / output pairs are known). Unsupervised machine learning algorithms refer to a set of algorithms for analyzing and clustering unlabeled data sets (e.g., unlabeled data 610). These algorithms are configured to recognize patterns or data groupings without human intervention. In some embodiments, any suitable number of (one or more) models 602 can be trained during the training phase 604.

[0093] The model(s) 602 may include any suitable number of models. The model(s) 602 may be trained to identify, from training data 608 discussed below, the likelihood that consumption data associated with a downstream device (e.g., a device to which power is distributed via a row-level device) will breach a budget threshold (e.g., a budgeted amount of power) associated with an upstream device (e.g., a higher-level device such as a row-level device). In some embodiments, the model(s) 602 may be configured to determine and output one or more predicted consumption amounts that are likely to occur in the future, and the determination of whether the predicted amount(s) breaches the budget threshold (e.g., a budgeted amount of power associated with the upstream device) may be determined by Figure 4 The power management service 450 is made.

[0094] As a non-limiting example, at least one of the (one or more) models 602 may be trained using a supervised learning algorithm and labeled data 606 during a training phase 604 to identify a likelihood value and / or one or more predicted consumption values ​​(e.g., predicted aggregate consumption / cumulative consumption of one or more devices). The likelihood value may be a binary indicator, a percentage, a confidence value, etc. indicating a degree of likelihood. The likelihood value may be a binary indicator indicating that a particular budget amount is likely or unlikely to be breached, or the likelihood value may indicate the likelihood that a particular predicted consumption value will be experienced in the future. The labeled data 606 may be any suitable portion of potential training data (e.g., training data 608) that may be used to train various models to likelihood values. The labeled data 606 may include any suitable number of examples of historical consumption data corresponding to devices and / or racks to which power is distributed by row-level devices. In some embodiments, the labeled data 606 may include tags that identify known likelihood values. Using the labeled data 606, a model (e.g., an inference function) that maps an input (e.g., one or more instances of historical consumption data corresponding to one or more devices) to an output (e.g., the likelihood of a consumption exceeding amount for one or more devices) may be learned. In some embodiments, the amount to be used may be included in the training data 608 and included as an input. In some embodiments, the model(s) 602 may provide as an output the amount by which the cumulative consumption from the perspective of a given component (e.g., a row-level device) is expected to change and a likelihood or confidence value indicating the likelihood / confidence corresponding to the amount. In some embodiments, Figure 5 The power management service 506 may determine that output(s) 612 including an amount and / or likelihood value indicate that consumption of downstream devices associated with the row level device is likely (eg, exceeding a threshold level) to exceed a power budget for the row level device.

[0095] Model(s) 602 and the various types of those models discussed above may include any suitable number of models that are trained using unsupervised learning techniques to identify the likelihood / confidence that downstream devices associated with row-level devices will exceed the budgeted power associated with row-level devices. Alternatively, unsupervised learning techniques may be used to identify the amount by which the consumption of downstream devices is expected to increase (and potentially the corresponding likelihood / confidence). Unsupervised machine learning algorithms are configured to learn patterns from unlabeled data. In some embodiments, the training phase 604 may utilize an unsupervised machine learning algorithm to generate one or more models. For example, the training data 608 may include unlabeled data 610 (e.g., historical consumption data instances corresponding to devices and / or device racks receiving power through a given row-level device). The unlabeled data 610 may be used together with an unsupervised learning algorithm to segment entries of the unlabeled data 610 into groups. The unsupervised learning algorithm may be configured so that similar entries are grouped together into a common group. Examples of unsupervised learning algorithms may include clustering methods such as k-means clustering, DBScan, and the like. In some embodiments, unlabeled data 610 may be clustered with labeled data 606 such that unlabeled instances of a given group may be assigned the same label as other labeled instances within the group.

[0096] In some embodiments, any suitable portion of the training data 608 may be used to train the model(s) 602 during the training phase 604. For example, 70% of the labeled data 606 and / or the unlabeled data 610 may be used to train the model(s) 602. Once trained, or at any suitable time, the model(s) 602 may be evaluated to assess its quality (e.g., the accuracy of the output(s) 612 with respect to the labels corresponding to the labeled data 606). For example, a portion of an example of the labeled data 606 and / or the unlabeled data 610 may be used as an input to the model(s) 602 to generate the output(s) 612. For example, an example of the labeled data 606 may be provided as an input, and the corresponding output (e.g., the output(s) 612) may be compared to labels that are already known to be associated with the example. If some portion of the output (e.g., the labels) matches the example labels, then this portion of the output may be considered accurate. Any suitable number of labeled examples may be utilized, and the amount of accurate labels may be compared to the total number of examples provided (and / or the total number of previously identified labels) to determine an accuracy value for a given model, which quantifies how accurate the model is. For example, if 90 of 100 input examples generate output labels that match previously known example labels, the model being evaluated is determined to be 90% accurate.

[0097] In some embodiments, when the model(s) 602 are used for subsequent inputs, subsequent outputs generated by the model(s) 602 may be added to the corresponding inputs and used to retrain and / or update the model(s) 602 at 616. In some embodiments, examples may not be used to retrain or update models until a feedback procedure 614 is executed. In the feedback procedure 614, examples (e.g., examples including one or more historical consumption data instances corresponding to one or more devices and / or racks) and corresponding outputs generated by one of the model(s) 602 for the examples are presented to a user, and the user identifies whether the generated outputs (e.g., quantities and / or likelihood confidence values) are correct for a given example.

[0098] Figure 6 The training process shown in (eg, method 600 ) may be performed any suitable number of times, at any suitable intervals and / or according to any suitable schedule, such that the accuracy of any suitable model(s) 602 improves over time.

[0099] In some embodiments, any suitable number of model(s) 602 and / or combination of model(s) 602 may be used to determine outputs. In some embodiments, power management service 450 may utilize any suitable combination of outputs provided by model(s) 602 to determine whether a budget threshold for a given component (e.g., a row-level PDU) is likely to be breached (and / or likely to be breached by some amount). Thus, in some embodiments, power management service 450 may utilize models that have been trained using any suitable combination of supervised and unsupervised learning algorithms.

[0100] Figure 7 is a block diagram illustrating an example method for managing power according to at least one embodiment. Figure 4 The method 700 may be performed at least in part by one or more components of the system 400. For example, the method 700 may be performed at least in part by Figure 4 The operations of method 700 may be performed in any suitable order. Figure 7 More or fewer operations may be performed as shown.

[0101] At 702, a plurality of components of a power system (eg, system 400) may be identified. In some embodiments, the plurality of components may be organized according to a power distribution hierarchy (eg, Figure 3In some embodiments, a node in the plurality of nodes of the power distribution hierarchy represents a corresponding component in the plurality of components. In some embodiments, a subset of nodes (e.g., a first level) of the plurality of levels (e.g., level 1) (e.g., Figure 3 The nodes 314 and 316 may be selected from a specific node (eg, a row level) at a second level (eg, a row level) higher than the first level among the multiple levels. Figure 3 In some embodiments, a group of lower level components in the plurality of components represented by a subset of nodes in the first level receive power distributed through higher level components in the plurality of components represented by a particular node in the second level.

[0102] At 704, power consumption of a set of lower level components represented by the subset of nodes at the first level may be monitored. In some embodiments, monitoring power consumption may include receiving device and / or rack consumption data, as described above with respect to Figure 5 The method 500 is described in detail.

[0103] At 706, it can be determined, based at least in part on the monitoring, whether power consumption of a set of lower-level components has breached a budget threshold associated with a higher-level component. In some embodiments, the budget threshold can be considered breached when power consumption of a set of lower-level components exceeds the budgeted amount of power allocated to the higher-level component. In some embodiments, the budget threshold can be considered breached when power consumption of a set of lower-level components approaches (e.g., is within a threshold) the budgeted amount of power allocated to the higher-level component. In some embodiments, the determination that the budget threshold has been breached can be as described above with respect to Figure 5 516 relates to the method described herein for utilizing historical consumption data.

[0104] In response to determining that power consumption of a set of lower-level components has breached a budget threshold associated with a higher-level component, a power cap value for the lower-level component may be transmitted at 708. Transmission of the power cap value may cause the lower-level component to store the power cap value in a memory while allowing the corresponding power consumption of the lower-level component to exceed the power cap value until a time period corresponding to the timing value expires.

[0105] Figure 8 is a block diagram illustrating another example method 800 for managing power according to at least one embodiment. The method 800 may be Figure 4 The method 800 may be performed at least in part by one or more components of the system 400. For example, the method 800 may be performed at least in part by Figure 4 The operations of method 800 may be performed in any suitable order. Figure 8 More or fewer operations may be performed as shown.

[0106] At 802, a plurality of components of a power system (eg, system 400) may be identified. In some embodiments, the plurality of components are organized according to a power distribution hierarchy (eg, Figure 3 In some embodiments, a node in the plurality of nodes of the power distribution hierarchy represents a corresponding component in the plurality of components. In some embodiments, a subset of nodes (e.g., a first level (e.g., level 1) of the plurality of levels (e.g., Figure 3 The nodes 314 and 316 may be selected from a specific node (eg, a row level) at a second level (eg, a row level) higher than the first level among the multiple levels. Figure 3 In some embodiments, a group of lower level components represented by a subset of nodes at a first level of the plurality of components receive power distributed through a higher level component represented by a particular node at a second level of the plurality of components.

[0107] At 804, power consumption of a set of lower level components represented by a subset of nodes at the first level may be monitored. In some embodiments, monitoring power consumption may include receiving device and / or rack consumption data, as described above with respect to Figure 5 The method 500 is described in detail.

[0108] At 806, it can be determined, based at least in part on the monitoring, whether power consumption of a set of lower-level components has breached a budget threshold associated with a higher-level component. In some embodiments, the budget threshold can be considered breached when power consumption of the set of lower-level components exceeds the budgeted amount of power allocated to the higher-level component. In some embodiments, the budget threshold can be considered breached when power consumption of the set of lower-level components approaches (e.g., is within a threshold) the budgeted amount of power allocated to the higher-level component. In some embodiments, the determination that the budget threshold has been breached can be as described above with respect to Figure 5 516 relates to the method described herein for utilizing historical consumption data.

[0109] At 808, in response to determining that power consumption of a group of lower-level components has breached a budget threshold associated with a higher-level component, a timer corresponding to a timing value may be initiated. In some embodiments, expiration of the timer indicates expiration of a time period corresponding to the timing value. Figure 4 The server corresponding to the (one or more) device 404 can store the power cap value in the memory during the time period. In some embodiments, the enforcement of the power cap value for the lower-level component is delayed until the timer expires.

[0110] Figure 9-13A number of example environments are depicted that may be hosted by components (eg, device(s) 404) of a data center. Figure 9-13 The environment depicted in the description depicts a cloud computing, multi-tenant environment. As described above, cloud computing and / or multi-tenant environments and other environments can benefit from utilizing the power management techniques disclosed herein. These techniques enable implementation of Figure 9-13 A data center with components of the described environment and the like is able to utilize the power resources of the data center more efficiently than conventional techniques that leave a large amount of power unutilized.

[0111] Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), or the like). In some cases, IaaS providers can also supply a variety of services to accompany those infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, because these services can be policy-driven, IaaS users can be able to implement policies to drive load balancing to maintain application availability and performance.

[0112] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet, and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create a virtual machine (VM), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise software into the VM. Customers can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application problems, monitoring performance, managing disaster recovery, and more.

[0113] In most cases, the cloud computing model will require the involvement of a cloud provider. However, a cloud provider is not necessarily a third-party service that specializes in providing (e.g., supplying, renting, selling) IaaS. An entity may also choose to deploy a private cloud and become its own infrastructure service provider.

[0114] In some examples, IaaS deployment is the process of putting a new application or a new version of an application onto a prepared application server or the like. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling (OS), middleware, and / or application deployment (e.g., on a self-service virtual machine (e.g., which can be spun up on demand) or the like).

[0115] In some examples, IaaS provisioning can refer to obtaining a computer or virtual host for use, and even installing required libraries or services on it. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0116] In some cases, there are two distinct challenges with IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before running anything. Second, after everything has been provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both of these challenges can be addressed by enabling the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., what components are required and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how each of them works together) can be described declaratively. In some cases, after the topology is defined, workflows can be generated to create and / or manage the different components described in the configuration files.

[0117] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs) (e.g., a potential on-demand pool of configurable and / or shared computing resources), also referred to as a core network. In some examples, there can also be one or more inbound / outbound traffic group rules and one or more virtual machines (VMs) that are provided to define how inbound and / or outbound traffic for the network will be established. Other infrastructure elements such as load balancers, databases, or the like can also be provided. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve gradually.

[0118] In some cases, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. In addition, the described techniques may enable infrastructure management within these environments. In some examples, a service team may write code that is expected to be deployed to one or more (but typically multiple) different production environments (e.g., across various geographic locations, sometimes even across the world). However, in some examples, the infrastructure on which the code will be deployed must first be established. In some cases, provisioning may be done manually, provisioning tools may be utilized to provision resources, and / or once the infrastructure is provisioned, deployment tools may be utilized to deploy the code.

[0119] Fig. 9 900 is a block diagram illustrating an example mode of an IaaS architecture according to at least one embodiment. A service operator 902 may be communicatively coupled to a secure host lease 904 that may include a virtual cloud network (VCN) 906 and a secure host subnet 908. In some examples, the service operator 902 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays), running software (such as Microsoft Windows Mobile®) and / or a variety of mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and enabling Internet, email, short message service (SMS), Blackberry® or other communication protocols. Alternatively, the client computing device may be a general-purpose personal computer, including, for example, a personal computer and / or laptop computer running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device may be a workstation computer running any of a variety of commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as, for example, Google Chrome OS). Alternatively or in addition, the client computing device may be any other electronic device capable of communicating over a network that can access VCN 906 and / or the Internet, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device), and / or a personal messaging device.

[0120] The VCN 906 may include a local peer gateway (LPG) 910, which may be communicatively coupled to a secure shell (SSH) VCN 912 via the LPG 910 contained in a SSH VCN 912. The SSH VCN 912 may include an SSH subnet 914, and the SSH VCN 912 may be communicatively coupled to a control plane VCN 916 via the LPG 910 contained in a control plane VCN 916. Furthermore, the SSH VCN 912 may be communicatively coupled to a data plane VCN 918 via the LPG 910. The control plane VCN 916 and the data plane VCN 918 may be contained in a service lease 919, which may be owned and / or operated by an IaaS provider.

[0121] The control plane VCN 916 may include a control plane demilitarized zone (DMZ) layer 920, which acts as a perimeter network (e.g., the portion of a corporate network between a corporate intranet and an external network). DMZ-based servers may have limited liability and help keep breaches contained. In addition, the DMZ layer 920 may include one or more load balancer (LB) subnets 922, a control plane application layer 924 that may include (one or more) application subnets 926, and a control plane data layer 928 that may include (one or more) database (DB) subnets 930 (e.g., (one or more) front-end DB subnets and / or (one or more) back-end DB subnets). The LB subnet(s) 922 included in the control plane DMZ layer 920 may be communicatively coupled to the application subnet(s) 926 included in the control plane application layer 924 and the Internet gateway 934 that may be included in the control plane VCN 916, and the application subnet(s) 926 may be communicatively coupled to the DB subnet(s) 930 included in the control plane data layer 928 and the service gateway 936 and the network address translation (NAT) gateway 938. The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0122] The control plane VCN 916 may include a data plane mirror application layer 940, which may include application subnet(s) 926. The application subnet(s) 926 contained in the data plane mirror application layer 940 may include a virtual network interface controller (VNIC) 942, which may execute a compute instance 944. The compute instance 944 may communicatively couple the application subnet(s) 926 of the data plane mirror application layer 940 to the application subnet(s) 926 that may be contained in the data plane application layer 946.

[0123] The data plane VCN 918 may include a data plane application layer 946, a data plane DMZ layer 948, and a data plane data layer 950. The data plane DMZ layer 948 may include LB subnet(s) 922 communicatively coupled to the application subnet(s) 926 of the data plane application layer 946 and the internet gateway 934 of the data plane VCN 918. The application subnet(s) 926 may be communicatively coupled to the service gateway 936 of the data plane VCN 918 and the NAT gateway 938 of the data plane VCN 918. The data plane data layer 950 may also include DB subnet(s) 930 communicatively coupled to the application subnet(s) 926 of the data plane application layer 946.

[0124] The Internet gateway 934 of the control plane VCN 916 and the data plane VCN 918 may be communicatively coupled to the metadata management service 952, which may be communicatively coupled to the public Internet 954. The public Internet 954 may be communicatively coupled to the NAT gateway 938 of the control plane VCN 916 and the data plane VCN 918. The service gateway 936 of the control plane VCN 916 and the data plane VCN 918 may be communicatively coupled to the cloud service 956.

[0125] In some examples, a service gateway 936 of a control plane VCN 916 or a data plane VCN 918 may make an application programming interface (API) call to a cloud service 956 without traversing the public Internet 954. The API call from the service gateway 936 to the cloud service 956 may be one-way: the service gateway 936 may make the API call to the cloud service 956, and the cloud service 956 may send the requested data to the service gateway 936. However, the cloud service 956 may not initiate the API call to the service gateway 936.

[0126] In some examples, secure host lease 904 can be directly connected to service lease 919, which would otherwise be isolated. Secure host subnet 908 can communicate with SSH subnet 914 through LPG 910, which can enable two-way communication through otherwise isolated systems. Connecting secure host subnet 908 to SSH subnet 914 can give secure host subnet 908 access to other entities within service lease 919.

[0127] The control plane VCN 916 may allow users of the service lease 919 to establish or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 916 may be deployed or otherwise used in the data plane VCN 918. In some examples, the control plane VCN 916 may be isolated from the data plane VCN 918, and the data plane mirror application layer 940 of the control plane VCN 916 may communicate with the data plane application layer 946 of the data plane VCN 918 via a VNIC 942 that may be included in the data plane mirror application layer 940 and the data plane application layer 946.

[0128] In some examples, a user or customer of the system may make a request (e.g., a create, read, update, or delete (CRUD) operation) over the public internet 954, which may communicate the request to the metadata management service 952. The metadata management service 952 may communicate the request to the control plane VCN 916 through the internet gateway 934. The request may be received by the LB subnet(s) 922 contained in the control plane DMZ layer 920. The LB subnet(s) 922 may determine that the request is valid, and in response to this determination, the LB subnet(s) 922 may communicate the request to the application subnet(s) 926 contained in the control plane application layer 924. If the request is validated and a call to the public internet 954 is required, the call to the public internet 954 may be communicated to the NAT gateway 938, which may make the call to the public internet 954. Metadata that may be desired to be stored by the request may be stored in the DB subnet(s) 930.

[0129] In some examples, the data plane mirror application layer 940 may facilitate direct communication between the control plane VCN 916 and the data plane VCN 918. For example, it may be desirable to apply changes, updates, or other suitable modifications to the configuration to the resources contained in the data plane VCN 918. The control plane VCN 916 may communicate directly with the resources contained in the data plane VCN 918 via the VNIC 942 and may thereby perform changes, updates, or other suitable modifications to the configuration of the resources contained in the data plane VCN 918.

[0130] In some embodiments, the control plane VCN 916 and the data plane VCN 918 may be included in the service lease 919. In this case, the user or customer of the system may not own or operate the control plane 916 or the data plane VCN 918. Instead, the IaaS provider may own or operate the control plane VCN 916 and the data plane VCN 918, both of which may be included in the service lease 919. This embodiment may enable network isolation that may prevent users or customers from interacting with other users or other customers' resources. In addition, this embodiment may allow users or customers of the system to store databases privately without relying on the public Internet 954, which may not have the desired level of threat prevention.

[0131] In other embodiments, the LB subnet(s) 922 contained in the control plane VCN 916 may be configured to receive signals from the service gateway 936. In this embodiment, the control plane VCN 916 and the data plane VCN 918 may be configured to be called by customers of the IaaS provider without calling the public Internet 954. Customers of the IaaS provider may desire this embodiment because the database(s) used by the customer may be controlled by the IaaS provider and may be stored on a service lease 919 that may be isolated from the public Internet 954.

[0132] Fig.10 1 is a block diagram 1000 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 1002 (e.g., Fig. 9 The service operator 902) may be communicatively coupled to the secure host lease 1004 (e.g., Fig. 9 904), which may include a virtual cloud network (VCN) 1006 (e.g., Fig. 9 VCN 906) and secure host subnet 1008 (e.g., Fig. 9 The VCN 1006 may include a local peer gateway (LPG) 1010 (e.g., Fig. 9 910 ), which may be communicatively coupled to a secure shell (SSH) VCN 1012 (e.g., Fig. 9 SSH VCN 1012 may include SSH subnet 1014 (e.g., Fig. 9 SSH subnet 914), and SSH VCN 1012 may be communicatively coupled to control plane VCN 1016 via LPG 1010 included in control plane VCN 1016 (e.g., Fig. 9The control plane VCN 916). The control plane VCN 1016 may be included in a service lease 1019 (e.g., Fig. 9 service lease 919), and the data plane VCN 1018 (e.g., Fig. 9 The data plane VCN 918) may be contained in a customer lease 1021 which may be owned or operated by a user or customer of the system.

[0133] The control plane VCN 1016 may include (one or more) LB subnets 1022 (e.g., Fig. 9 The control plane DMZ layer 1020 (e.g., Fig. 9 control plane DMZ layer 920), may include (one or more) application subnets 1026 (e.g., Fig. 9 The control plane application layer 1024 (e.g., Fig. 9 The control plane application layer 924 of FIG. 100 may include (one or more) database (DB) subnets 1030 (e.g., similar to Fig. 9 The control plane data layer 1028 (e.g., Fig. 9 The LB subnet(s) 1022 contained in the control plane DMZ layer 1020 may be communicatively coupled to the application subnet(s) 1026 contained in the control plane application layer 1024 and the internet gateway 1034 (e.g., Fig. 9 934), and the application subnet(s) 1026 may be communicatively coupled to the DB subnet(s) 1030 and service gateway 1036 (e.g., Fig. 9 Serving gateway 936) and network address translation (NAT) gateway 1038 (e.g., Fig. 9 The control plane VCN 1016 may include a service gateway 1036 and a NAT gateway 1038.

[0134] The control plane VCN 1016 may include a data plane mirror application layer 1040 (e.g., Fig. 9 The data plane image application layer 940 of FIG. 1 may include (one or more) application subnets 1026. The (one or more) application subnets 1026 included in the data plane image application layer 1040 may include a computing instance 1044 (e.g., similar to Fig. 9The compute instance 1044 may include a virtual network interface controller (VNIC) 1042 (e.g., VNIC 942 ) of the compute instance 944 . The compute instance 1044 may facilitate the application subnet(s) 1026 of the data plane mirroring application layer 1040 and the application subnets 1026 that may be included in the data plane application layer 1046 (e.g., Fig. 9 Communication between (one or more) application subnets 1026 in the data plane application layer 946) via VNIC 1042 contained in the data plane mirror application layer 1040 and VNIC 1042 contained in the data plane application layer 1046.

[0135] The Internet gateway 1034 included in the control plane VCN 1016 can be communicatively coupled to the metadata management service 1052 (e.g., Fig. 9 1052 ), which can be communicatively coupled to a public Internet 1054 (e.g., Fig. 9 The public Internet 1054 can be communicatively coupled to a NAT gateway 1038 included in the control plane VCN 1016. The service gateway 1036 included in the control plane VCN 1016 can be communicatively coupled to a cloud service 1056 (e.g., Fig. 9 Cloud services 956).

[0136] In some examples, the data plane VCN 1018 may be contained in a customer lease 1021. In this case, the IaaS provider may provide a control plane VCN 1016 for each customer, and the IaaS provider may establish a unique compute instance 1044 contained in a service lease 1019 for each customer. Each compute instance 1044 may allow communication between the control plane VCN 1016 contained in the service lease 1019 and the data plane VCN 1018 contained in the customer lease 1021. The compute instance 1044 may allow resources provisioned in the control plane VCN 1016 contained in the service lease 1019 to be deployed or otherwise used in the data plane VCN 1018 contained in the customer lease 1021.

[0137] In other examples, a customer of the IaaS provider may have a database that exists in the customer tenancy 1021. In this example, the control plane VCN 1016 may include a data plane mirror application layer 1040, which may include (one or more) application subnets 1026. The data plane mirror application layer 1040 may reside in the data plane VCN 1018, but the data plane mirror application layer 1040 may not exist in the data plane VCN 1018. That is, the data plane mirror application layer 1040 may have access to the customer tenancy 1021, but the data plane mirror application layer 1040 may not exist in the data plane VCN 1018 or be owned or operated by the customer of the IaaS provider. The data plane mirror application layer 1040 may be configured to make calls to the data plane VCN 1018, but may not be configured to make calls to any entity contained in the control plane VCN 1016. A customer may desire to deploy or otherwise use resources provisioned in control plane VCN 1016 in data plane VCN 1018, and data plane mirror application layer 1040 may facilitate the customer's desired deployment or other use of the resources.

[0138] In some embodiments, a customer of an IaaS provider may apply filters to the data plane VCN 1018. In this embodiment, the customer may determine what the data plane VCN 1018 may access, and the customer may restrict access to the public Internet 1054 from the data plane VCN 1018. The IaaS provider may not apply filters or otherwise control access to any external network or database by the data plane VCN 1018. The customer applies filters and controls to the data plane VCN 1018 contained in the customer lease 1021, which can help isolate the data plane VCN 1018 from other customers and from the public Internet 1054.

[0139] In some embodiments, cloud services 1056 may be called by service gateway 1036 to access services that may not exist on the public Internet 1054, on control plane VCN 1016, or on data plane VCN 1018. The connection between cloud services 1056 and control plane VCN 1016 or data plane VCN 1018 may not be active or continuous. Cloud services 1056 may exist on different networks owned or operated by IaaS providers. Cloud services 1056 may be configured to receive calls from service gateway 1036 and may be configured not to receive calls from the public Internet 1054. Some cloud services 1056 may be isolated from other cloud services 1056, and control plane VCN 1016 may be isolated from cloud services 1056 that may not be in the same region as control plane VCN 1016. For example, control plane VCN 1016 may be located in "Region 1", while cloud service "Deployment 9" may be located in Region 1 and "Region 2". If the call to deployment 9 is made by a service gateway 1036 contained in a control plane VCN 1016 located in region 1, the call may be transmitted to deployment 9 in region 1. In this example, the control plane VCN 1016 or deployment 9 in region 1 may not be communicatively coupled to deployment 9 in region 2, or may otherwise be in communication with deployment 9 in region 2.

[0140] Fig.11 1 is a block diagram 1100 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 1102 (e.g., Fig. 9 The service operator 902) may be communicatively coupled to the secure host lease 1104 (e.g., Fig. 9 The secure host lease 1104 may include a virtual cloud network (VCN) 1106 (e.g., Fig. 9 VCN906) and secure host subnet 1108 (e.g., Fig. 9 VCN 1106 may include LPG 1110 (e.g., Fig. 9 110 ), which may be communicatively coupled to the SSH VCN 1112 via the LPG 1110 contained in the SSH VCN 1112 (e.g., Fig. 9 SSH VCN 1112 may include SSH subnet 1114 (e.g., Fig. 9 SSH subnet 914), and SSH VCN 1112 may be communicatively coupled to control plane VCN 1116 via LPG 1110 included in control plane VCN 1116 (e.g., Fig. 9The control plane VCN 916 of the present invention is communicatively coupled to the data plane VCN 1118 via the LPG 1110 contained in the data plane VCN 1118 (e.g., Fig. 9 The control plane VCN 1116 and the data plane VCN 1118 may be included in a service lease 1119 (e.g., Fig. 9 Service lease 919).

[0141] The control plane VCN 1116 may include a load balancer (LB) subnet 1122 (e.g., Fig. 9 The control plane DMZ layer 1120 (e.g., Fig. 9 The control plane DMZ layer 920 may include (one or more) application subnets 1126 (e.g., similar to Fig. 9 The control plane application layer 1124 (e.g., Fig. 9 of the control plane application layer 924), which may include a control plane data layer 1128 of (one or more) DB subnets 1130 (e.g., Fig. 9 1128). The LB subnet(s) 1122 contained in the control plane DMZ layer 1120 may be communicatively coupled to the application subnet(s) 1126 contained in the control plane application layer 1124 and to the Internet gateway 1134 (e.g., Fig. 9 1124 ), and the application subnet(s) 1126 may be communicatively coupled to the DB subnet(s) 1130 contained in the control plane data layer 1128 and to the service gateway 1136 (e.g., Fig. 9 Serving gateway) and network address translation (NAT) gateway 1138 (e.g., Fig. 9 The control plane VCN 1116 may include a service gateway 1136 and a NAT gateway 1138.

[0142] Data plane VCN 1118 may include a data plane application layer 1146 (e.g., Fig. 9 Data plane application layer 946), data plane DMZ layer 1148 (e.g., Fig. 9 Data plane DMZ layer 948), and data plane data layer 1150 (eg, Fig. 9The data plane DMZ layer 1148 may include (one or more) LB subnets 1122 that may be communicatively coupled to the trusted application subnets 1160 and (one or more) untrusted application subnets 1162 of the data plane application layer 1146 and the internet gateway 1134 included in the data plane VCN 1118. The trusted application subnets 1160 may be communicatively coupled to the service gateway 1136 included in the data plane VCN 1118, the NAT gateway 1138 included in the data plane VCN 1118, and the DB subnets 1130 included in the data plane data layer 1150. The untrusted application subnets 1162 may be communicatively coupled to the service gateway 1136 included in the data plane VCN 1118 and the DB subnets 1130 included in the data plane data layer 1150. Data plane data layer 1150 may include DB subnet(s) 1130 , which may be communicatively coupled to a service gateway 1136 included in data plane VCN 1118 .

[0143] The untrusted application subnet(s) 1162 may include one or more primary VNICs 1164(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1166(1)-(N). Each tenant VM 1166(1)-(N) may be communicatively coupled to a respective application subnet 1167(1)-(N) that may be contained in a respective container egress VCN 1168(1)-(N), which may be contained in a respective customer tenancy 1170(1)-(N). Respective secondary VNICs 1172(1)-(N) may facilitate communications between the untrusted application subnet(s) 1162 contained in the data plane VCN 1118 and the application subnets contained in the container egress VCNs 1168(1)-(N). Each container egress VCN 1168(1)-(N) may include a NAT gateway 1138 that may be communicatively coupled to the public Internet 1154 (e.g., Fig. 9 of the public Internet 954).

[0144] The Internet gateway 1134 included in the control plane VCN 1116 and included in the data plane VCN 1118 can be communicatively coupled to the metadata management service 1152 (e.g., Fig. 9The metadata management service 1152 may be communicatively coupled to the public Internet 1154. The public Internet 1154 may be communicatively coupled to the NAT gateway 1138 included in the control plane VCN 1116 and included in the data plane VCN 1118. The service gateway 1136 included in the control plane VCN 1116 and included in the data plane VCN 1118 may be communicatively coupled to the cloud service 1156.

[0145] In some embodiments, data plane VCN 1118 can be integrated with customer lease 1170. In some cases, such integration can be useful or desirable for customers of the IaaS provider, such as where support may be desired when executing code. A customer may provide code to run that may be destructive, may communicate with other customer resources, or may otherwise cause adverse effects. In response to this, the IaaS provider can determine whether to run the code given to the IaaS provider by the customer.

[0146] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider and request a function to be attached to the data plane application layer 1146. Code running the function may be executed in a VM 1166(1)-(N), and the code may not be configured to run anywhere else on the data plane VCN 1118. Each VM 1166(1)-(N) may be connected to one customer tenancy 1170. The corresponding container 1171(1)-(N) contained in the VM 1166(1)-(N) may be configured to run the code. In this case, there can be double isolation (e.g., containers 1171(1)-(N) running code, where containers 1171(1)-(N) can be contained at least in VM 1166(1)-(N), which VM 1166(1)-(N) is contained in (one or more) untrusted application subnet(s) 1162), which can help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging a different customer's network. Containers 1171(1)-(N) can be communicatively coupled to customer lease 1170 and can be configured to send or receive data from customer lease 1170. Containers 1171(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 1118. After running the code is complete, the IaaS provider can terminate or otherwise dispose of containers 1171(1)-(N).

[0147] In some embodiments, the trusted application subnet(s) 1160 may run code owned or operated by the IaaS provider. In this embodiment, the trusted application subnet(s) 1160 may be communicatively coupled to the DB subnet(s) 1130 and configured to perform CRUD operations in the DB subnet(s) 1130. The untrusted application subnet(s) 1162 may be communicatively coupled to the DB subnet(s) 1130, but in this embodiment, the untrusted application subnet(s) may be configured to perform read operations in the DB subnet(s) 1130. The containers 1171(s)-(N) that may be included in each customer's VM 1166(s)-(N) and that may run code from the customer may not be communicatively coupled to the DB subnet(s) 1130.

[0148] In other embodiments, the control plane VCN 1116 and the data plane VCN 1118 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1116 and the data plane VCN 1118. However, communication may occur indirectly through at least one method. The LPG 1110 may be established by an IaaS provider, which may facilitate communication between the control plane VCN 1116 and the data plane VCN 1118. In another example, the control plane VCN 1116 or the data plane VCN 1118 may make a call to the cloud service 1156 via the service gateway 1136. For example, the call from the control plane VCN 1116 to the cloud service 1156 may include a request for a service that may communicate with the data plane VCN 1118.

[0149] Fig.12 12 is a block diagram 1200 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 1202 (e.g., Fig. 9 The service operator 902) may be communicatively coupled to the secure host lease 1204 (e.g., Fig. 9 The secure host lease 1204 may include a virtual cloud network (VCN) 1206 (e.g., Fig. 9 VCN906) and secure host subnet 1208 (for example, Fig. 9 VCN 1206 may include LPG 1210 (e.g., Fig. 9 1210 ), which may be communicatively coupled to an SSH VCN 1212 via an LPG 1210 included in an SSH VCN 1212 (e.g., Fig. 9SSH VCN 1212 may include SSH subnet 1214 (e.g., Fig. 9 SSH subnet 914), and SSH VCN 1212 may be communicatively coupled to control plane VCN 1216 via LPG 1210 included in control plane VCN 1216 (e.g., Fig. 9 916) and is communicatively coupled to a data plane VCN 1218 via an LPG 1210 included in the data plane VCN 1218 (e.g., Fig. 9 The control plane VCN 1216 and the data plane VCN 1218 may be included in a service lease 1219 (e.g., Fig. 9 Service lease 919).

[0150] The control plane VCN 1216 may include (one or more) LB subnets 1222 (e.g., Fig. 9 (one or more) LB subnets 922))) of the control plane DMZ layer 1220 (e.g., Fig. 9 The control plane DMZ layer 920 may include (one or more) application subnets 1226 (e.g., Fig. 9 (one or more) application subnets 926) of the control plane application layer 1224 (e.g., Fig. 9 control plane application layer 924), may include (one or more) DB subnets 1230 (e.g., Fig.11 The control plane data layer 1228 (e.g., Fig. 9 The LB subnet(s) 1222 contained in the control plane DMZ layer 1220 may be communicatively coupled to the application subnet(s) 1226 contained in the control plane application layer 1224 and the internet gateway 1234 (e.g., Fig. 9 934), and the application subnet(s) 1226 may be communicatively coupled to the DB subnet(s) 1230 contained in the control plane data layer 1228 and to the service gateway 1236 (e.g., Fig. 9 ) and a network address translation (NAT) gateway 1238 (e.g., Fig. 9 The control plane VCN 1216 may include a service gateway 1236 and a NAT gateway 1238.

[0151] Data plane VCN 1218 may include a data plane application layer 1246 (e.g., Fig. 9 Data plane application layer 946), data plane DMZ layer 1248 (e.g., Fig. 9 Data plane DMZ layer 948), and data plane data layer 1250 (eg, Fig. 9 The data plane DMZ layer 1248 may include (one or more) LB subnets 1222, which may be communicatively coupled to (one or more) trusted application subnets 1260 (e.g., Fig.11 Trusted application subnet(s) 1160) and untrusted application subnet(s) 1262 (e.g., Fig.11 The trusted application subnet(s) 1260 may be communicatively coupled to the service gateway 1236 contained in the data plane VCN 1218, the NAT gateway 1238 contained in the data plane VCN 1218, and the DB subnet(s) 1230 contained in the data plane data layer 1250. The untrusted application subnet(s) 1262 may be communicatively coupled to the service gateway 1236 contained in the data plane VCN 1218 and the DB subnet(s) 1230 contained in the data plane data layer 1250. The data plane data layer 1250 may include the DB subnet(s) 1230 that may be communicatively coupled to the service gateway 1236 contained in the data plane VCN 1218.

[0152] The untrusted application subnet(s) 1262 may include primary VNICs 1264(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1266(1)-(N) residing within the untrusted application subnet(s) 1262. Each tenant VM 1266(1)-(N) may run code in a respective container 1267(1)-(N) and may be communicatively coupled to an application subnet 1226 that may be contained in a data plane application layer 1246 that may be contained in a container egress VCN 1268. Corresponding secondary VNICs 1272(1)-(N) may facilitate communications between the untrusted application subnet(s) 1262 contained in the data plane VCN 1218 and the application subnets contained in the container egress VCN 1268. The container egress VCN may include a NAT gateway 1238 that may be communicatively coupled to the public Internet 1254 (e.g., Fig. 9 of the public Internet 954).

[0153] The Internet gateway 1234 included in the control plane VCN 1216 and included in the data plane VCN 1218 can be communicatively coupled to the metadata management service 1252 (e.g., Fig. 9 1254), which can be communicatively coupled to the public Internet 1254. The public Internet 1254 can be communicatively coupled to the NAT gateway 1238 included in the control plane VCN 1216 and included in the data plane VCN 1218. The service gateway 1236 included in the control plane VCN 1216 and included in the data plane VCN 1218 can be communicatively coupled to the cloud service 1256.

[0154] In some examples, the Fig.12 The block diagram 1200 of the architectural diagram may be viewed as Fig.11 1100 , and may be desired by customers of the IaaS provider if the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). The corresponding container 1267(1)-(N) contained in each customer's VM 1266(1)-(N) is accessible to the customer in real time. The container 1267(1)-(N) may be configured to make calls to the corresponding secondary VNIC 1272(1)-(N) contained in the application subnet(s) 1226 of the data plane application layer 1246, which may be contained in the container egress VCN 1268. The secondary VNIC 1272(1)-(N) may communicate the call to the NAT gateway 1238, which may communicate the call to the public Internet 1254. In this example, containers 1267(1)-(N), which may be accessed by customers in real time, may be isolated from control plane VCN 1216 and may be isolated from other entities contained in data plane VCN 1218. Containers 1267(1)-(N) may also be isolated from resources from other customers.

[0155] In other examples, a customer may use containers 1267(1)-(N) to call cloud service 1256. In this example, a customer may run code in container 1267(1)-(N) that requests a service from cloud service 1256. Container 1267(1)-(N) may transmit this request to secondary VNIC 1272(1)-(N), which may transmit the request to a NAT gateway, which may transmit the request to public Internet 1254. Public Internet 1254 may transmit the request to LB subnet(s) 1222 contained in control plane VCN 1216 via Internet gateway 1234. In response to determining that the request is valid, LB subnet(s) may transmit the request to application subnet(s) 1226, which may transmit the request to cloud service 1256 via service gateway 1236.

[0156] It should be understood that the IaaS architectures 900, 1000, 1100, 1200 depicted in the figures may have other components in addition to those depicted. In addition, the embodiments shown in the figures are only some examples of cloud infrastructure systems that can be combined with embodiments of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0157] In some embodiments, the IaaS system described herein may include a set of application, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) provided by the applicant.

[0158] Fig.13 An example computer system 1300 is illustrated in which various embodiments may be implemented. System 1300 may be used to implement any of the computer systems described above. As shown, computer system 1300 includes a processing unit 1304 that communicates with a plurality of peripheral subsystems via a bus subsystem 1302. These peripheral subsystems may include a processing acceleration unit 1306, an I / O subsystem 1308, a storage subsystem 1318, and a communication subsystem 1324. Storage subsystem 1318 includes a tangible computer-readable storage medium 1322 and a system memory 1310.

[0159] The bus subsystem 1302 provides a mechanism for allowing the various components and subsystems of the computer system 1300 to communicate with each other as expected. Although the bus subsystem 1302 is schematically shown as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. The bus subsystem 1302 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such an architecture can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus that can be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0160] The processing unit 1304, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1300. One or more processors may be included in the processing unit 1304. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1304 may be implemented as one or more independent processing units 1332 and / or 1334, each of which includes a single-core or multi-core processor. In other embodiments, the processing unit 1304 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0161] In various embodiments, the processing unit 1304 can execute a variety of programs in response to program code and can maintain multiple concurrently executed programs or processes. At any given time, some or all of the program code to be executed may reside in (one or more) processors 1304 and / or storage subsystem 1318. Through appropriate programming, (one or more) processors 1304 can provide the various functions described above. In addition, the computer system 1300 may include a processing acceleration unit 1306, which may include a digital signal processor (DSP), a special processor and / or the like.

[0162] I / O subsystem 1308 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touch pads or touch screens incorporated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors that enable users to control input devices (such as Microsoft Xbox® 360 game controllers) and interact with them through natural user interfaces using gestures and verbal commands. User interface input devices may also include eye gesture recognition devices, such as Google Glass® blink detectors, which detect eye activity from users (e.g., "blinking" when taking pictures and / or making menu selections), and convert eye gestures into input to input devices (e.g., Google Glass®). In addition, user interface input devices may include voice recognition sensing devices that enable users to interact with voice recognition systems (e.g., Siri® navigators) through voice commands.

[0163] The user interface input device may also include, but is not limited to, a three-dimensional (3D) mouse, a joystick or pointing stick, a game controller, and a graphics tablet, as well as audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, and eye tracking devices. In addition, the user interface input device may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound equipment. The user interface input device may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0164] The user interface output device may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1300 to a user or other computers. For example, the user interface output device may include, but is not limited to, a variety of display devices that deliver text, graphics, and audio / video information in a visual manner, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0165] Computer system 1300 may include a storage subsystem 1318 that may provide a tangible, non-transitory computer-readable storage medium for storing software and data structures that provide the functionality of the embodiments described in the present disclosure. Software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of processing unit 1304, provide the functionality described above. Storage subsystem 1318 may also provide a repository for storing data used in accordance with the present disclosure.

[0166] like Fig.13 As shown in the example of , storage subsystem 1318 may include various components, including system memory 1310, computer-readable storage media 1322, and computer-readable storage media reader 1313. System memory 1310 may store program instructions that may be loaded and executed by processing unit 1304. System memory 1310 may also store data used during instruction execution and / or data generated during program instruction execution. Various different types of programs may be loaded into system memory 1310, including but not limited to client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0167] The system memory 1310 may also store an operating system 1316. Examples of the operating system 1316 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, a variety of commercial UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems. In some implementations where the computer system 1300 executes one or more virtual machines, the virtual machines along with their guest operating systems (GOS) may be loaded into the system memory 1310 and executed by one or more processors or cores of the processing unit 1304.

[0168] The system memory 1310 can have different configurations depending on the type of computer system 1300. For example, the system memory 1310 can be volatile memory, such as random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM), flash memory, etc. Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, the system memory 1310 can include a basic input / output system (BIOS), which contains the basic routines that help to transfer information between elements within the computer system 1300, such as during startup.

[0169] Computer-readable storage media 1322 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information used by computer system 1300 , including instructions executable by processing unit 1304 of computer system 1300 .

[0170] Computer readable storage media 1322 may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information. This may include tangible computer readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices or other tangible computer readable media.

[0171] For example, the computer-readable storage medium 1322 may include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a disk drive that reads from or writes to a removable non-volatile disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk (such as a CD ROM, a DVD, and a Blu-Ray® optical disk) or other optical media. The computer-readable storage medium 1322 may include, but is not limited to, a Zip® drive, a flash memory card, a universal serial bus (USB) flash memory drive, a secure digital (SD) card, a DVD disk, a digital video tape, and the like. The computer-readable storage medium 1322 may also include a solid-state drive (SSD) based on non-volatile memory (such as a flash-based SSD, an enterprise flash drive, a solid-state ROM, and the like), an SSD based on volatile memory (such as a solid-state RAM, dynamic RAM, static RAM, DRAM-based SSD), a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of DRAM and a flash-based SSD. The disk drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computer system 1300 .

[0172] Machine-readable instructions executable by one or more processors or cores of processing unit 1304 may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include a physically tangible memory or storage device including a volatile memory storage device and / or a non-volatile storage device. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., magnetic disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM or flash memory, a hard disk drive, a floppy disk drive, a removable memory drive (e.g., a USB drive), or other types of storage devices.

[0173] The communication subsystem 1324 provides an interface to other computer systems and networks. The communication subsystem 1324 is used as an interface for receiving data from other systems from the computer system 1300 and sending data to other systems. For example, the communication subsystem 1324 can enable the computer system 1300 to be connected to one or more devices via the Internet. In some embodiments, the communication subsystem 1324 may include a radio frequency (RF) transceiver component, a global positioning system (GPS) receiver component, and / or other components for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technology, such as 3G, 4G or EDGE (Enhanced Data Rate Global Evolution), WiFi (IEEE 802.11 series standards, or other mobile communication technologies, or any combination thereof). In some embodiments, in addition to or in lieu of a wireless interface, the communication subsystem 1324 may provide wired network connectivity (e.g., Ethernet).

[0174] In some embodiments, the communication subsystem 1324 may also receive incoming communications in the form of structured and / or unstructured data feeds 1326 , event streams 1328 , event updates 1330 , and the like on behalf of one or more users who may use the computer system 1300 .

[0175] For example, the communication subsystem 1324 can be configured to receive data feeds 1326 in real time from users of a social network and / or other communication service, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0176] In addition, the communication subsystem 1324 can also be configured to receive data in the form of a continuous data stream, which can include an event stream 1328 of real-time events and / or event updates 1330, which can be continuous or unbounded in nature without a clear end. Examples of applications that generate continuous data can include, for example, sensor data applications, financial stocks, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0177] The communication subsystem 1324 may also be configured to output structured and / or unstructured data feeds 1326 , event streams 1328 , event updates 1330 , etc. to one or more databases that may be in computer communication with one or more streaming data source computers coupled to the computer system 1300 .

[0178] Computer system 1300 can be one of various types, including a handheld portable device (e.g., an iPhone® cell phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0179] Due to the ever-changing nature of computers and networks, the description of the computer system 1300 depicted in the figure is intended only as a specific example. Many other configurations with more or fewer components than the system depicted in the figure are possible. For example, custom hardware can also be used and / or specific elements can be implemented in hardware, firmware, software (including applets) or in combination. In addition, connections with other computing devices (such as network input / output devices) can be adopted. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will understand other ways and / or methods of implementing various embodiments.

[0180] The embodiments may be implemented by using a computer program product, which comprises a computer program / instructions which, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

[0181] Although specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are also included within the scope of the present disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can freely operate within multiple data processing environments. In addition, although the embodiments have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above-described embodiments may be used alone or in combination.

[0182] In addition, although the embodiments have been described using a specific combination of hardware and software, it should be appreciated that other combinations of hardware and software are also within the scope of the present disclosure. The embodiments may be implemented with hardware only, or with software only, or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors. Therefore, where a component or module is described as being configured to perform certain operations, such a configuration may be implemented, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or any combination thereof. The process may communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0183] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the claims. Therefore, although specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0184] The use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (especially in the context of the appended claims) should be interpreted to cover both the singular and the plural, unless otherwise stated herein or clearly contradicted by the context. Unless otherwise stated, the terms "comprising", "having", "including" and "containing" should be interpreted as open-ended terms (i.e., meaning "including, but not limited to"). The term "connected to" should be interpreted as partially or fully contained, attached to, or combined together, even if something intervenes. Unless otherwise stated herein, the recitation of numerical ranges herein is intended only to serve as a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated into the specification as if it were individually recited herein. Unless otherwise stated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. Unless otherwise claimed, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments and does not limit the scope of the present disclosure. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0185] Unless expressly stated otherwise, disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood within the context as generally used to indicate that an item, term, etc., can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present.

[0186] Preferred embodiments of the present disclosure are described herein, including known best modes for carrying out the present disclosure. After reading the foregoing description, the variations of those preferred embodiments may become clear to those of ordinary skill in the art. Those of ordinary skill should be able to appropriately adopt such variations, and the present disclosure may be put into practice except as specifically described herein. Therefore, the present disclosure includes all modifications and equivalents of the subject matter listed in the appended claims as permitted by applicable law. In addition, unless otherwise specified herein, any combination of the above-mentioned elements in all possible variations thereof is included in the present disclosure.

[0187] Example embodiments of the present disclosure may be described in view of the following clauses.

[0188] Clause 1. A method is disclosed. The method may include identifying, by a power management service, a plurality of components of a power system arranged according to a power distribution hierarchy, the power distribution hierarchy may include a plurality of nodes organized according to respective levels of a plurality of levels. In some embodiments, a node in the plurality of nodes of the power distribution hierarchy represents a corresponding component in the plurality of components. A subset of nodes of a first level in the plurality of levels may descend from a specific node of a second level in the plurality of levels that is higher than the first level. A group of lower-level components in the plurality of components may be represented by a subset of nodes of the first level that receive power allocated by a higher-level component in the plurality of components represented by a specific node of the second level. The method may include monitoring, by the power management service, power consumption of the group of lower-level components represented by the subset of nodes of the first level. The method may include determining, by the power management service, at least in part based on the monitoring, that power consumption of the group of lower-level components has exceeded a budget threshold associated with the higher-level components. The method may also include, in response to determining that power consumption of the group of lower-level components has exceeded a budget threshold associated with the higher-level components, transmitting, by the power management service, a power cap value of the lower-level components. In some embodiments, transmitting the power cap value may cause the lower level component to store the power cap value in memory while allowing the corresponding power consumption of the lower level component to exceed the power cap value until a time period corresponding to the timing value expires.

[0189] Clause 2. The method of clause 1, wherein storing a power cap value in a memory while allowing corresponding power consumption of lower-level components to exceed the power cap value allows the group of lower-level components to continue to exceed the budget threshold associated with the higher-level components for at least a portion of the time period corresponding to the timing value.

[0190] Clause 3. The method of clause 1 or 2, wherein the method may further include transmitting, via a power management service, a timing value to an intermediate component represented by a specific node of an intermediate level of a power distribution hierarchy between a first level and a second level. In some embodiments, a set of lower-level components are configured to receive power allocated from a higher-level component via the intermediate component. In some embodiments, transmitting the timing value causes the intermediate component to generate a timer associated with a time period corresponding to the timing value. In some embodiments, the intermediate component is configured to instruct the lower-level component to trigger an operation to throttle its operation based at least in part on the expiration of the timer based at least in part on the power capping value.

[0191] Clause 4. The method of any of clauses 1-3, wherein determining that power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component may also include determining that power consumption of the group of lower-level components is likely to exceed a budgeted amount of power allocated to the higher-level component.

[0192] Clause 5. The method of clause 4, further comprising obtaining a confidence value based at least in part on historical power consumption data associated with the set of lower-level components that receive power allocated by the higher-level component, the confidence value indicating that power consumption corresponding to the set of lower-level components is likely to exceed a budgeted amount of power allocated to the higher-level component. The method may include determining that the confidence value exceeds a threshold.

[0193] Clause 6. Some embodiments include a system. The system may include a memory configured to store instructions and one or more processors configured to execute instructions of a method. The method may include identifying, by a power management service, a plurality of components of a power system arranged according to a power distribution hierarchy, the power distribution hierarchy including a plurality of nodes organized according to respective levels of a plurality of levels. In some embodiments, a node of a plurality of nodes of the power distribution hierarchy represents a corresponding component of a plurality of components. In some embodiments, a subset of nodes of a first level of a plurality of levels descends from a particular node of a second level of a plurality of levels that is higher than the first level. In some embodiments, a group of lower-level components of a plurality of components represented by a subset of nodes of the first level receive power allocated by a higher-level component of a plurality of components represented by a particular node of the second level. The method may include monitoring, by the power management service, power consumption of the group of lower-level components represented by the subset of nodes of the first level. The method may include determining, by the power management service, at least in part based on the monitoring, that power consumption of the group of lower-level components has breached a budget threshold associated with a higher-level component. The method may include, in response to determining that power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component, transmitting a power cap value for the lower-level component through a power management service, such that the lower-level component stores the power cap value in a memory while allowing corresponding power consumption of the lower-level component to exceed the power cap value until a time period corresponding to a timing value expires.

[0194] Clause 7. The system of clause 6, wherein expiration of the timer corresponding to the time period triggers the lower-level component to limit power consumption based at least in part on at least one of: dynamic frequency scaling or dynamic voltage scaling.

[0195] Clause 8. A system of clause 6 or 7, wherein a second level of the power distribution hierarchy includes two or more nodes from a plurality of nodes, the two or more nodes representing at least a higher-level component corresponding to a first higher-level component and a second higher-level component different from the first higher-level component, wherein executing instructions further causes the system to at least: 1) determine, via a power management service, with respect to the second higher-level component, that there is an unused portion of power supplied to the second higher-level component; 2) determine, via a power management service, a specific power cap value for a second lower-level component in a second group of lower-level components that receive power via the second higher-level component; 3) transmit, via the power management service, a corresponding power cap value for the second lower-level component, wherein transmitting the corresponding power cap value triggers the second lower-level component to throttle operations at the second lower-level component as part of implementing the corresponding power cap value; and 4) transmit, via the power management service, a cancel signal that causes a timer corresponding to a time period to be canceled, wherein canceling the timer causes the lower-level component to refrain from implementing the power cap value after the time period expires.

[0196] Clause 9. A system of any of clauses 6-8, wherein executing instructions further causes the system to at least, in response to determining that power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component, determine a power cap value through a power management service based at least in part on at least one of: a difference between power consumption of the group of lower-level components, a priority value associated with a workload being executed at the group of lower-level components, or an individual consumption rate of the group of lower-level components.

[0197] Clause 10. The system of any of clauses 6-9, wherein the power capping value is determined based on a monitoring data value obtained upon or after determining that power consumption of the set of lower level components has breached a budget threshold associated with a higher level component.

[0198] Clause 11. Some embodiments include a method. The method includes identifying, by a power management service, a plurality of components of a power system arranged according to a power distribution hierarchy, the power distribution hierarchy including a plurality of nodes organized according to respective levels of a plurality of levels. In some embodiments, a node in the plurality of nodes of the power distribution hierarchy represents a corresponding component in the plurality of components. In some embodiments, a subset of nodes of a first level in the plurality of levels descends from a particular node of a second level in the plurality of levels that is higher than the first level. In some embodiments, a group of lower-level components in the plurality of components represented by the subset of nodes of the first level receive power allocated by a higher-level component in the plurality of components represented by a particular node of the second level. The method may include monitoring, by the power management service, power consumption of the group of lower-level components represented by the subset of nodes of the first level. The method may include determining, by the power management service, at least in part based on the monitoring, that power consumption of the group of lower-level components has breached a budget threshold associated with the higher-level components. The method may include starting a timer corresponding to a timing value in response to determining that power consumption of the group of lower-level components has breached a budget threshold associated with the higher-level components. In some embodiments, expiration of the timer indicates expiration of a time period corresponding to the timing value, a lower-level component in the group of lower-level components stores a power cap value in a memory during the time period, and enforcement of the power cap value for the lower-level component is delayed until the timer expires.

[0199] Clause 12. The method of Clause 11 may also include: also in response to determining that the power consumption of the group of lower-level components has exceeded the budget threshold associated with the higher-level components, determining a timing value by a power management service based at least in part on at least one of: the rate of change of power consumption of the group of lower-level components, the direction of change corresponding to the rate of change of power consumption of the group of lower-level components, the tolerance of the power circuit associated with the higher-level components, or the expected time required to initiate power capping at each component in the group of lower-level components.

[0200] Clause 13. The method of clause 11 or 12, wherein starting a timer corresponding to the timing value further comprises transmitting the timing value to an intermediate component from which the lower-level component receives power through a power management service. In some embodiments, transmitting the timing value causes the intermediate component to generate a timer associated with a time period corresponding to the timing value.

[0201] Clause 14. The method of clause 13, wherein the intermediate component triggers enforcement of the power capping value of the lower level component upon expiration of the timer.

[0202] Clause 15. The method of any of clauses 11-14 further includes: 1) obtaining a probability value indicating that the combined power consumption corresponding to the second group of lower-level components receiving power allocated by the higher-level components will likely exceed a budget threshold associated with the higher-level components based at least in part on historical power consumption data associated with the second group of lower-level components receiving power allocated by the higher-level components, and 2) in response to determining that the probability value exceeds the probability threshold, transmitting an additional power cap value to a second lower-level component in the second group of lower-level components via a power management service, wherein the transmitting of the additional power cap value causes the additional power cap value to be used to constrain the corresponding power consumption at the second lower-level component in the second group of lower-level components.

[0203] Clause 16. In some embodiments, a system is disclosed. The system may include a memory configured to store instructions and one or more processors configured to execute instructions for performing a method. The method may include identifying, by a power management service, a plurality of components of a power system arranged according to a power distribution hierarchy, the power distribution hierarchy including a plurality of nodes organized according to respective levels of a plurality of levels, the nodes of the plurality of nodes of the power distribution hierarchy representing corresponding components of a plurality of components. In some embodiments, a subset of nodes of a first level of the plurality of levels descends from a particular node of a second level of the plurality of levels that is higher than the first level. In some embodiments, a group of lower-level components of the plurality of components is represented by a subset of nodes of the first level, the subset of nodes of the first level receiving power allocated by a higher-level component of the plurality of components represented by a particular node of the second level. The method may include monitoring, by the power management service, power consumption of the group of lower-level components represented by the subset of nodes of the first level. The method may also include determining, by the power management service, at least in part based on the monitoring, that power consumption of the group of lower-level components has breached a budget threshold associated with the higher-level components. The method may include starting a timer corresponding to a timing value in response to determining that power consumption of the group of lower-level components has breached a budget threshold associated with the higher-level components. In some embodiments, expiration of the timer indicates expiration of a time period corresponding to the timing value, a lower-level component in the group of lower-level components stores a power cap value in a memory during the time period, and enforcement of the power cap value for the lower-level component is delayed until the timer expires.

[0204] Clause 17. A system of clause 16, wherein executing instructions further causes the system to at least, in response to determining that the power consumption of the group of lower-level components has exceeded a budget threshold associated with a higher-level component, determine a timing value through a power management service based at least in part on at least one of: the rate of change of power consumption of the group of lower-level components, the direction of change corresponding to the rate of change of power consumption of the group of lower-level components, a tolerance of a power circuit associated with the higher-level component, or an expected time required to initiate power capping at each component in the group of lower-level components.

[0205] Clause 18. The system of clause 16 or 17, wherein the group of lower-level components is deemed to have exceeded a budget threshold based at least in part on a determination, through a power management service and at least in part on historical consumption data associated with the group of lower-level components, that power consumption of the group of lower-level components is likely to exceed a budget threshold.

[0206] Clause 19. A system of any of clauses 16-18, wherein a second level of a power distribution hierarchy comprises two or more nodes of a plurality of nodes, wherein the two or more nodes represent at least a higher-level component corresponding to a first higher-level component and a second higher-level component different from the first higher-level component, wherein execution of instructions further causes the system to at least: 1) determine, via a power management service, with respect to the second higher-level component, that there is an unutilized portion of power supplied to the second higher-level component; 2) determine, via a power management service, a specific power cap value for a second lower-level component in a second group of lower-level components that receive power via the second higher-level component; 3) transmit, via the power management service, the specific power cap value for the second lower-level component, wherein transmitting the specific power cap value for the second lower-level component triggers power consumption constraints to be performed at the second lower-level component as part of implementing the specific power cap value; and 4) transmit, via the power management service, a cancel signal causing the timer to be canceled.

[0207] Clause 20. The system of any of clauses 16-19, wherein the power capping value is determined based on a monitoring data value obtained upon or after determining that power consumption of the set of lower level components has breached a budget threshold associated with a higher level component.

[0208] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference in its entirety and were set forth herein.

[0209] In the foregoing description, aspects of the present disclosure are described with reference to specific embodiments of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects disclosed above may be used alone or in combination. In addition, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Therefore, the description and drawings are considered to be illustrative rather than restrictive.

Claims

1. A method comprising: identifying, by a computer, a plurality of components of an electric power system arranged according to a power distribution hierarchy, the power distribution hierarchy comprising a plurality of nodes organized according to respective levels of a plurality of levels, nodes of the plurality of nodes of the power distribution hierarchy representing corresponding components of the plurality of components, a subset of nodes of a first level of the plurality of levels descending from a particular node of a second level of the plurality of levels higher than the first level, and a set of lower level components of the plurality of components represented by the subset of nodes of the first level receiving power distributed through higher level components of the plurality of components represented by the particular node of the second level; monitoring, by the computer, power consumption of the set of lower-level components represented by the subset of nodes of the first level; determining, by the computer, based at least in part on the monitoring, that the power consumption of the set of lower-level components has breached a budget threshold associated with the higher-level component; as well as In response to determining that the power consumption of the group of lower-level components has exceeded the budget threshold associated with the higher-level component, the power cap value of the lower-level component is transmitted through the computer, so that the lower-level component stores the power cap value in the memory while allowing the corresponding power consumption of the lower-level component to exceed the power cap value until a time period corresponding to the timing value expires.

2. A method according to claim 1, wherein storing the power cap value in a memory while allowing the corresponding power consumption of the lower-level components to exceed the power cap value allows the group of lower-level components to continue to exceed the budget threshold associated with the higher-level component during at least a portion of the time period corresponding to the timing value.

3. The method according to claim 1 or 2, further comprising: transmitting, by the computer, the timing value to an intermediate component represented by a particular node at an intermediate level between a first level and a second level of the power distribution hierarchy, wherein the set of lower level components are configured to receive power distributed from the higher level components through the intermediate component; wherein transmitting the timing value causes the intermediate component to generate a timer associated with the time period corresponding to the timing value; and Wherein the intermediate component is configured to, based at least in part on expiration of the timer, instruct the lower-level component to trigger an operation for throttling its operation based at least in part on the power capping value.

4. The method of any one of claims 1-3, wherein determining that the power consumption of the set of lower-level components has exceeded the budget threshold associated with the higher-level component also includes determining that the power consumption of the set of lower-level components is likely to exceed the budgeted power allocated to the higher-level component.

5. The method according to claim 4, further comprising obtaining a confidence value based at least in part on historical power consumption data associated with the set of lower-level components that receive power allocated by the higher-level component, the confidence value indicating that the power consumption corresponding to the set of lower-level components is likely to exceed the budgeted power allocated to the higher-level component; and It is determined that the confidence value exceeds a threshold.

6. A system comprising: a memory configured to store instructions; as well as One or more processors configured to execute the instructions to at least: identifying, by the one or more processors, a plurality of components of an electric power system arranged according to a power distribution hierarchy, the power distribution hierarchy comprising a plurality of nodes organized according to respective levels of a plurality of levels, nodes of the plurality of nodes of the power distribution hierarchy representing corresponding components of the plurality of components, a subset of nodes of a first level of the plurality of levels descending from a particular node of a second level of the plurality of levels higher than the first level, and a set of lower level components of the plurality of components represented by the subset of nodes of the first level receiving power distributed through higher level components of the plurality of components represented by the particular node of the second level; monitoring, by the one or more processors, power consumption of the set of lower-level components represented by the subset of nodes of the first level; determining, by the one or more processors based at least in part on the monitoring, that the power consumption of the set of lower-level components has breached a budget threshold associated with the higher-level component; as well as In response to determining that the power consumption of the set of lower-level components has exceeded the budget threshold associated with the higher-level component, the power cap value of the lower-level component is transmitted through the one or more processors, so that the lower-level component stores the power cap value in the memory while allowing the corresponding power consumption of the lower-level component to exceed the power cap value until a time period corresponding to the timing value expires.

7. The system of claim 6, wherein expiration of a timer corresponding to the time period triggers the lower-level component to limit power consumption based at least in part on at least one of: dynamic frequency scaling or dynamic voltage scaling.

8. The system of claim 6 or 7, wherein the second level of the power distribution hierarchy includes two or more nodes of the plurality of nodes, the two or more nodes representing at least the higher-level component corresponding to the first higher-level component and a second higher-level component different from the first higher-level component, wherein execution of the instructions further causes the system to at least: determining, by the one or more processors with respect to a second higher-level component, that there is an unutilized portion of power supplied to the second higher-level component; determining, by the one or more processors, a specific power capping value for a second lower level component in a second set of lower level components receiving power via a second higher level component; transmitting, by the one or more processors, the respective power cap value for a second lower level component, wherein transmitting the respective power cap value triggers the second lower level component to throttle operations at the second lower level component as part of enforcing the respective power cap value; and A cancel signal is transmitted by the one or more processors, the cancel signal causing a timer corresponding to the time period to be canceled, wherein canceling the timer causes the lower-level component to refrain from enforcing the power capping value after expiration of the time period.

9. A system according to any one of claims 6-8, wherein executing the instructions also causes the system to at least, in response to determining that the power consumption of the set of lower-level components has exceeded the budget threshold associated with the higher-level component, determine the power cap value by the one or more processors based at least in part on at least one of: the difference between the power consumption of the set of lower-level components, the priority value associated with the workload being executed at the set of lower-level components, or the individual consumption rate of the set of lower-level components.

10. The system of any one of claims 6-9, wherein the power capping value is determined based on a monitoring data value obtained upon or after determining that the power consumption of the set of lower-level components has breached the budget threshold associated with the higher-level component.

11. A method comprising: identifying, by a computer, a plurality of components of an electric power system arranged according to a power distribution hierarchy, the power distribution hierarchy comprising a plurality of nodes organized according to respective levels of a plurality of levels, nodes of the plurality of nodes of the power distribution hierarchy representing corresponding components of the plurality of components, a subset of nodes of a first level of the plurality of levels descending from a particular node of a second level of the plurality of levels higher than the first level, and a set of lower level components of the plurality of components represented by the subset of nodes of the first level receiving power distributed through higher level components of the plurality of components represented by the particular node of the second level; monitoring, by the computer, power consumption of the set of lower-level components represented by the subset of nodes of the first level; determining, by the computer, based at least in part on the monitoring, that the power consumption of the set of lower-level components has breached a budget threshold associated with the higher-level component; as well as in response to determining that the power consumption of the set of lower-level components has breached the budget threshold associated with the higher-level component, starting a timer corresponding to a timing value; wherein the expiration of the timer indicates the expiration of a time period corresponding to the timing value; wherein a lower level component in the set of lower level components stores a power capping value in a memory during the time period; and Wherein enforcement of the power capping value on the lower level component is delayed until the expiration of the timer.

12. The method according to claim 11, further comprising: Also in response to determining that the power consumption of the set of lower-level components has exceeded the budget threshold associated with the higher-level component, the timing value is determined by the computer at least in part based on at least one of: the rate of change of the power consumption of the set of lower-level components, the direction of change corresponding to the rate of change of the power consumption of the set of lower-level components, the tolerance of the power circuit associated with the higher-level component, or the expected time required to initiate the power capping at each component in the set of lower-level components.

13. The method according to claim 11 or 12, wherein starting the timer corresponding to the timing value further comprises: The timing value is communicated by the computer to an intermediate component from which the lower-level component receives power, wherein communicating the timing value causes the intermediate component to generate the timer associated with the time period corresponding to the timing value.

14. The method of claim 13, wherein the intermediate component triggers enforcement of the power capping value on the lower level component upon expiration of the timer.

15. The method according to any one of claims 11 to 14, further comprising: obtaining a likelihood value indicating that a combined power consumption corresponding to a second set of lower-level components that receive power allocated by the higher-level component will more likely than not exceed the budget threshold associated with the higher-level component based at least in part on historical power consumption data associated with a second set of lower-level components that receive power allocated by the higher-level component, and In response to determining that the likelihood value exceeds a likelihood threshold, an additional power cap value is transmitted, by the computer, to a second lower-level component in a second group of lower-level components, wherein transmitting the additional power cap value causes the additional power cap value to be used to constrain corresponding power consumption at the second lower-level component in the second group of lower-level components.

16. A system comprising: a memory configured to store instructions; as well as One or more processors configured to execute the instructions to at least: identifying, by the one or more processors, a plurality of components of an electric power system arranged according to a power distribution hierarchy, the power distribution hierarchy comprising a plurality of nodes organized according to respective levels of a plurality of levels, nodes of the plurality of nodes of the power distribution hierarchy representing corresponding components of the plurality of components, a subset of nodes of a first level of the plurality of levels descending from a particular node of a second level of the plurality of levels higher than the first level, and a set of lower level components of the plurality of components represented by the subset of nodes of the first level receiving power distributed through higher level components of the plurality of components represented by the particular node of the second level; monitoring, by the one or more processors, power consumption of the set of lower-level components represented by the subset of nodes of the first level; determining, by the one or more processors based at least in part on the monitoring, that the power consumption of the set of lower-level components has breached a budget threshold associated with the higher-level component; In response to determining that the power consumption of the set of lower-level components has breached the budget threshold associated with the higher-level component, starting a timer corresponding to a timing value; and wherein the expiration of the timer indicates the expiration of a time period corresponding to the timing value; wherein a lower level component in the set of lower level components stores a power capping value in a memory during the time period; and Wherein enforcement of the power capping value on the lower level component is delayed until the expiration of the timer.

17. A system according to claim 16, wherein executing the instructions also causes the system to at least, in response to determining that the power consumption of the set of lower-level components has exceeded the budget threshold associated with the higher-level component, determine the timing value by the one or more processors based at least in part on at least one of: the rate of change of the power consumption of the set of lower-level components, the direction of change corresponding to the rate of change of the power consumption of the set of lower-level components, the tolerance of the power circuit associated with the higher-level component, or the expected time required to initiate the power capping at each component in the set of lower-level components.

18. A system according to claim 16 or 17, wherein the set of lower-level components is deemed to have exceeded the budget threshold based at least in part on determining, by the one or more processors and at least in part on historical consumption data associated with the set of lower-level components, that the power consumption of the set of lower-level components is likely to exceed the budget threshold.

19. A system according to any one of claims 16 to 18, wherein wherein the second level of the power distribution hierarchy includes two or more nodes of the plurality of nodes, the two or more nodes representing at least the higher-level component corresponding to the first higher-level component and a second higher-level component different from the first higher-level component, wherein execution of the instructions further causes the system to at least: determining, by the one or more processors with respect to a second higher-level component, that there is an unutilized portion of power supplied to the second higher-level component; determining, by the one or more processors, a specific power capping value for a second lower level component in a second set of lower level components receiving power via a second higher level component; transmitting, by the one or more processors, the particular power cap value for a second lower level component, wherein transmitting the particular power cap value for the second lower level component triggers performing power consumption constraints at the second lower level component as part of enforcing the particular power cap value; and A cancel signal is transmitted through the one or more processors causing the timer to be canceled.

20. The system of any one of claims 16-19, wherein the power capping value is determined based on a monitoring data value obtained upon or after determining that the power consumption of the set of lower level components has breached the budget threshold associated with the higher level component.

21. A non-transitory computer readable medium having instructions stored thereon, which when executed by a processor, cause the processor to perform the method according to any one of claims 1-5 and 11-15.

Images