AUTOMATIC POWER CONSUMPTION MANAGEMENT BY APPLYING A SYSTEM POWER CAP TO HETEROGENEOUS SYSTEMS

Abstract

A method for applying a system power limit to a heterogeneous system, the method comprising: Obtaining, by a processor (802, 904), power limit ranges for a plurality of controllable system equipment and power limit values ​​for a plurality of non-controllable system equipment, wherein the plurality of controllable system equipment comprises a plurality of different equipment types, each having a different power limit range, and wherein the plurality of controllable system equipment and the plurality of non-controllable system equipment are grouped into a plurality of pools; Calculating a system power limit range for the heterogeneous system from the power limit ranges and power limit values; Receiving the requested power limit specified for the heterogeneous system via an interface receiver (112);Determining, by a processor (802, 904), power limits for the plurality of controllable system equipment of the plurality of pools from a comparison of a requested power limit for the heterogeneous system with the calculated system power limit range, wherein determining the power limits for the plurality of controllable system equipment of each pool of the plurality of pools comprises: setting the power limits for the plurality of controllable system equipment for the respective pool to minimum power limits of the different power limit ranges; and increasing the power limits from the minimum power limits based on a remainder of the requested power limit after setting the power limits for a pool prior to each pool;and provide, by means of a processor (802, 904), which applies certain power limitations to the heterogeneous system, the certain power limitations being applied to each of the plurality of controllable system equipment.;

Description

background

[0001] High-performance computing (HPC) refers to the systems used to solve large and complex computational problems. HPC typically requires specialized high-performance hardware powering massively parallel central processing units (CPUs). For many years, supercomputers were the predominant hardware for performing large-scale computations. However, recent technological advances have created alternative ways to perform HPC that are far more cost-effective than traditional supercomputers.

[0002] One of the new approaches in HPC is the use of clusters. Clusters are independent system devices networked together to form a parallel processor system. Each system device runs independently and solves a part of a distributed computation. Thanks to the availability of inexpensive but powerful personal computers combined with fast networking technologies, clustering is as effective as supercomputers in solving large computational problems, but at a much lower cost. Although clustering system devices is beneficial for deploying HPC, managing clustered systems is not trivial. Managing hundreds of independently running systems presents numerous challenges, including physical aspects (heat dissipation, access for maintenance, etc.) and system administration tasks (machine setup, status checking, etc.).Approaches to address these and similar problems may therefore be desirable. Brief description of the drawings

[0003] The present disclosure will be described in detail in accordance with one or more various embodiments with reference to the following figures. The figures are for illustrative purposes only and represent typical or exemplary implementations. Fig. 1 shows an example of a system architecture and corresponding high-level message flow in which the present disclosure may be implemented. Fig. 2 is a flowchart of an example process for distributing a system performance cap to compute nodes of an HPC system according to the implementations disclosed herein. Fig.3 shows a schematic diagram of a decision graph for determining a system-wide power cap for the controllable compute nodes according to an example implementation. Fig. Figure 4 is a bar chart showing the overall power utilization of different distribution schemes for allocating power caps to controllable compute nodes according to an example implementation. Fig. 5 is a flowchart of an example process for distributing a system performance cap across pools of compute nodes of an HPC system in accordance with the implementations disclosed herein. Fig. Figure 6 is a sequence diagram of an example message flow illustrating the operations for creating a pool according to the implementations disclosed herein. Fig.7 is a sequence diagram of an example message flow illustrating the operations for distributing system performance caps according to the implementations disclosed herein. Fig. 8 is an example of a computer component that may be used to implement various features of system power cap distribution in accordance with the implementations disclosed herein. Fig. 9 is an example of a computer system that may be used to implement various system power cap distribution features of the present disclosure.

[0004] The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed. Detailed description

[0005] As described above, managing numerous independently running system devices to perform a computation (also referred to as a workload or job) presents a significant challenge. For example, managing the power consumption of numerous independently running system devices is becoming increasingly complex as the HPC landscape rapidly changes due to rising energy prices, increasing regulatory requirements for data center sustainability (e.g., reducing carbon emissions, overall grid power load, etc.), and the increasing power consumption of HPC systems as they grow larger. System operators and administrators are seeking solutions that enable efficient management of this changing landscape.The implementations disclosed here address the above-mentioned problems by providing systems and methods for distributing a system-wide power cap across a set of system devices that can be implemented for HPC.

[0006] The basic setting of a power cap is a relatively simple process, especially in the case of a homogeneous HPC system, since all system devices are equal. Thus, a power cap can be distributed across the system by allocating the same proportion (e.g., the same ratio) of the total system power budget. A power budget can refer to a target power consumption that the entire system must not exceed, whereas a power cap limits the allowable power consumption of system devices. To determine the power cap, the power consumption of system devices that do not consume controllable power (e.g., static power, overhead power, etc.) can be subtracted from the power budget, and the remainder can be the maximum power cap that can be allocated to controllable system devices.

[0007] However, given the increasing variety of accelerator types and the need to support a wide range of workloads as efficiently as possible, heterogeneous systems are becoming increasingly common in HPC system architectures. System architectures are also increasingly moving toward heterogeneous compute units. This has the following consequences: Different device architectures with different types of compute devices have different minimum and maximum power limits; different power cap values ​​have different impacts on the performance of the device architecture; and a system-wide power cap may not be evenly distributed across the number of devices to efficiently utilize the power budget.Different types of computing devices can be differentiated by different stock keeping units (SKUs) or other unique designations to distinguish between individual product models.

[0008] For example, a system-wide power budget can be expressed as Equation 1 as the sum of non-controllable, power-consuming system devices (referred to here as non-controllable system devices) and controllable, power-consuming system devices (referred to here as controllable system devices). The terms "controllability" and "non-controllability" of system devices are used here to refer to the ability to control or change a power cap setting in a respective device. System Power Budget=∑j=1Snoncontrollable consumersNameplate Power+∑j=1N(NodePowerBase Power+∑k=1CCPUk+∑l=1AAcceleratorl)

[0009] where ∑j=1Snoncontrollable consumersNameplate Power represents the sum of the power consumption of non-controllable system nodes, such as the supporting infrastructure (e.g., power distribution, system cooling, etc.) and system components and / or system nodes that either cannot be controlled, should not be controlled, or are classified as non-controllable, such as login nodes, network equipment, system management controllers, input / output subsystems, etc., for which the term "plate power" or any alternative value for protection power may be used; ∑j=1N() stands for system components and / or compute nodes whose power consumption can be controlled (e.g., for those compute nodes of an HPC system that can be controlled); NodePower Base Power represents a sum of all non-controllable consumers (or node components) of a respective system equipment; ∑k=1CCPUk+∑l=1AAcceleratorl stands for node components whose power consumption can be controlled (e.g. individual computing units on the nodes).

[0010] As shown in Equation 1, the system-wide power consumption can be expressed as the sum of two terms. One term (e.g., ∑j=1Snoncontrollable consumersNameplate Power ) defines the maximum possible power consumption of system devices that are not controllable, and the other (e.g., ∑j=1N(Node PowerBase Power+∑k=1CCPUk+∑l=1AAcceleratorl) ) defines the sum of the power consumption of all controllable system devices. In this example, the sum of the power consumed by a controllable system device is based on the sum of a fixed base power consumption – which may also include memory power depending on the design of the compute unit – and the power consumption of the various compute units or components on the node. Therefore, a power cap for the system can be determined by ∑j=1Snoncontrollable consumersNameplate Power deducted from a system performance budget.

[0011] Establishing meaningful performance limits for a heterogeneous system, i.e., a system consisting of a number of heterogeneous devices, is challenging. Table 1 provides examples of hardware performance limits (e.g., hardware-defined performance limits) for two types of heterogeneous hardware: Table 1: Example, heterogeneous hardware power limit ranges Model Type Min. power limit in watts Max Power Cap in Watts Max - Min power consumption (Delta) in watts # Junctions Equipment Type 1 - Homogeneous Node (2 CPUs) 350 925 575 1536 Device type 2 - Heterogeneous node (1 CPU, 4 GPUs) 764 2754 1990 2560

[0012] A traditional approach would take a system-wide power cap (e.g., the combined allowable power consumption of all system devices), divide the system-wide power cap by the number of system devices, and set the resulting value as a uniform power cap for all system devices throughout the system. As shown in Table 1, depending on the system-wide power cap, there may be little or no overlap between the hardware-defined power ranges for the different hardware architectures implemented as the different node types. For example, Table 1 specifies a maximum power cap of 925 watts for Device Type 1 and a minimum power cap of 764 watts for Device Type 2, and setting a uniform power cap for all nodes can establish a cap within this range.Given the small overlap between the hardware-defined power capping ranges, the uniform power capping would not effectively utilize the power capping delta of device type 2, as much of the power capping range does not overlap with the device type 1 range. Therefore, it may be difficult to find a "universally" applicable power cap that can be applied uniformly to all device types in a heterogeneous system. Furthermore, the likelihood that a uniform device power cap calculated from a system-wide power cap will fall within the allowable device power limits decreases as the diversity of device types in a heterogeneous HPC system increases.

[0013] The technology disclosed here enables a system administrator or operator to set a system-wide power cap that can be distributed among system devices for efficient use of power cap ranges, without requiring knowledge of the intricacies of the heterogeneous system architecture. Implementations of the disclosed technology provide a mechanism configured to intelligently set power caps on a per-system device basis according to a specified system-wide power cap and distribution policies.The system-wide power limit can be divided into individual device-based power limits according to device characteristics and end-user-defined tradeoffs to achieve optimal power limit distribution for homogeneous and / or heterogeneous system and device architectures using out-of-band (OOB) system control. Out-of-band (OOB) system control refers to systems and devices that enable access to and management of networked architectures from a remote location separate from the networked architecture.

[0014] The example implementations described herein calculate and distribute power caps based on a requested system-wide power cap, a distribution policy, and the power management capabilities of individual devices. The technology described here determines an optimal distribution of a system-wide power cap for a system, which may have a multi-device architecture. The optimal distribution may be based on an end-user-defined / requested power cap and the application of a distribution scheme that defines an allocation of power caps to system devices to efficiently distribute the requested power cap among the controllable system devices in the system. The system architecture may include a number of controllable system devices and a number of non-controllable system devices.The controllable system equipment may be heterogeneous because the controllable system equipment may comprise a number of different equipment types, each of which has a hardware-defined performance upper limit range that is different from that of other equipment types.

[0015] In an example implementation, the disclosed technology obtains power cap ranges from system devices in a system that may be heterogeneous or homogeneous. For example, the system may include a plurality of controllable compute nodes and a plurality of non-controllable system devices. The disclosed technology may obtain power cap ranges for the majority of the controllable system devices and power cap values ​​for the majority of the non-controllable system devices. The power cap values ​​for the non-controllable system devices may be fixed values ​​defined by the hardware architecture of the respective system devices. Similarly, the power cap ranges may be defined by the hardware architectures of the respective system devices.The plurality of controllable system devices may include a number of different device types, each having different power cap ranges. A system-wide power cap range for the system may be determined from the power cap ranges and the power cap values. A system-wide power cap for the plurality of controllable system devices may be set based on a requested power cap specified for the system, e.g., based on input from an end user (e.g., operator, administrator, or other user).Based on the set system-wide power cap, individual power caps can be determined for each of the multiple controllable system devices based on a comparison of the set system-wide power cap with the system power cap range. The individual power caps for the different device types are based on the different power cap ranges. For example, a distribution scheme can be applied to the multiple controllable system devices that determines an individual power cap allocation for each controllable system device, based in part on the set system-wide power cap. This allocation can be applied to different power caps, respectively, thereby efficiently utilizing each different power cap range.The disclosed technology can then provide the system with the determined power limits, which can be applied to each of the multiple controllable system devices to manage power consumption throughout the system.

[0016] In an example implementation, input may be received from a user specifying a Requested_Power_Cap. A system-wide power cap range may be determined from the power cap ranges of each individual controllable system device in the system. For example, the system-wide power cap may be calculated from the sum of the minimum allowable power cap of all controllable system devices (Sum_Min) and the sum of the maximum allowable power cap of all controllable system devices (Sum_Max). An optimal power cap distribution may be determined based on comparing the requested power cap with the system-wide power cap range and applying a power distribution scheme selected from a variety of power distribution schemes.

[0017] The power consumption of the controllable loads can then be limited by applying an optimal power cap distribution. For example, if the requested power cap is greater than or equal to Sum_Max, the requested power cap can be set to the maximum, and the power caps for all controllable loads can be set to the maximum power cap of the ranges defined in the hardware. If the requested power cap is less than Sum_Min, an error can be returned, the requested power cap can be set to Sum_Min, and the power caps for the controllable loads can be set to the minimum power cap, and / or certain controllable loads can be disabled to achieve the requested power cap.Otherwise, one or more dispatch schemes may be applied, each of which may determine a power cap allocation amount to increase the power caps of the multiple controllable system devices within the respective power cap ranges of the controllable system devices. Each dispatch scheme calculates the power cap for each controllable system device by determining an allocation amount to increase the power caps of all controllable system devices until the system power consumption when operating at the highest allowable power consumption (e.g., sum of the maximum allocated power caps) equals the requested power cap. An optimal dispatch scheme may be selected from one or more dispatch schemes that enable optimal system power utilization, such as the dispatch scheme that provides system power utilization closest to the Requested_Power_Cap.

[0018] The determined power cap ranges can then be fed back to the system for setting as actual power caps on each of the controllable system devices. For example, the implementations disclosed herein can transmit a message wrapped in instructions for applying power caps to each controllable system device. Each controllable system device can unwrap the instructions and set its actual power cap accordingly. As a result, the system can operate such that system-wide power consumption does not exceed the requested power cap due to the power caps set on the system devices.

[0019] In another example implementation, the disclosed technology provides for the distribution of a system-wide power cap according to the clustering of compute nodes. For example, the compute nodes of a system may be clustered into a number of pools, where each pool includes one or more compute nodes and / or system nodes. An optimal distribution of a system-wide power cap may be determined based on a user-specified / requested power cap on a pool-by-pool basis by applying one or more distribution schemes that assign power caps to the controllable compute nodes of each pool. The pools may be prioritized by power consumption and traversed in order of prioritization so that controllable compute nodes of higher-priority pools can be assigned higher power caps.For example, node pools can be prioritized so that performance caps are assigned to higher-priority pools first, before performance caps are assigned to lower-priority pools. This allows higher-priority pools to be assigned higher performance caps compared to lower-priority pools.

[0020] In an example implementation, the disclosed technology receives a Requested_Power_Cap for the system, obtains configurations of pools of controllable compute nodes, and obtains priorities assigned to each pool. The disclosed technology can determine an optimal distribution of the power cap for each individual pool according to priority by comparing the requested power cap to a system-wide power cap range and applying a distribution scheme. This means, for example, that each pool of compute nodes is assigned a priority level, and the requested power cap can be distributed on a pool-by-pool basis in order of priority level (e.g.,starting with the pool with the highest priority in terms of power consumption and proceeding in order to the pool with the lowest priority in terms of power consumption), an optimal power cap distribution for the controllable compute nodes of a respective pool can be determined by applying a distribution scheme as described above.

[0021] Furthermore, the implementations disclosed herein may be automated based on the occurrence of a trigger event. Triggering events may include system events, such as the start of a job or workload; the receipt of a requested power cap or power budget, such as an update to a previously entered requested power cap / power budget; and a period of time, such as detected by a periodic timer. In one example, the disclosed technology may be configured to detect or otherwise sense the occurrence of one or more trigger events and, in response to the detection, compute an optimal power cap distribution for controllable compute nodes, as described above.

[0022] The technology according to the present disclosure offers several non-limiting advantages. For example, the implementations disclosed herein can be provided for OOB system power management of both homogeneous and heterogeneous systems, meeting the OOB power management requirements of system administrators and / or operators. The implementations disclosed herein provide solutions for setting a system-wide power cap that can be distributed among system devices of different types according to device characteristics (e.g., hardware-defined power cap ranges) and system trade-offs (e.g., trade-offs between power consumption and computation completion time), thereby providing optimal power distribution for both homogeneous and heterogeneous systems. Considering system trade-offs enables system optimization between energy efficiency and performance.

[0023] To further illustrate the benefits offered by the disclosed technology, some use case examples are provided below, illustrating non-exhaustive benefits that can be achieved with the disclosed technology. For example, an application running on a system device may use only two GPUs instead of the four provided by the device (e.g., device type 2 in Table 1). The power distribution within a power cap for the equipment can therefore be shifted to provide more power to the used compute units or components (e.g., two GPUs) or even to shift unused power between system equipment and jobs.As another example, a site with a heterogeneous system might need to set a system power cap of 80% of a maximum power capacity to account for lower power availability or lower operating costs. The disclosed technology can accommodate this reduced power cap and redistribute the power share among disparate system devices to efficiently allocate resources while maintaining a reduced power cap. In another example, a site might need to reduce system power consumption below a supported minimum power cap. Instead of shutting down the entire system (e.g., because the power cap is not supported), a policy is created that determines which system devices should be shut down first to provide enough power to operate a subset of system devices at the minimum supported power.In addition, a system administrator or operator can require minimum guarantees for device performance. By setting minimum performance limits for different device types, the presented technology can guarantee minimum device performance even with a reduced system-wide performance limit.

[0024] The term "heterogeneous device" as used here refers to a system device consisting of multiple different computing architectures used within the system device. For example, a system device consisting of one CPU and four GPUs can be considered a heterogeneous device. The term "heterogeneous system" as used here refers to a system consisting of different device architectures.

[0025] As used herein, the term "homogeneous device" refers to a system device consisting of a common or identical computing architecture used within the system device. For example, a system device consisting of two CPUs, each with the same speed and core size, can be considered a homogeneous device. As another example, a system device consisting of two CPUs of the same model (e.g., the same SKU) and / or type can be considered a homogeneous device. As used herein, the term "homogeneous system" refers to a system consisting only of system devices with the same architecture.

[0026] It should be noted that the terms "optimize," "optimal," and the like, as used herein, may be used to make or achieve performance as effective or perfect as possible. However, as one skilled in the art reading this document will recognize, perfection cannot always be achieved. Accordingly, these terms may also mean making or achieving performance as good or effective as possible or practical under the circumstances, or making or achieving performance better than that achievable with other settings or parameters.

[0027] Fig.1 shows an example high-level system architecture 100 and the corresponding message flow in which the present disclosure may be implemented. The architecture 100 can be logically divided into three layers: a compute layer having a plurality of nodes 122a-n and 124a-n, a services layer, and an end-user layer where distribution policies and energy budgets can be entered by end users (e.g., owners and / or administrators), and where analytics can be presented and evaluated. The services layer of the architecture 100 includes a power cap distribution system 110 connected to the HPC system and the nodes 122a-n (hereinafter referred to as compute node 122) and the nodes 124a-n (hereinafter referred to as system node 124) via a communication interface 140. The communication interface 140 can be a physical cable connection and / or a wireless communication network (e.g., a WLAN, VLAN, or the like).) act.

[0028] Nodes 122 and 124 may collectively form an HPC system 120 to execute one or more workloads or jobs by clustering the compute nodes to perform distributed computation. The HPC system may include a fabric interconnect (e.g., wireless or wired connections) connecting nodes 122 and 124 in a networked configuration to perform distributed computation. Nodes 122 may include controllable compute nodes 122, which are examples of system equipment that provide hardware resources for performing computations. Compute nodes may be implemented as CPUs, GPUs, accelerators, and the like. In one example, HPC system 120 may be a homogeneous system in which compute nodes 122 consist of nodes of the same architecture (e.g., the same type).In another example, the HPC system 120 may be a heterogeneous system in which the compute nodes 122 consist of a number of different or distinct architectures (e.g., different types of compute nodes, as illustrated in Table 1 as an example). The compute nodes 122 may include heterogeneous subsets of compute nodes, where each subset of compute nodes may be homogeneous within the entire subset.

[0029] Nodes 124 may include non-controllable system nodes, which are examples of system devices that provide hardware resources with fixed performance limits. Nodes 124 may include supporting infrastructure (e.g., power distribution, system cooling, etc.) and system components that either cannot be controlled, should not be controlled, or are classified as non-controllable, such as, but are not limited to, login nodes, network devices, system management controllers, input / output subsystems, etc. System nodes 124 may include switches, PDUs, controllers, cooling systems, etc.

[0030] While in Fig. 1 and in the description in connection with Fig.1-7, reference is made to nodes 122 and 124 as examples of system equipment, the technology disclosed herein is not so limited. System equipment may be represented as a system of a series of systems that together form an HPC system, a compute node of an HPC system, a system node of an HPC system, a compute unit, device, or component that forms a node (e.g., a compute or system node), or any component of the HPC system that consumes energy.

[0031] The architecture 100 includes the power cap distribution system 110, which communicates with the HPC system 120 and the nodes 122 and 124. The power cap distribution system 110 can be located in a public network, a private network, or a hybrid network. The power cap distribution system 110 includes a controller 115, an energy cap interface 116, an inventory interface 118, and a storage 114. The power cap distribution system 110 can be implemented as a server running in a public, private, or hybrid network. A public network can share publicly available resources / services, e.g., via the Internet, whereas a private network cannot be shared and can only offer resources / services via a private data network. In a hybrid network, public and private clouds can be shared depending on the purpose of the services.The power cap distribution system 110 may be cloud-based, which those skilled in the art would understand as being hosted, for example, on a system / server in a network (rather than on local servers / computers) and being accessible remotely. Such a cloud-based system allows access to the system from various locations, not just the location where the system is hosted (e.g., an OOB system). Thus, an end user using a mobile device or personal computer as a front-end system 130 may have access to a power cap distribution system 110. It should be noted that the power cap distribution system 110 does not need to be located on the same network in which the computing nodes 122 are distributed.

[0032] The power cap distribution system 110 and the nodes 122 and 124 can communicate with each other under various circumstances. For example, the power cap distribution system 110 can include a device gateway 117 with the power cap interface 116 and the inventory interface 118, as well as an API receiver 112. The device gateway 117 can be a mechanism / interface implemented as APIs for communicating with the HPC system 120 and the nodes 122 and 124, while the API receiver 112 can interface with the aforementioned front-end system 130, which provides access to a dashboard 132. The dashboard 132 can be hosted by the power cap distribution system 110, accessible via a web portal, or hosted locally on the front-end system 130.End users can provide inputs via the dashboard 132, the power cap distribution system 110 can receive inputs from the front-end system 130, and the power cap distribution system 110 can provide information or data about the HPC system 120 and / or the nodes 122 and 124 to the front-end system 130.

[0033] The power cap distribution system 110 may request an inventory of the HPC system resources (e.g., an inventory of the nodes 122 and / or 124) via an inventory interface 118. The HPC system 120 may respond with information and data of the various nodes 122 and 124 (e.g., architectures, identifiers, etc.), which the power cap distribution system 110 stores as inventory information in the memory 114. In an example implementation, the inventory interface 118 issues a code call (e.g., a GET command) to the HPC system 120 to retrieve a list of the nodes 122 and / or 124. In return, the HPC system 120 transmits the inventory information to the controller 115 via the inventory interface 118. The total number of compute nodes 122 and the unique identifiers of the compute nodes 122 (e.g., IP address, MAC address, etc.) can be read from the inventory information.Similarly, a count of the total number of system nodes 124 and the unique identifiers of the system nodes 124 may be maintained. The inventory may include a number of unique model or type identifiers (e.g., SKUs or other identifiers that distinguish between unique product models) that distinguish between the types of compute nodes and a number of compute nodes for each type.

[0034] The power cap distribution system 110 may also request information about the power caps of the HPC system resources (e.g., information about the power caps of compute nodes 122) via the power cap interface 116 and store the returned power cap information in memory 114. In an example implementation, the power cap interface 116 issues a code call (e.g., a GET command) to the HPC system to retrieve hardware-defined power caps of the node types on the HPC system 120. For example, the power cap interface 116 packages the unique model or type identifiers into a code call (e.g., a GET command) for power caps for identified compute node types and system node types. The HPC system 120, in turn, transmits power cap information to the controller 115 via the power cap interface 116.The power cap information may include power cap values ​​(e.g., in the case of non-controllable system nodes 124), as well as minimum and maximum power caps for each compute node type, as defined by the hardware architecture of the compute node types. In some examples, the power cap information may include a power cap delta (e.g., the difference between the maximum and minimum power caps) that defines a power cap range for each compute node type. In another example, the power cap range (or delta) may be determined from the minimum and maximum power caps. The power limits for each compute node type (or system node type) may differ from the power limits of other compute nodes (or other system nodes).The returned performance limits can be linked to the unique model or type identifier of the corresponding compute node type and / or system node.

[0035] The power cap distribution system 110 may receive power cap information as inputs at a front-end system 130, e.g., from an end user (e.g., owner, administrator, or operator of the HPC system) specifying a power budget. The end user may also enter a system-wide power cap, or the system-wide power cap may be derived from the power budget (e.g., from Equation 1 above). The power cap information may define an end-user-specified maximum power budget and / or a maximum power cap.

[0036] The power cap distribution system 110 may include, be communicatively coupled to, or otherwise have access to a storage 114. In one example implementation, the storage 114 may be implemented, for example, as one or more databases. For example, the power cap distribution system 110 may include one or more database servers that manage the storage 114. The power cap distribution system 110 may communicate data to be stored in storage 114 and / or request access to data stored in storage 114. Any suitable database may be used, including, without limitation, MySQL™, Oracle™, IBM™, Microsoft SQL™, Sybase™, Access™, and the like, including cloud-based database instances and proprietary databases.In another example, memory 114 may be implemented as random access memory (RAM) or another dynamic memory or non-transitory storage medium that can be used to store information and instructions executed by a hardware processor. In some examples, memory 114 (or a portion of memory 114) may also be implemented as read-only memory ("ROM") or another static storage device.

[0037] Fig.2 is a flowchart of an example process 200 for distributing a system power cap to compute nodes of an HPC system according to the implementations disclosed herein. The process 200 may be implemented as instructions stored, for example, in memory, that, when executed by one or more processors, perform one or more operations of the process 200. For example, the process 200 may be distributed by the power cap distribution system 110 of Fig. 1, such as by the controller 115 in communication with the front-end system 130 and the compute node 122.

[0038] The process 200 can be divided into several phases, for example, an input phase 202, a context definition phase 210, a computation phase 220, and an application phase 230. During the input phase 202, the process 200 receives inputs defining requested system performance limits and / or power budgets, distribution policies, and the like, for example, from an end user via the front-end system 130. During the context definition phase 210, the process 200 determines an execution context, such as a system-wide software configuration (e.g., the configuration of the algorithm shown in the process 200), distribution policies of the system (e.g., policies set by end users for the execution of the process 200), and the current environment (e.g., the configuration of the current system, such as the number of compute and / or system nodes, power limit ranges and / or values, the current power consumption of each node, etc.).The process 200 executes the compute phase 220 to assign power caps to the compute nodes according to the requested power cap, and during the apply phase 230, the assigned power caps are provided to the HPC system for application to the compute nodes.

[0039] By executing the phases of process 200, the implementations disclosed herein are capable of determining an optimal distribution of power caps for homogeneous or heterogeneous HPC systems with homogeneous and / or heterogeneous node architectures according to user-definable policies and extensible distribution schemes. That is, process 200 can be executed to determine power caps for each compute node in an HPC system, regardless of whether the HPC system is homogeneous or heterogeneous, such that a system-wide power cap is optimally distributed among the compute nodes of the HPC system. Process 200 can then issue instructions to the HPC system to apply the determined power caps to each individual compute node of the HPC system.The process 200 may be executed OOB and then provided as instructions to application-aware performance and power management software for execution therein, in combination with hardware-provided power control interfaces and hardware-based node power distribution logic (e.g., static and dynamic) to establish optimal node power guardrails according to a system-wide power cap and the application's power requirements. An example of aware power and power management software can be found in U.S. App. No. 17 / 337,107, the disclosure of which is incorporated herein by reference in its entirety. The application-aware in-band power and power management software may reside on the HPC system.

[0040] In operation, the process 200 receives a Requested_Power_Cap as input during the input phase 202. In an example implementation, an end user may specify a power budget for an HPC system (e.g., via the front-end system 130 of Fig.1). From the specified power budget, a Requested_Power_Cap for the controllable compute nodes on the HPC system can be determined (e.g., from Equation 1). In another example, a Requested_Power_Cap for the controllable compute nodes can be specified by an end user. In some implementations, the input can optionally include a designation of compute nodes to be targeted by process 200. This means, for example, the optional designation of the targeted compute nodes can identify the designated compute nodes as controllable, while the remaining nodes are identified as uncontrollable. If the optional designation is not used, the default configurations of "controllable" and "uncontrollable" apply.

[0041] During the context definition phase 210, the power cap distribution system 110 may obtain an inventory of the compute nodes 122 and system nodes 124 on the HPC system via the inventory interface 118. The inventory information may include a count of the total number of compute nodes and unique identifiers of the compute nodes 122, as well as the number and identifiers of the system nodes 124. The inventory may include a number of unique model or type identifiers that distinguish between node types and a number of nodes for each type. The inventory may include configuration information that identifies the system equipment and components (e.g., network switches, login nodes, compute nodes) as controllable or non-controllable power consumers.When the targeted compute nodes are determined in the input phase 202, the determined compute nodes may be designated as controllable compute nodes, while other compute nodes are designated as non-controllable.

[0042] At operation 212, the performance cap ranges for each compute node and / or system node type are identified in the inventory information. For example, the performance cap distribution system 110 may retrieve hardware-defined performance cap information for each node type via the performance cap interface 116. As described above, the performance cap information may include minimum and maximum performance caps for compute nodes 122 established according to the hardware architecture for each compute node type. Additionally, the performance cap information may include a performance cap delta representing a performance cap range between the minimum and maximum performance caps for each compute node type. In another example, the performance cap delta may be determined from the minimum and maximum power caps.The power cap information may also include system node 124 power cap values ​​set for each node type according to the hardware architecture.

[0043] At operation 214, a power cap for the controllable compute nodes may be calculated from the Requested_Power_Cap received in the input phase 202. In an example implementation, a system-wide power cap for the controllable compute nodes may be calculated based on the Requested_Power_Cap compared to an effective tunable range of power caps. For example, if the Requested_Power_Cap exceeds or is equal to the effective tunable range of power caps, the system-wide power cap for the controllable compute nodes may be set to an upper bound (e.g., the maximum) of the effective tunable range of power caps. If the Requested_Power_Cap is less than a lower bound of the effective tunable range of power caps (e.g.,a minimum), an error may be returned because a solution is not possible, the requested power cap may be set to the lower limit, and / or controllable compute nodes may be disabled to achieve the requested power cap (referred to here as "starvation"). Otherwise, the "Requested_Power_Cap" may be set as a system-wide power cap for the controllable compute nodes, from which individual power caps for each controllable compute node can be calculated as described here.

[0044] In an illustrative example, the effectively adjustable range of the power caps can be derived from the power caps of all compute nodes in the HPC system. For example, the sum of the minimum power caps for all compute nodes in the HPC system (Sum_Min) can be determined using the minimum power caps of each compute node type multiplied by the number of compute nodes of that type. Analogously, the sum of the maximum power caps for all compute nodes (Sum_Max) can be determined using the maximum power caps of each compute node type multiplied by the number of compute nodes of that type. The effectively adjustable range of the power caps can then be defined as the range between Sum_Min and Sum_Max.Therefore, if the requested power cap is greater than or equal to Sum_Max, the requested power cap is set to the maximum, and all controllable compute nodes can be set to the maximum power cap. If the Requested_Power_Cap is less than Sum_Min, an error can be returned, the Requested_Power_Cap is set to Sum_Min, and the controllable compute nodes are set to the minimum power cap, and / or the controllable compute nodes can be disabled to achieve the Requested_Power_Cap (e.g., starvation). Otherwise, the Requested_Power_Cap can be set as a system-wide power cap, from which individual power caps can be calculated for each controllable compute node.

[0045] Fig.3 shows a schematic diagram of a decision diagram 300 for determining a system-wide power cap for the controllable compute nodes according to an example implementation. The decision diagram 300 schematically illustrates at least a portion of the process 214 for calculating a power cap for the controllable compute nodes based on the requested power cap compared to an effective adjustable range of power caps.

[0046] The decision diagram 300 shows a range of all possible system power limits 310. The range of possible system power limits 310 extends from a minimum system power consumption (System_Min) 314 (e.g., zero watts) to a maximum system power consumption (System_Max) 312 and includes all possible amounts of power consumption in between. The System_Max 312 can be determined according to Equation 1. For example, System_Max 312 can be determined as the sum of all maximum power limit values ​​of controllable compute nodes and all power limit values ​​of non-controllable system nodes. The decision diagram 300 also shows Sum_Max 322 and Sum_Min 324 as described above, which define an effective tunable range 320. Above Sum_Max 322 (e.g., the upper limit of the effective tunable range 320) lies a non-consumable range, and below Sum_Min 324 (e.g.,the lower limit of the effective tunable range 320) is a non-tunable range. As described above, the power cap for the controllable compute nodes may be set to Sum_Max 322 if the requested power cap is equal to or higher than Sum_Max 322 (e.g., within the non-consumable range). In some scenarios, it may be possible for System_Max 312 to be less than Sum_Max 322, in which case System_Max 312 may become the upper limit of the effective tunable range 320. As described above, the requested power cap may not be supported by the controllable nodes if the requested power cap is less than Sum_Min 324 (e.g., within the non-tunable range).In this case, an error may be returned to the front-end system 130 indicating that the requested power cap is not available without reducing the number of controllable compute nodes or disabling a number of compute nodes to lower Sum_Min 324 (e.g., starvation). Otherwise, if the requested power cap is equal to or greater than Sum_Min 324 or less than Sum_Max 322, the requested power cap is set as the power cap for the controllable compute nodes and can be used downstream to calculate individual power caps for each controllable compute node.

[0047] Back to Fig.2: In operation 222, a solution for assigning a power cap to the controllable compute nodes may be calculated based on the results of operation 214 and the application of a distribution scheme. For example, if a power cap is determined as above in operation 214 (and with reference to Fig.3), a power cap distribution may be calculated for each controllable compute node, assigning a power cap value to each controllable compute node. The distribution of the power cap among the controllable compute nodes may be determined using a distribution scheme. The distribution scheme, according to various implementations, calculates the allocation of power caps for the controllable compute nodes by determining an amount to increase a power cap of each controllable compute node until the system power consumption when operating at the highest allowable power consumption equals the requested power cap. This disclosure includes a non-exhaustive list of example distribution schemes that may be used by process 200, such as "even_split," "equal_percentage," "count_down," and "delete_by_delta."While they are different schemes or algorithms, each of them determines an incremental step (which may be the same or different for different compute node types) to be applied to the power cap of the controllable compute nodes, based in part on the power cap ranges of each controllable compute node type.

[0048] The distribution schemes may be stored as executable instructions in the distribution scheme file 224. The distribution scheme file 224 may be accessed by operation 222 according to a distribution policy determined in the context definition phase 210 based on input from an end user, e.g., via the front-end system 130. That is, operation 222 may access the distribution scheme file 224 and execute one or more distribution schemes stored therein to assign power caps to the controllable compute nodes. Executing a distribution scheme includes calculating an increment in power caps that may be assigned to each controllable compute node such that the power cap determined in operation 214 may be optimally distributed among the controllable compute nodes according to the distribution policy.In various examples, each controllable compute node type may be assigned a power cap that applies to all controllable compute nodes of that type. Thus, while the power caps are different for different compute node types, the power caps may be the same for all compute node types.

[0049] According to the implementations disclosed here, a number of distribution schemes may be used, and an optimal one may be selected. For example, different power cap allocations may be calculated using different distribution schemes, and an optimal distribution may be selected that provides the best overall power utilization (as in Fig.4 below) relative to the total power utilization of the other distribution schemes. The total power utilization can be defined according to the end-user's power management goal (e.g., a trade-off between efficiency in power consumption and performance in time to resolution), the end-user's distribution policy (e.g., favoring high-power compute nodes over low-power compute nodes), and the type of hardware (e.g., providing more power to better-utilized compute node architectures). In an example implementation, the total power utilization can be a measure of how close the sum of the allocated power caps comes to the requested power cap without exceeding the requested power cap. This definition of the total power utilization ensures maximum utilization of the Requested_Power_Cap.

[0050] As previously mentioned, example distribution schemes include the even_split scheme, the equal_percentage scheme, the count_down scheme, and the delete_by_delta scheme, each of which is described in detail below. While each of these is a different scheme for distributing the requested power cap, each determines an increment (or allocation) that can be applied to the power caps of the controllable compute nodes, based in part on the power cap ranges of each controllable compute node. While this disclosure provides certain example distribution schemes, the implementations disclosed herein are not limited to these example schemes. Any distribution scheme may be used as desired for a particular application.Thus, the present disclosure enables the simulation and evaluation of different system power distribution schemes, enabling the adaptation of power management system solutions to end-user requirements.

[0051] Now for the example distribution schemes. One example is the even_split scheme, where the difference between Requested_Power_Cap and Sum_Min is distributed evenly among all controllable compute nodes. For example, the number of controllable compute nodes in the HPC system can be determined and a Sum_Min can be determined as described above. The difference between Requested_Power_Cap and Sum_Min can be calculated and divided by the number of controllable compute nodes. The resulting value is assigned to each controllable compute node as the power cap for a corresponding controllable compute node. This scheme can be optimal when the power cap ranges of the controllable compute nodes overlap or when the HPC system is homogeneous.

[0052] Another example of a distribution scheme is the equal_percentage scheme. This scheme calculates a power cap delta (or range) for each controllable compute node and divides the power cap delta into n discrete steps. In one example, n discrete steps might be 10,000 discrete steps, but any number of steps can be used. Starting with a maximum power cap for each controllable compute node, this scheme decreases a power cap for each controllable compute node until the sum of the power caps for all controllable compute nodes is less than or equal to the requested power cap. In some implementations, the discrete steps may result in power cap values ​​that have a decimal value. In this case, the power cap values ​​may be truncated to an integer, which may be required for hardware settings.The reason for this is that hardware implementations may only allow whole wattages (e.g., in steps of one). 10,000 discrete steps, also called a "decrease quantum," were chosen to ensure a sufficiently high resolution in this example so that all discrete steps for all controllable compute nodes would be no larger than 1 W. If the decrease quantum is larger than 1 W (e.g., a 2 W decrease), the solution may not be able to consume all available power (i.e., it may not be able to optimally utilize all requested power).

[0053] Another example of a distribution scheme is the count_down scheme. This scheme decreases the power cap values ​​for each controllable compute node by one watt from the maximum power cap until the sum of the power caps is less than or equal to the requested power cap. In a sample implementation, the wattage is 1W or an integer number of watts, because integer wattages are required in the hardware settings. The count_down scheme is similar to the equal_percentage scheme, but instead of all compute node types having the same number of discrete steps, each compute node type has a different number of available steps, allowing compute node types with smaller power cap deltas to reach a minimum power cap, as defined by the hardware architecture, before those with larger ranges.For example, node type 1 with a delta of 575 W may be exhausted before node type 2 with a delta of 1990 W (e.g., set to the minimum power cap), as shown in Table 1.

[0054] Another example of a distribution scheme is a delete_by_delta scheme. In this scheme, controllable compute nodes are divided into groups based on a difference (e.g., delta) between the minimum and maximum power cap values. Consequently, controllable compute nodes with identical power cap deltas are placed in the same group. For each group, the range between the minimum and maximum power cap values ​​is calculated, and the power cap values ​​for all controllable compute nodes are set to the maximum power cap. Then, starting with a group of controllable compute nodes with the smallest delta, the power caps of the entire group are set to the minimum power cap value.After the group is set, the sum of all controllable compute nodes is calculated, and a determination is made as to whether the sum of the power cap values ​​is less than or equal to the requested power cap. If the sum is greater than the Requested_Power_Cap, the scheme continues with the next group in the sequence (e.g., the next smallest delta). The scheme is repeated until the sum of the power cap values ​​is less than or equal to the requested power cap.

[0055] The "delete_by_delta" scheme is reversible, meaning that instead of processing groups by the smallest delta (denoted as "delete_by_delta_smallest-to-largest"), groups can also be processed by the largest delta, so that the groups with the largest delta are removed first, followed by the group with the next largest delta (denoted as "delete_by_detal_largest-to-smallest"). This scheme aims to maintain the largest (or conversely, the smallest) range of power caps at maximum performance, which would cause the system to prioritize powering compute nodes with a larger (or smaller) range of power caps.

[0056] Other variations of delete_by_delta are possible. For example, but not limited to, delete_by_component_count_least-to-most (e.g., grouping controllable compute nodes by the number of compute nodes of each type and setting the power caps of the group with the fewest number of compute nodes to the minimum power cap values, then moving on to the next group); delete_by_component_count_most-to-least (e.g., the inverse of delete_by_component_count_least-to-most); delete_by_max_power_cap_largestto-smallest (e.g., grouping controllable compute nodes by the maximum power cap value and setting the power caps of the group with the largest maximum power cap value to the minimum power cap values, then moving on to the next group); delete_by_max_power_cap_largest-to-smallest (e.g., the inverse of delete_by_max_power_cap_smallest-to-largest); delete_by_min_power_cap_largest-tosmallest (e.g., grouping the controllable compute nodes by the value of the minimum power capacity and setting the power capacities of the group with the largest value of the minimum power capacity to the minimum values ​​of the power capacity, then moving on to the next group); delete_by_min_power_cap_largest-to-smallest (e.g., the inverse of delete_by_min_power_cap_smallest-to-largest); among others.

[0057] Fig. Figure 4 is a bar chart showing the overall power utilization of various distribution schemes for allocating a power cap to controllable compute nodes according to an example implementation. Several distribution schemes were simulated in operation 222 to calculate power cap allocations that distribute a system-wide power cap to controllable compute nodes, and the bar chart in Fig.Figure 4 provides a visual comparison to evaluate the performance of each distribution scheme. Base_solution refers to setting the power caps for each controllable compute node to the corresponding maximum power cap and, if a maximum setting is not possible, setting the power caps for each controllable compute node to the corresponding minimum power cap.

[0058] In the example implementation of Fig. 4, the compute node profiles shown in Table 1 were used for the simulations. To create Fig.4, a series of tests were performed across the entire range of system power cap limits (e.g., Sum_Min to Sum_Max), from just below the minimum valid power cap solution (e.g., all nodes at Min, also called Sum_Min) to just above the maximum valid power cap solution (all nodes at Max, also called Sum_Max). The tests were divided into whole percentages, resulting in 70 separate test cases (e.g., 70 unique possible Requested_Power_Cap values). Each scheme was then executed using the 70 test cases, and the solution utilization per scheme (e.g., per test case) was determined. For each scheme, a mean, standard deviation (STDDEV), and variance over the population of the total power utilization were calculated for each test case. Fig.Figure 4 shows the mean and standard deviation, while the mean, standard deviation, and variance are presented in Table 2 below. The mean indicates how effective a particular scheme was across all test cases, where a score of 1.0 would mean that the scheme allocated 100% of the requested power cap. The standard deviation is the dispersion of solution utilization from the mean, and the variance measures the average extent to which each solution utilization deviates from the mean. For both the standard deviation and variance, a lower value indicates better performance. Table 2 Distribution scheme Utilization average Use of STDDEV Deviation from capacity utilization Basic solution 6.35E-01 2.64E-01 6.97E-02 Countdown 1.00E+00 2.04E-04 4.18E-08 Delete_by_delta_largestto_smallest 6.02E-01 2.17E-01 4.73E-02 Delete_by_delta_smallestto_largest 6.49E-01 1.78E-01 3.17E-02 Same_Percentage 1.00E+00 2.71E-04 7.37E-08 Straight_Split 9.99E-01 3.4E-04 1.74E-01

[0059] As can be seen from Table 2 and Fig.As can be seen in Figure 4, the equal_percentage scheme and the count_down scheme provide the most optimal (e.g., most efficient in terms of meeting the requested power cap) power utilization. Although both distribution schemes provide almost the same utilization of the available power, they have different impacts on application performance. For example, the equal_percentage distribution scheme can provide a more uniform power reduction across all controllable compute node types because the power reduction is based on the same percentage of the maximum power cap value. In contrast, the count_down scheme reduces power by a fixed amount regardless of the maximum power cap. Therefore, the count_down scheme exhausts a compute node with a smaller power cap delta (e.g., node type 1 in Table 1) before those with a larger power cap delta (e.g.,Node type 2 in Table 1) by setting compute nodes to the minimum power earlier than in the equal_percentage distribution scheme. Therefore, the count_down scheme may favor compute node types with larger power cap deltas. Alternatively, the equal_percentage scheme may ensure that all compute node types have at least some power above the minimum, as long as the system power cap is greater than Sum_Min. Furthermore, it is possible that sites optimize according to other criteria, such as providing maximum performance to compute nodes with accelerators compared to nodes with only CPUs, although the above optimization is for maximum solution utilization.

[0060] Back to Fig.2: Once the power cap allocation has been calculated in operation 222, process 200 proceeds to 232, where the power cap distribution solution is applied to the controllable compute nodes. For example, the power cap values ​​allocated according to a distribution scheme in operation 222 may be communicated to the HPC system, and the power cap values ​​may be set in the hardware of each compute node. As mentioned above, in the case where a list of target compute nodes was identified in the input phase 202, the distribution scheme assigns power caps for these target compute nodes (e.g., by classifying all other nodes as non-controllable) and sets the power caps accordingly. In an example implementation referring to Fig.1, the controller 115 may execute the process 200 and interface with the HPC system 120 to transmit a message packaged with instructions for assigning power caps to each controllable compute unit 122 via the power cap interface 116. The HPC system 120 may then forward instructions to the controllable compute nodes, each of which unpacks the instructions and sets its respective power caps according to the instructions from the controller 115. In this way, each compute node 122 may be controlled to set its own power cap according to the instructions received from the controller 115.As a result, the HPC system can be operated such that the system-wide energy consumption does not exceed the requested power cap due to the power caps set in the controllable compute nodes, while the available power is efficiently used to meet the requested power cap without exceeding the requested power cap.

[0061] According to various implementations, the process 200 can be applied to an entire HPC data center, recursively down to individual systems, recursively down to a single compute node. For example, individual accelerators (e.g., compute units) can be subjected to a power limit. Each hierarchy level can be viewed as a system that can be divided into nodes representing an energy consumer (e.g., uncontrollable consumers and controllable consumers). Therefore, solutions for one level can be recursively applied to other levels of the power management hierarchy, as shown in Equations 2-4. For example, Equation 2 below shows a first level (plant power), which can include a plurality of systems that consume energy. The majority of systems at this level can be viewed as compute nodes, which can include controllable and uncontrollable consumers.At the next level (e.g., Equation 3), a system of the device of Equation 2 may comprise a plurality of compute nodes, which may include controllable and non-controllable loads. At the next level (e.g., Equation 4), a particular compute node of the system of Equation 3 may comprise a plurality of compute units (or components). The plurality of compute units at this level may include controllable and non-controllable power loads. Facility Power=∑i=1Cnoncontrollable consumersMax Power+∑j=1SSystem Power System Power=∑i=1Cnoncontrollable consumersMax Power+∑j=1NCompute Node Power Compute Node Power=∑i=1Cnoncontrollable consumersMax Power+∑j=1CCompute Unit

[0062] Accordingly, the implementations disclosed herein can be used by the end user to set a system-wide performance cap, from which individual performance caps for compute nodes in the system can be determined and optimized according to a distribution scheme. For a given system-wide performance cap, multiple distribution solutions can be simulated based on different distribution schemes. The simulated distributions can then be evaluated based on solution utilization, which can be defined according to a distribution policy for the end user. The most appropriate distribution (e.g., the optimal resource utilization according to the end user's distribution policy) can be automatically applied to all controllable compute nodes of the system based on the compute node type.

[0063] According to various implementations, performance caps for compute nodes can be set through the OOB execution of the examples disclosed here. These performance caps for compute nodes can become guardrails and starting setpoints when an application-aware in-band component is available. For example, a combination of OOB control and an application-aware in-band component could be used to manage the performance caps for compute nodes according to the requirements of the running application. For example, an application could require only two of four GPUs to execute a job and shift performance from those nodes of the job to other nodes in the system.

[0064] Process 200 may also be automated in response to detecting a trigger event. For example, process 200 may optionally include detecting one or more trigger events in operation 204 that may trigger execution of process 200. Operation 204 may be optional, as indicated by the dashed lines. The trigger event may be a system event, such as starting a job or workload; receiving a requested power cap or budget, such as an update to a previously entered requested power cap / budget; or a period of time, such as detected by a timer. In the case of a timer, which may be included in controller 115, for example, a period of time may be set in advance, defining an interval between repeated executions of process 200.

[0065] Operation 204 may also include obtaining inputs from input phase 202, such as requested system power caps and / or power budgets, distribution policies, and the like. In one example, receiving one or more end-user inputs may act as a trigger event. For example, an end user may enter a requested system power cap (e.g., Requested_Power_Cap, as described above) and / or a power budget, which triggers execution process 200 according to the input. In some examples, the input may be an updated system power cap or power budget, which triggers reoptimization of the power cap allocation.

[0066] If a detected trigger event is based on the expiration of a time period, a job / workload, or otherwise not based on updating end-user input, the input phase 202 may include retrieving previously stored power budgets and / or requested power caps (e.g., stored in memory 114) for use in the process 200.

[0067] Fig. 5 is a flowchart of an example process 500 for distributing a system power cap among pools of compute nodes of an HPC system in accordance with the implementations disclosed herein. The process 500 may be implemented as instructions stored, for example, in memory, which, when executed by one or more processors, perform one or more operations of the process 500. For example, the process 500 may be distributed by the power cap distribution system 110 of Fig.1, such as by the controller 115 in communication with the front-end system 130 and the computing node 122. The processing described herein with reference to Fig. 5 is not limited to the particularly illustrated sequence, and the associated operations may be performed in other suitable sequences, in parallel or otherwise.

[0068] In this example, an HPC system, such as HPC system 120, includes a plurality of controllable compute nodes (e.g., compute node 122) and non-controllable system nodes (system node 124). Nodes 122 and / or 124 may be grouped into node pools, where each pool may include one or more compute nodes 122 and / or one or more system nodes 124. Each pool may include any number of nodes. The configuration of each pool may be determined by the end user during system startup. In some implementations, nodes 122 and / or 124 may be clustered based on a job or workload. For example, a particular pool may be defined to include nodes 122 and / or 124 for executing a job or workload.As another example, a group of pools may be defined for executing a job or workload, with each pool being assigned to a compute task of the job or workload. In another example, nodes 122 and / or 124 may be clustered based on geographical proximity. For example, a distributed HPC system may be located in different data centers with different geographical locations. Each pool may include those nodes that are geographically close together. In another example, pools may be defined based on node type and / or controllability. For example, pools may be defined to include nodes of the same node architecture (e.g., homogeneous pools). For another example, a particular pool may include only controllable compute nodes, while another pool may include only non-controllable system nodes.In another example, a pool may be created that contains nodes that are not to be managed (e.g., a "no manage" pool). In this case, controllable compute nodes 122 may be assigned to a "no manage" pool and treated as non-controllable nodes. Pools may be defined according to the examples above, other configurations, or any combination thereof for a desired application.

[0069] The description in connection with Fig. 5 (and the following Fig. 6 and Fig.7) While referring to pools of compute and / or system nodes, this is only one example of the technology disclosed herein. The disclosed technology is not intended to be limited to pools of compute nodes. For example, the methods disclosed herein may be applied to pools of any system devices, such as, but not limited to, compute nodes, system nodes, compute units, system devices, or any power-consuming components of the HPC system.

[0070] In operation 502, a trigger event may be detected that initiates the initiation of process 500. The trigger event may be a system event, such as the start of a job or workload; the receipt of a requested performance cap or budget, such as an update to a previously entered requested performance cap / budget; or a period of time, such as that detected by a timer. In the case of a timer, which may be included in a controller 115, for example, a period of time may be set in advance that defines an interval between repeated executions of process 500. The period of time may be any period of time desired for a particular application, for example, but not limited to, once per week, once per day, once per hour, once per minute, every 10 seconds, every 1 second, etc.In an illustrative example, the time period might be 20 seconds, so process 500 might be executed every 20 seconds. In any case, after each time period, process 500 is executed, providing continuous power management by enabling dynamic optimization of the system's power cap allocation. This means the system can continuously monitor power consumption and current configurations to ensure optimal allocation of power caps to controllable compute nodes in the system.

[0071] Operation 502 also includes determining user-defined system configuration parameters, such as a requested system performance cap, performance budgets, distribution policies, pool labels, pool priorities, pool-specific performance caps, and the like. A pool-specific performance cap (or pool performance cap) refers to a performance cap set for a particular pool. Each pool of nodes can be assigned a corresponding pool performance cap. However, if no pool performance cap is assigned, the pool performance cap is set to the maximum performance cap of that pool (e.g., the sum of all maximum performance caps of the nodes that comprise the pool). The user-defined system configuration parameters can be received as inputs, for example, from an end user via the front-end system 130.In one example, the receipt of one or more user-defined system configuration parameters may act as a triggering event. For example, an end user may enter a requested system power cap (e.g., Requested_Power_Cap, as described above), a requested power budget, and / or one or more pool power caps, which trigger process 500 in response to the input. In some examples, the input may be an updated system / pool power cap or an updated power budget, which triggers re-optimization of the power cap allocation. As another example, pool labels and / or pool priorities may be entered and / or updated by the end user, triggering process 500.

[0072] If a detected trigger event is based on the expiration of a time period, a job / work effort, or otherwise not based on updating custom system configuration parameters, operation 502 may include retrieving previously stored custom system configuration parameters (e.g., stored in storage 114) for use in process 500. Alternatively, the custom system configuration parameters may be retrieved at a later time in process 500.

[0073] In operation 504, an inventory of the nodes in the system is performed. In the case where a previous instance of process 500 was executed, the inventory obtained in operation 504 may be used to update a previously obtained inventory stored, for example, in memory 114. Operation 504 may be similar to the steps described above in connection with Fig.2. That is, the power cap distribution system 110 may, for example, obtain an inventory of the compute nodes 122 and system nodes 124 via the inventory interface 118. The inventory information may include a count of the total number of compute nodes and unique identifiers, as well as the number and identifiers of the system nodes 124. The inventory may include a number of unique model or type identifiers that distinguish between the different types of compute nodes, as well as a number of compute nodes for each type. The inventory may include configuration information that identifies the system components as controllable or non-controllable power consumption components. In the case where a previous instance of the process 500 was executed, the inventory obtained at operation 504 may be used to update a previously obtained inventory stored, for example, in memory 114.

[0074] In operation 506, performance limit ranges are determined for each compute node and / or system node type identified in the inventory information. Operation 506 can be performed similarly to operation 212 of Fig. 2. For example, the power cap distribution system 110 may retrieve hardware-defined power cap information for each node type via the power cap interface 116. As described above, the power cap information may include minimum and maximum power caps for compute nodes 122. Additionally, the power cap information may include a power cap delta for each compute node type, or the power cap delta may be determined from the maximum and minimum power caps. The power cap information may also include power cap values ​​for system nodes 124.

[0075] In operation 508, power pool and node power limits are determined. Operation 508 can be executed similarly to operation 212 in Fig.2. For example, the power cap distribution system 110 may use the power cap interface 116 to obtain the current power cap settings of all nodes 122 and 124 on the HPC system. Furthermore, the power cap distribution system 110 may retrieve the current power cap settings of each pool of nodes via the power cap interface 116 or may be determined from a summation of the current power cap settings of the nodes assigned to each pool. In some cases, the current power cap settings may be the result of a previously executed iteration of process 500. The current power cap settings may be incorporated into the power cap information, e.g., by appending the power cap settings to the maximum and minimum power cap values ​​determined in operation 506.

[0076] At operation 510, the current system-wide configuration status is determined. The system-wide configuration status may be obtained, for example, as configuration information defining the current configuration of the system, such as the number of compute and / or system nodes, power cap ranges and / or values, the current power consumption of each node, etc. The system-wide configuration may also include information identifying each pool and the corresponding nodes 122 and / or 124 clustered within each pool. In an example implementation, storage 114 may include pool identifiers representing each pool, where each pool identifier is assigned to one or more nodes 122 and / or 124 by linking to the unique identifiers of the one or more nodes 122 and / or 124.The system-wide configuration may also include priority labels associated with each pool, where the priority label indicates a priority level assigned to each respective pool. Each priority level may be assigned to one or more pools (e.g., multiple pools or a pool may be assigned a particular priority level). In some implementations, such as when custom system configuration parameters were not defined or retrieved in operation 502, operation 510 may include retrieving previously set (e.g., current) custom system configuration parameters, such as by issuing a code call (e.g., a GET command) to storage 114 to retrieve the previously set custom system configuration parameters. Further details regarding pool creation are provided below in connection with . Fig. 6 described.

[0077] In some implementations, controllable compute nodes can be pooled, and all non-clustered compute nodes can be added to a reserve pool. The reserve pool can include non-controllable system nodes as well as non-clustered compute nodes. In this example, non-clustered compute nodes are then treated as non-controllable system nodes, for which a performance cap value is set as the maximum performance cap value of the performance cap range of the non-clustered compute nodes.

[0078] In another example, controllable compute nodes not grouped in a user-defined pool can be grouped into a residual pool separate from non-controllable system nodes. The compute nodes in the residual pool can be assigned performance caps using the optimization techniques disclosed here, but with the lowest priority, so that the residual pool is the last pool to be assigned, as described below.

[0079] In operation 512, process 500 determines whether the requested performance cap for the entire system (e.g., from the user-defined system configuration parameters) is within an effective tunable range. That is, if the requested performance cap exceeds or is equal to the effective tunable range of the performance caps, the system-wide performance cap for the controllable compute nodes may be set to an upper bound (e.g., the maximum) of the effective tunable range of the performance caps. If the requested performance cap is less than a lower bound of the effective tunable range of the performance caps (e.g.,a minimum), an error may be returned because a solution is not possible under the current system configuration, the requested power cap may be adjusted to the lower limit, and / or controllable compute nodes may be disabled to achieve the requested power cap. Otherwise, the requested power cap is assumed to be within the effectively tunable range, and the requested power cap may be set as the system-wide power cap for the controllable compute nodes, from which the power caps for the individual pools can be calculated as described here. Further details on the determination in Operation 512 are provided above in conjunction with the . Fig. 2 and Fig. 3.

[0080] In some implementations, in response to determining that the Requested_Power_Cap is less than a lower bound of the effective tunable range, operation 512 may also include generating a recommendation for a corrective action and transmitting the recommendation to the front-end system 130 for consideration by an end user. For example, a corrective action may include increasing the Requested_Power_Cap to at least the lower bound (or higher). In another example, operation 512 may determine one or more compute nodes (or pools) to be disabled to reduce the lower bound to equal or lower than the Requested_Power_Cap and provide a recommendation identifying the particular compute nodes (or pools) to the end user via the front-end system 130.The end user may accept the recommendation, enabling the calculation of a solution and allowing process 500 to proceed to operation 514.

[0081] If the requested performance cap is within the effective tunable range, in operation 512, the performance caps for each pool (including a residual pool, if any) are tabulated with a pool minimum performance cap value (Pool_Sum_Min). The pool minimum performance cap may be a summation of the minimum performance caps of all compute nodes included in each respective pool. That is, for each pool, a sum of the minimum performance caps for all compute nodes may be determined by multiplying the minimum performance caps of each compute node type by the number of compute nodes of the respective type that comprise the respective pool.

[0082] However, if the determination in operation 512 is "no" (e.g., because the requested power cap is outside the tunable range), the process continues with operation 524. If the requested power cap exceeds or is equal to the effective tunable range of power caps, all controllable compute nodes may be set to the maximum power cap. If the requested power cap is less than a lower limit of the effective tunable range of power caps, an error may be returned, the requested power cap may be set to the lower limit, and / or controllable compute nodes may be disabled to achieve the requested power cap, and the process may be repeated.

[0083] At operation 514, each pool is considered iteratively in order of priority level. For example, each pool may be assigned a priority level (e.g., in the custom system configuration parameters) that ranks the pools in a prioritized order with respect to power consumption (e.g., pools that a user may allow or desire to have priority access to power). The highest priority pool(s) may be considered first, and the remaining pools are traversed according to the prioritized order until all pools are considered (or no more power budget is available to allocate to subsequent pools).

[0084] In operation 516, it is determined whether a pool delta for a current pool is less than an available power budget. The pool delta refers to the absolute difference between a Pool_Sum_Max and a Pool_Sum_Min. Pool_Sum_Min is described above, and Pool_Sum_Max refers to the sum of the maximum power caps for all compute nodes, which can be determined by multiplying the maximum power caps of each compute node type by the number of compute nodes of that type that comprise the current pool.The available power budget is the power budget remaining from a requested power budget after subtracting the power cap values ​​of all uncontrollable compute nodes (and the maximum power cap values ​​of all controllable system devices deemed unmanageable) and the power cap values ​​assigned to each pool from a requested power budget. Thus, for a first iteration, where all pools are enumerated at Pool_Sum_Min, the available power budget is the power budget remaining after subtracting the sum of the power cap values ​​of all uncontrollable compute nodes (and the maximum power cap values ​​of all controllable compute nodes deemed unmanageable) and the sum of all Pool_Sum_Mins for the pools (e.g., reserve power).) In this example, the requested power budget is provided as the power cap determined in operation 512. After each iteration, the Pool_Sum_Min for a considered pool is replaced by the sum of the allocated power caps, as described below in operation 522.

[0085] If the pool delta for the current pool is equal to or greater than the available power budget, the pool power cap is calculated at operation 518 using a maximum power cap. For example, the power cap for each controllable compute node of the current pool is set to a maximum power cap value, and the power cap for the pool can be calculated as the sum of the power caps of the controllable compute nodes (e.g., Pool_Sum_Max).

[0086] If the pool delta is smaller than the available power budget, the pool's power cap is adjusted in operation 520, e.g., by increasing the power cap from the minimum power cap value. For example, operation 520 may include calculating the optimized power caps of the controllable compute nodes for the current pool. Operation 520 may be substantially similar to operation 222 of Fig. 2, where a solution for allocating power limits to the controllable compute nodes can be calculated based on the application of one or more distribution schemes, as described above in connection with Fig.2. The current pool at operation 520 may be treated as the system in the context of operation 222, so that an optimal allocation of power caps for controllable compute nodes may be calculated. Operation 520 may access the distribution scheme file 224 according to a distribution policy specified in the user-defined system configuration parameters to execute distribution schemes, as described above in connection with the Fig. 2 and Fig. 4 described.

[0087] In any case, in operation 522, a remaining power budget is tabulated and set as the updated available power budget for the next pool according to the prioritized order. For example, the power caps assigned to a pool's controllable compute nodes may be summed and subtracted from the available power budget from operation 514. Process 500 then repeats steps 516-522 for the next pool, which may include applying the same or a different distribution scheme to the next pool. That is, the distribution schemes applied to each pool may be the same or different, depending on the distribution policies specified by the end user.

[0088] Once all pools have been considered (or no more power budget is available), process 500 proceeds to operation 524, where the calculation of the solution for allocating the power cap is completed and an optimal solution is achieved, as described above. In operation 526, the solution for distributing the power cap may be applied system-wide to the controllable compute nodes. Operation 526 may be essentially similar to operation 232 of Fig. 2 correspond.

[0089] In operation 528, the system configuration states are updated and saved. The system configuration, pool boundaries, per-pool performance limits, and node performance limits, for example, may be saved in memory 114 for later access, e.g., in a subsequent iteration of process 500.

[0090] If multiple pools are assigned to the same priority level, each pool of the same priority level can be processed concurrently. For example, a power budget can be assigned to each priority level (e.g., based on input from an end user). For a given priority level, in operation 514, the assigned power budget is distributed among the pools of the priority level, and then the power cap distributions per pool are determined. For example, a power budget per pool can be determined from the priority level power budget by subtracting a sum of the Pool_Sum_Mins of all pools assigned to the priority level and distributing this result among the pools. The resulting power budget per pool can be determined according to one of the distribution schemes described here (e.g., "even_split," "equal_percentage," "count_down," and "delete_by_delta," etc.).Once a per-pool performance budget is determined for each pool, a distribution of performance caps for the nodes of each pool can be determined by operations 516-520, with each pool processed in parallel.

[0091] In some implementations, a pool may comprise a plurality of nodes, which in turn may be divided into a number of subpools. In this case, the process 500 may be performed for the subpools by treating the subpools as the pools described in 500. Thus, the method 500 may be applied to any number of hierarchical levels of pool abstractions, which may be subdivided to any desired level.

[0092] Although process 500 is described as calculating a minimum pool power cap in operation 512 and then adjusting the power cap values ​​until the assigned power cap and the sum of the power cap values ​​of the non-controllable system nodes reach the requested power budget, other implementations are possible within the scope of this disclosure. For example, operation 512 may include tabulating pools at Pool_Sum_Max and then iterating over the pools in reverse priority order (e.g., lowest priority pools first). For each pool, power cap values ​​are calculated at operation 520 that reduce the power caps from the maximum power cap values ​​to an optimal distribution, and an updated power budget is tabulated at operation 522.The process continues iteratively through each pool in reverse priority order until the updated power budget at operation 522 reaches the requested power budget.

[0093] Fig. 6 is a sequence diagram of an example message flow 600 illustrating the operations for creating a pool according to the implementations disclosed herein. Fig. Figure 6 shows components of the system architecture, such as architecture 100. In particular, Fig. 6 a front-end system 130 communicatively coupled to the power cap distribution system 110, which includes an API receiver 112, a controller 115 and a memory 114, as described above in connection with Fig. 1 described.

[0094] In operation, the front-end system 130 transmits a message 602 to the architecture 100. The message 602 includes a request to create a pool and to assign nodes (e.g., compute node 122 and / or system node 124) to the requested pool. Message 602 may include a payload that contains information to identify the one or more nodes to be grouped into the requested pool, e.g., by listing each node's unique identifiers. In some implementations, the payload of message 602 may also include a description of the requested pool; a management flag to toggle whether the pool should be managed or not (e.g., if set to "True," the pool can be managed; if set to "False," the pool is not managed); the requested power cap and / or power budget (e.g.,B, a power cap may be specified as an upper and lower limit); a priority level specified for the requested pool; and an identification of a distribution scheme to be applied (if none is specified, an optimal distribution scheme may be determined by comparing the results of distribution schemes, as described above). The message 602 is received by the API receiver 112, which validates the request in process 604. For example, the API receiver 112 verifies that the message 602 is complete and processable (e.g., by verifying that the information contained in the payload is not nonsense or otherwise unknowable to the energy cap distribution system 110 and thus can be processed). In one example, the message 602 is generated in response to input from an end user on the dashboard 132 indicating that nodes should be clustered into a requested pool.

[0095] After validation, API receiver 112 forwards the request to create a pool as message 606 to control unit 115. Control unit 115 validates message 606 (e.g., similar to the validation in process 604) and, after validating message 606, creates a pool identifier and links each node identified in message 606 to the pool identifier, thereby creating the requested pool. In an example implementation, the created pool identifier is linked or otherwise connected to the unique identifiers of the nodes listed in message 606. In an example implementation, all nodes 122 and 124 may be assigned to a reserve pool (e.g., based on the information obtained via inventory interface 118).Upon receiving message 606, the control unit 115 transfers the nodes identified in message 606 to a created pool by linking the nodes' unique identifiers to the pool identifier. The control unit 115 then stores the pool in memory 114 by storing the identifier mappings.

[0096] Once the pool is created and stored in storage 114, an acknowledgment that the pool has been created is transmitted to the front-end system 130. For example, the storage 114 sends message 612 with an acknowledgment that the pool has been saved. The controller 115 then creates a message 614 acknowledging that the pool has been created and saved in storage 114, which is transmitted to the API receiver 112. The API receiver 112 packages the acknowledgment in message 616 along with an identification of the pool identifier, which is sent back to the front-end system 130 as message 616. The pool identifier can then be made available to the end user via the dashboard 132 running on the front-end system 130.

[0097] Message flow 600 may be executed multiple times to create a number of pools. Furthermore, message flow 600 may be executed multiple times in parallel, sequentially, or in any combination thereof to create a number of pools concurrently and / or sequentially. In some examples, message 602 may include a request to create a number of pools, with a list of nodes to be clustered in each pool. Message flow 600 may then be executed to create the number of pools concurrently.

[0098] While Fig.6 shows an example of creating a requested pool, as previously mentioned, reserve and / or residual pools may also be provided. That is, based on a request to create a pool, the control unit 115 may create the pool in process 608, for example, by transferring identified nodes from a reserve pool of all nodes in the system, leaving all nodes not identified in the create-pool message 606 associated with the reserve pool. In another example, the control unit 115 may create a requested pool in process 608 and simultaneously create a residual pool of non-clustered nodes not identified in the create-pool message 606. The residual pool may be assigned a pool identifier, which may be linked to unique identifiers of all non-clustered compute nodes to create the residual pool.

[0099] Fig. 7 is a sequence diagram of an example message flow 700 illustrating the operations for distributing system power caps according to the implementations disclosed herein. Fig. Figure 7 shows components of the system architecture, such as architecture 100. In particular, Fig. 7 a system for distributing energy caps 110, including, but not limited to, controller 115, memory 114, inventory interface 118 and energy cap interface 116, as described above in connection with Fig. 1 described.

[0100] In operation, in process 701, a trigger event from controller 115 may be detected or otherwise captured. Also in 502, user-defined system configuration parameters are determined, such as requested system power limits and / or power budgets, distribution policies, pool labels (if any), pool priorities (if any), and the like. The user-defined system configuration parameters may be received as inputs, for example, from an end user via front-end system 130. For example, process 701 may be an example of process 502 of Fig. 5 and / or process 204 of Fig. 2 be.

[0101] The controller 115 transmits message 702 to the inventory interface 118 to request the inventory of an HPC system. In an example implementation, the message 702 includes a code call (e.g., a GET command) that requests inventory information of the HPC system. In response to the message 702, the inventory interface 118 performs the process 704 to update the HPC system inventory. For example, the inventory interface 118 requests updated inventory information from the HPC system, which the inventory interface 118 uses to update the system states (e.g., current system configurations, such as compute and system nodes on the HPC system, number of compute / system nodes, node types, number of each type, etc.). The updated system inventory is sent back to the controller 115 as message 706. One or more of messages 702, 704 and 706 may be sent as part of operation 504 by Fig.5. In another example, one or more of messages 702, 704, and 706 may be included as part of the context definition phase 210 of Fig. 2, e.g., as part of operation 212. In some implementations, control unit 115 may store the returned system inventory information in memory 114.

[0102] The control unit 115 then transmits message 708 to the power cap interface 116 to request information about the HPC system's power cap. In an example implementation, message 708 includes a code call (e.g., a GET command) that requests information about the power cap, as described above. In response to message 708, the power cap interface 116 performs process 710 to update the HPC system's power cap status. For example, the power cap interface 116 requests updated power caps from the HPC system, which the power cap interface 116 uses to update the power cap status of each node type in the HPC system (e.g., maximum / minimum power cap ranges for controllable nodes, power cap values ​​for non-controllable nodes, etc.).The updated system power limit information is returned to controller 115 as message 712. One or more of message 708, process 710, and message 712 may be executed as part of operation 506 of . Fig. 5. In another example, one or more of messages 708, process 710, and message 712 may be included as part of the context definition phase 210 of Fig. 2, for example, as part of operation 212. In some implementations, the controller 115 may store the returned system power cap information in memory 114.

[0103] Controller 115 may then transmit message 714 to memory 114 to request system states. Message 714 may include a code call (e.g., a GET command) requesting a system-wide configuration state of the HPC system, to which memory 114 may respond with message 716 containing the current system-wide configuration state.

[0104] In an example implementation, messages 714 and / or 716 may be examples of operations performed during the context definition phase 210. For example, the control unit 115 may query the current configuration of the system, e.g., the number of compute and / or system nodes, the ranges and / or values ​​of the power limits, the current power consumption of each node, etc.

[0105] In another example, messages 714 and / or 716 may be examples of operations 508 and 510 of Fig.5. Thus, the control unit 115 can query the pool and node utilization, as well as the number of compute and / or system nodes, the ranges and / or values ​​of the power limits, the current power consumption of each node, etc. Furthermore, the system-wide configuration received in the messages 716 can also contain information delimiting the pools created during the above message flow 600, which group one or more nodes 122 and / or 124 into respective pools.

[0106] Once the current system configuration states are obtained, the control unit 115 calculates the system power caps in process 718. In one example, process 718 may be an example of the calculation phase 220, during which a solution for assigning power caps may be calculated as described above. In another example, process 718 may be an example of operations 512-522 of Fig.5, during which a pool-based power cap allocation solution can be calculated as described above.

[0107] In any case, after calculating an optimal solution for assigning power caps, the control unit 115 transmits message 720 to the power cap interface 116. The message 720 contains instructions for setting power caps for each controllable compute node, which are forwarded by the power cap interface 116 to the HPC system. The message 720 may be an example of the process 232 of Fig. 2 and / or process 526 of Fig.5. The power cap interface 116 responds with message 722 acknowledging receipt of message 720. In one example, after setting power caps to the controllable compute node, the power interface 116 transmits an acknowledgment along with updated system configuration states, which may be included in the acknowledgment or sent as a second message 722. Upon receiving the acknowledgment from the power interface 116, the controller 115 transmits message 724 to the memory 114 to save the updated system configuration states, and the memory 114 returns an acknowledgment message 726 that the updated system configuration states are saved. The message flow 600 then waits for the next trigger event to be detected. The message 724 may be an example of the process 528 of Fig. 5.

[0108] Fig.Figure 8 shows an example of a computing component that can be used to distribute a system performance cap in accordance with various implementations. As shown in Fig. 8, the computing component 800 may be, for example, a server computer, a controller, or other similar computing component that can process data. In the example implementation of Fig. 8, the computer component 800 includes a hardware processor 802 and a machine-readable storage medium 804.

[0109] The hardware processor 802 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices capable of retrieving and executing instructions stored in the machine-readable storage medium 804. The hardware processor 802 may fetch, decode, and execute instructions, such as instructions 806 through 812, to control processes or operations for allocating a system performance cap among controllable compute nodes. Alternatively, or in addition to fetching and executing instructions, the hardware processor 802 may include one or more electronic circuits comprising electronic components for performing the functionality of one or more instructions, such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other electronic circuitry.

[0110] A machine-readable storage medium, such as machine-readable storage medium 804, may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. For example, machine-readable storage medium 804 may be a RAM (Random Access Memory), an NVRAM (Non-Volatile RAM), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a storage device, an optical disk, or the like. In some implementations, machine-readable storage medium 804 may be a non-transitory storage medium, where the term "non-transitory" does not include transitory transfer signals. As described in detail below, machine-readable storage medium 804 may be encoded with executable instructions, for example, instructions 806-812.

[0111] The hardware processor 802 may execute instruction 806 to obtain power limit ranges for a plurality of controllable power consumers and power limit values ​​for a plurality of non-controllable power consumers. The plurality of controllable power consumers may comprise a number of different types, each having different power limit ranges, and the plurality of controllable power consumers and the plurality of non-controllable system equipment are grouped into a plurality of pools. As described above in connection with Fig.2, for example, power cap ranges for controllable power consumers of a system can be determined based on minimum and maximum power cap values ​​for each compute node. As described above, energy consumers can refer to any system equipment, such as subsystems, compute nodes, system nodes, compute units, and / or any component or computing device / resource of an HPC system. As described in connection with the Fig. As described in paragraphs 5 to 7, the controllable and / or non-controllable energy consumers may be grouped into a number of pools, each of which may be associated with a pool power cap that may be set in advance (e.g. by an end-user).

[0112] The hardware processor 802 may execute instruction 808 to calculate a system power cap range for the system based on the power cap ranges and power cap values. For example, as described above in connection with Fig. 2, a system performance range can be determined from the performance limit ranges and performance limit values, and an effective adjustable range of performance limits can be determined from the performance limit ranges, such as a Sum_Min and Sum_Max to define the effective adjustable range. In addition, as in conjunction with Fig. 5, the minimum pool power cap (e.g., Pool_Sum_Min) and / or the maximum pool power cap (e.g., Pool_Sum_Max) are tabulated for each pool, and the difference between them results in a pool delta.

[0113] Hardware processor 802 may execute instruction 810 to determine, for each pool of a plurality of pools, the power caps for the plurality of controllable power consumers of a respective pool from a comparison of the requested power cap to the system power cap range. For example, based on an input of a requested power cap, power caps for the plurality of controllable power consumers of a particular pool may be determined from a comparison of the requested power cap to the system power cap range. In various examples, the power caps for different types of power consumers are based on different power cap ranges.

[0114] As mentioned above in connection with Fig.5, an input may be received indicating a desired power cap for the system. In response to the input, a power cap may be set for the controllable power consumers by comparing the requested power cap with the effective adjustable range of power caps, as described above. Once the power cap is set for the controllable power consumers of the system, each pool may be considered on a pool-by-pool basis in order of a defined priority level to distribute an available power cap to controllable power consumers of the pool, based in part on a comparison of a pool delta with the available power cap and the application of a distribution scheme, as described above in connection with Fig.5. In some examples, a plurality of distribution schemes may be executed for each pool, and based on distribution policies, an optimal distribution of power caps may be determined for allocation to the controllable power consumers of each pool. For example, a solution utilization may be determined for each distribution scheme and compared with that of other distribution schemes to determine an optimal distribution of power caps among the controllable power consumers of a given pool.

[0115] Hardware processor 802 may execute instruction 812 to provide the determined power caps to the system, so that the determined power caps are applied to each of the multiple controllable power consumers. Thus, each controllable power consumer may be controlled to set a power cap according to the determined power cap distribution. This allows the system to operate such that system-wide power consumption does not exceed the required power cap.

[0116] Fig.9 shows a block diagram of an example computer system 900 in which various implementations described herein may be implemented. The computer system 900 includes a bus 902 or other communication mechanism for conveying information, one or more hardware processors 904 connected to the bus 902 for processing information. The computer system 900 may be an example implementation of components disclosed herein, such as the power cap distribution system 110, the controller 115, one or more of the compute nodes 122, and / or the front-end system 130 of Fig. 1. The hardware processor(s) 904 may be, for example, one or more general-purpose microprocessors.

[0117] Computer system 900 also includes main memory 906, such as random access memory (RAM), a cache, and / or other dynamic storage devices connected to bus 902, for storing information and instructions to be executed by processor 904. Main memory 906 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 904. For example, main memory 906 may store process 200 as instructions that can be executed by processor 904 to perform its operations. Likewise, main memory 904 may store instructions 806-812 that can be executed by processor 904.When such instructions are stored in storage media accessible by processor 904, computer system 900 becomes a special-purpose machine adapted to perform the operations specified in the instructions.

[0118] Computer system 900 also includes a read-only memory (ROM) 908 or other static storage device connected to bus 902 for storing static information and instructions for processor 904. A storage device 910, such as a magnetic disk, an optical disk, or a USB flash drive, etc., is provided and connected to bus 902 for storing information and instructions.

[0119] Computer system 900 may be connected to a display 912, such as a liquid crystal display (LCD) (or a touch screen), via bus 902 to display information to a computer user. An input device 914, including alphanumeric and other keys, is coupled to bus 902 to communicate information and command selections to processor 904. Another type of user input device is cursor control 916, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to processor 904 and controlling cursor movement on display 912. In some implementations, the same direction information and command selections as with cursor control may be implemented by receiving touches on a touchscreen without a cursor.

[0120] Computer system 900 may include a user interface module for implementing a graphical user interface, which may be stored on a mass storage device as executable software code executed by the computing device(s). This and other modules may include, for example, components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0121] In general, the terms “component,” “engine,” “system,” “database,” “data store,” and the like, as used herein, may refer to logic embodied in hardware or firmware, or to a collection of software instructions that may have entry and exit points and be written in a programming language such as Java, C, or C++. A software component may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It is understood that software components may be called by other components or by themselves, and / or may be called in response to detected events or interrupts. Software components configured to run on computing devices may be embodied on a computer-readable medium, such as a hard disk.a compact disc, digital video disc, flash drive, magnetic disk, or other tangible medium, or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code may be stored partially or entirely in a memory of the executing computing device for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. In addition, the hardware components may consist of interconnected logic units, such as gates and flip-flops, and / or programmable units, such as programmable gate arrays or processors.

[0122] Computer system 900 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes or programs computer system 900 into a special-purpose machine. According to one implementation, the techniques described herein are performed by computer system 900 in response to processor(s) 904 executing one or more sequences of one or more instructions contained in main memory 906. Such instructions may be read into main memory 906 from another storage medium, such as storage device 910. Execution of the instruction sequences contained in main memory 906 causes processor(s) 904 to perform the process steps described herein.In alternative implementations, hard-wired circuits may be used instead of or in combination with software instructions.

[0123] The term "non-volatile media" and similar terms as used herein refer to any media that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-volatile media may include non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic hard disks, such as storage device 910. Volatile media includes dynamic memory, such as main memory 906. Common forms of non-volatile media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or other magnetic data storage media, CD-ROMs, other optical data storage media, physical media with hole patterns, RAM, PROM and EPROM, FLASH EPROM, NVRAM, other memory chips or cartridges, and networked versions thereof.

[0124] Non-transitory media are distinct from transmission media but can be used in conjunction with them. Transmission media are involved in the transfer of information between non-transitory media. Examples of transmission media include coaxial cables, copper cables, and fiber optic cables, including the wires that make up bus 902. Transmission media can also take the form of sound or light waves, such as those generated in data communications via radio and infrared.

[0125] Computer system 900 also includes a communications interface 918 connected to bus 902. Network interface 918 establishes a two-way data communications connection to one or more network connections connected to one or more local area networks. For example, communications interface 918 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for establishing a data communications connection to a corresponding type of telephone line. As another example, network interface 918 may be a Local Area Network (LAN) card for establishing a data communications connection to a compatible LAN (or a WAN component for communicating with a WAN). Wireless connections may also be implemented.In each of these implementations, the network interface 918 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0126] A network connection typically enables data communication across one or more networks to other data devices. For example, a network connection may establish a connection across a local area network to a host computer or data devices operated by an Internet service provider (ISP). The ISP, in turn, provides data communication services across the worldwide packet data communications network, now commonly referred to as the "Internet." Both the local area network and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. The signals across the various networks and the signals on the network connection and across the communications interface 918 that carry the digital data to and from the computer system 900 are examples of transmission media.

[0127] Computer system 900 can send messages and receive data, including program code, over the network(s), the network connection, and the communications interface 918. In the Internet example, a server could transmit requested code for an application program over the Internet, the ISP, the local network, and the communications interface 918.

[0128] The received code may be executed by processor 904 upon receipt and / or stored in storage device 910 or other non-volatile memory for later execution.

[0129] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code components executed by one or more computer systems or computer processors comprising computer hardware. The one or more computer systems or computer processors may also operate to support the performance of the corresponding operations in a cloud computing environment or as software as a service (SaaS). The processes and algorithms may be partially or fully implemented in application-specific circuitry. The various features and methods described above may be used independently or combined in various ways.Various combinations and subcombinations are intended to be within the scope of this disclosure, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the associated blocks or states may be executed in other suitable orders, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed example implementations. The execution of certain operations or processes may be distributed among computer systems or computer processors located not only in a single machine, but distributed across a number of machines.

[0130] A circuit may be implemented in any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms may be implemented to form a circuit. In implementation, the various circuits described herein may be implemented as discrete circuits, or the described functions and features may be distributed, in part or in whole, among one or more circuits. Even though various features or functional elements are individually described or claimed as separate circuits, those features and functions may be shared by one or more common circuits, and such description is not intended to assume or imply that separate circuits are required to implement those features or functions.If a circuit is implemented in whole or in part with software, that software may be implemented to operate with a computer or processing system capable of performing the functionality described with respect to it, such as the computer system 900.

[0131] As used herein, the term "or" can be interpreted both inclusively and exclusively. Furthermore, descriptions of resources, operations, or structures in the singular should not be construed as excluding the plural. Conditional expressions such as "may," "could," "might," or "may" are generally intended to convey that certain implementations include certain features, elements, and / or steps, while other implementations do not, unless explicitly stated otherwise or understood otherwise by context.

[0132] Unless expressly stated otherwise, the terms and expressions used in this document, as well as their variations, are not to be interpreted as limiting but as open-ended. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar import are not to be construed as limiting the subject matter described to a particular period of time or to a subject matter available at a particular time, but should be understood to include conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future.The presence of broader words and phrases such as “one or more,” “at least,” “but not limited to,” or similar phrases in some cases should not be construed as meaning that the narrower case is intended or required in the absence of such broader phrases. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 17337107

[0039]

Claims

[1] A method for applying a system performance limit to a heterogeneous system, the method comprising: Obtaining power limit ranges for a plurality of controllable system equipment and power limit values for a plurality of non-controllable system equipment, wherein the plurality of controllable system equipment comprises a number of different equipment types, each having different power limit ranges, wherein the plurality of controllable system equipment and the plurality of non-controllable system equipment are grouped into a plurality of pools; Calculating a system power limit range for the heterogeneous system from the power limit ranges and power limit values; for each pool of the plurality of pools, determining power caps for the plurality of controllable system equipment of a respective pool from a comparison of a requested power cap with the system power cap range, wherein the power caps for the different equipment types of the respective pool are based on the different power cap ranges; and Providing the determined power limits to the heterogeneous system, wherein the determined power limits are applied to each of the plurality of controllable system devices. [2] The method of claim 1, wherein each of the plurality of pools is assigned a priority level, and wherein the power limits for the plurality of controllable system devices of the plurality are determined for each respective pool in an order based on the priority level assigned to each respective pool. [3] The method of claim 1, wherein the performance limits for a first pool assigned a first priority level are determined before the performance limits for a second pool assigned a second priority level lower than the first priority level. [4] The method according to claim 1 further comprises: for each pool from the multitude of pools, Determining a pool delta power cap based on the power cap ranges of the controllable system equipment grouped in the respective pool; and Determining that the pool delta power cap is greater than an available power budget, where the available power budget is based on the requested power cap, wherein the determination of the power caps for the plurality of controllable system equipments of the respective pool is based on the determination that the delta power cap of the pool is greater than an available power budget. [5] The method of claim 4, wherein the power caps for the plurality of controllable system devices of a respective pool are determined based on a distribution scheme configured to assign a power cap to each of the plurality of controllable system devices. [6] The method according to claim 5 further comprises: Determining a variety of benefit cap allocations using a variety of distribution schemes; Selecting an optimal performance cap allocation from the multitude of performance cap allocations; and Determine the power caps for the multiple controllable system devices of the respective pool based on the selected optimal power cap allocation. [7] The method according to claim 4 further comprises: in response to determining that the pool's delta power cap is less than or equal to the available power budget, determining the power caps for the multiple controllable system devices of the respective pool as the maximum power cap value. [8] The method according to claim 1 further comprises: Detection of a trigger event on the heterogeneous system, wherein obtaining power limit ranges for a plurality of controllable system devices and power limit values for a plurality of non-controllable system devices is responsive to the detected trigger event. [9] The method of claim 8, wherein the trigger event comprises one or more of the following events: job start, receipt of a requested power cap, receipt of a requested power budget, and a periodic timer. [10] The method of claim 1, wherein the power limit ranges are based on the hardware architectures of the plurality of controllable system devices and wherein the power limit values are based on the hardware architectures of the plurality of non-controllable system devices. [11] A power cap distribution system for applying a system power cap to a heterogeneous system, the power cap distribution system comprising: a memory configured to store instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to: Obtaining power limit ranges for a plurality of controllable system equipment and power limit values for a plurality of non-controllable system equipment, wherein the plurality of controllable system equipment comprises a number of different equipment types, each having different power limit ranges, wherein the plurality of controllable system equipment and the plurality of non-controllable system equipment are grouped into a plurality of pools; Calculating a system power limit range for the heterogeneous system from the power limit ranges and power limit values; for each pool of the plurality of pools, determining power caps for the plurality of controllable system equipment of a respective pool from a comparison of a requested power cap with the system power cap range, wherein the power caps for the different equipment types of the respective pool are based on the different power cap ranges; and Providing the determined power limits to the heterogeneous system, wherein the determined power limits are applied to each of the plurality of controllable system devices. [12] The system of claim 11, wherein each of the plurality of pools is assigned a priority level, and wherein the power limits for the plurality of controllable system devices of the plurality are determined for each respective pool in an order based on the priority level assigned to each respective pool. [13] The system of claim 11, wherein the performance limits for a first pool assigned a first priority level are determined before the performance limits for a second pool assigned a second priority level lower than the first priority level. [14] The system of claim 11, wherein the one or more processors are further configured to execute the instructions to: for each pool from the multitude of pools, Determining a pool delta power cap based on the power cap ranges of the controllable system equipment grouped in the respective pool; and determine that the pool delta power cap is greater than an available power budget, where the available power budget is based on the requested power cap, wherein the determination of the power caps for the plurality of controllable system equipments of the respective pool is based on the determination that the delta power cap of the pool is greater than an available power budget. [15] The system of claim 14, wherein the power caps for the plurality of controllable system devices of a respective pool are determined based on a distribution scheme configured to assign a power cap to each of the plurality of controllable system devices. [16] The system of claim 15, wherein the one or more processors are further configured to execute the instructions to: Determining a variety of benefit cap allocations using a variety of distribution schemes; Selecting an optimal performance cap allocation from the multitude of performance cap allocations; and Determine the power caps for the multiple controllable system devices of the respective pool based on the selected optimal power cap allocation. [17] The system of claim 14, wherein the one or more processors are further configured to execute the instructions to: in response to determining that the pool's delta power cap is less than or equal to the available power budget, determine the power caps for the multiple controllable system devices of the respective pool as the maximum power cap value. [18] The system of claim 11, wherein the one or more processors are further configured to execute the instructions to: detect a trigger event on the heterogeneous system, wherein obtaining power limit ranges for a plurality of controllable system devices and power limit values for a plurality of non-controllable system devices is responsive to the detected trigger event. [19] The system of claim 18, wherein the trigger event comprises one or more of the following events: job start, receipt of a requested power cap, receipt of a requested power budget, and a periodic timer. [20] A non-transitory, computer-readable storage medium for distributing energy caps, configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising: Calculating a power limit range for a plurality of controllable system devices from a system power range and a plurality of power limit ranges of the plurality of controllable system devices; Calculating a power cap allocation solution based on a power cap set for the system and the calculated power cap range, the power cap allocation solution distributing the power cap set for the system among the plurality of controllable system devices; Applying the solution for allocating the power cap to the system on the basis of each controllable system piece of equipment; and Automatic repetition of calculating, calculating and applying depending on a periodic timer.

Images