Systems and methods for a mesh interactive data center

Through the grid-aware power management system, the power supply of data centers is adjusted using control services and machine learning models, solving the problem of low power management efficiency in data centers and achieving efficient and low-carbon power use.

CN119053931APending Publication Date: 2024-11-29MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202380034588.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-10
Filing Date
2023-03-28
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Data centers consume a lot of electricity, and the existing technology is difficult to effectively manage power supply, resulting in increased operating costs and carbon emissions.

Method used

Using a grid-aware power management system, leverages control services and energy/workload controllers to adjust the power supply of data centers by obtaining the workload status, infrastructure parameters and public telemetry of the server rack, utilizing control services and energy/workload controllers, combining machine learning model prediction and Adjust power requirements to reduce dependence on common grids.

Benefits of technology

It realizes efficient power management in data centers, reduces operating costs and carbon emissions, and improves the availability and response speed of computing services, and reduces dependence on fossil fuel electricity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119053931A_ABST
    Figure CN119053931A_ABST
Patent Text Reader

Abstract

A power management method in a data center includes obtaining at least one workload condition of at least one server rack, obtaining at least one infrastructure parameter, obtaining at least one common telemetry, and comparing the at least one workload condition and the at least one common telemetry. The method further includes determining a workload demand based at least in part on a difference between the at least one workload condition and the at least one common telemetry, and changing the at least one infrastructure parameter based on the workload demand and the at least one infrastructure parameter.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Background and Related Technology

[0002] Data centers consume large amounts of electricity. Changes in the properties and / or power supply of the electricity provided by the grid can adversely affect the data center through increased operating costs, the ability to power computing resources, or increased carbon intensity. Summary of the invention

[0003] In some embodiments, a method for power management in a data center includes obtaining at least one workload state of at least one server rack, obtaining at least one infrastructure parameter, obtaining at least one utility telemetry, and comparing the at least one workload state and the at least one utility telemetry. The method also includes determining a workload demand based at least in part on a difference between the at least one workload state and the at least one utility telemetry, and changing at least one infrastructure parameter based on the workload demand and the at least one infrastructure parameter.

[0004] In some embodiments, a system for controlling power supply in a data center includes a control service, an energy controller in data communication with the control service, and a workload controller in data communication with the control service. The control service is configured to obtain at least one workload state of at least one server rack, obtain at least one infrastructure parameter, obtain at least one utility telemetry, and compare at least one workload state and at least one utility telemetry. The control service is also configured to determine a workload demand based at least in part on a difference between at least one workload state and at least one utility telemetry, and to change at least one infrastructure parameter based on the workload demand and at least one infrastructure parameter without outputting power to a utility grid.

[0005] In some embodiments, a method for power management in a data center includes obtaining at least one workload state of at least one server rack, obtaining at least one infrastructure parameter, obtaining at least one utility telemetry, and inputting the at least one utility telemetry, at least one workload state, and at least one infrastructure parameter into an ML model. The method also includes changing at least one infrastructure parameter based on at least one utility telemetry, at least one workload state, and at least one infrastructure parameter, and changing at least one workload state based on at least one utility telemetry, at least one workload state, and at least one infrastructure parameter.

[0006] This Summary provides an introduction to a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or underlying features of the claimed subject matter.

[0007] Additional features and advantages will be set forth in the following description, and in part will be apparent from the description, or may be learned through practice of the teachings herein. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. The features of the present disclosure will become more apparent from the following description and the appended claims, or may be learned through practice of the present disclosure set forth below. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to describe the manner in which the above-described and other features of the present disclosure can be obtained, a more specific description will be provided with reference to specific embodiments shown in the accompanying drawings. For better understanding, the same elements are designated with the same reference numerals in the various drawings. Although some of the drawings may be schematic or exaggerated representations of concepts, at least some of the drawings may be drawn to scale. It is understood that the drawings depict some example embodiments, which will be described and explained with additional uniqueness and detail through the use of the drawings, in which:

[0009] Figure 1 is a schematic representation of a data center according to at least some embodiments of the present disclosure;

[0010] Figure 2 is a flow chart illustrating a method of power management in a data center according to at least some embodiments of the present disclosure;

[0011] Figure 3 is a schematic representation of a machine learning model according to at least some embodiments of the present disclosure;

[0012] Figure 4 is a flowchart illustrating a method of training an ML model according to at least some embodiments of the present disclosure; and

[0013] Figure 5 Another method of power management in a data center is illustrated in accordance with at least some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] The present disclosure generally relates to systems and methods for power management in a data center. More specifically, the systems and methods described herein are grid-aware and can adjust power supply and consumption in a data center at least in part in response to receiving telemetry of a common grid that provides power to the data center and / or co-location(s) therein.

[0015] In some embodiments, systems and methods according to the present disclosure allow a data center or a colocation within a data center to provide computing services more efficiently and / or faster while reducing the operating costs and / or carbon impact of the data center operation. In some embodiments, a control service or control plane of the data center communicates with a substation that provides power to the colocation of server computers in the data center, and communicates with one or more controllers of the colocation to allow the control service to change the process allocation and power supply to the colocation based on the utility availability at the substation. In at least one embodiment, the control service can change the virtual machine (VM) allocation within the colocation and change or adjust at least one power source of the colocation in response to telemetry received from the utility substation.

[0016] In some embodiments, one or more host-hosted data centers including server computers include infrastructure resources configured to provide high availability (e.g., through power supply devices such as uninterruptible power supplies (UPS)) and software controllers that enable efficient use of the data center. For example, the software controller can efficiently use computing and / or information technology (IT) resources in the data center through one or more of power capping, workload reduction, and active migration (e.g., before a planned maintenance event). In some embodiments, redundant computing and / or IT resources are used only in planned maintenance or power outage scenarios, and the redundant resources may not be used during normal data center operations. Therefore, redundant computing and / or IT resources can provide opportunities for various grid interactive services, such as frequency regulation, frequency suppression, and demand response.

[0017] Such example scenarios may require different reaction times and durations, and are limited by battery capacity and the need to maintain sufficient backup energy to support any data center failure events. In some embodiments, systems and methods of power management according to the present disclosure utilize a combination of energy storage for rapid reaction in short periods of time and workload management for long-term regulation. On-site energy storage and / or hybrid approaches to power generation combined with workload management (e.g., power capping, workload shifting, power-aware scheduling) can further reduce reliance on fossil fuel-based electricity.

[0018] Figure 1is a system diagram illustrating an embodiment of a system 100 for power management. In some embodiments, a data center 102 site is comprised of one or more co-located data centers ("hosting sites" 104-1) that receive power from the same high voltage utility substation 106. In some embodiments, utility high voltage lines from a utility grid are injected into the substation 106, which in turn feeds multiple spaces (hosting sites 104-1) in one or more data centers 102 through a set of medium voltage transformers. In at least one embodiment, an external utility grid (e.g., an electric utility) supplies power to multiple hosting sites 104-1, and each hosting site may have its own transformer, UPS backup battery, generator, (multiple) fuel cells, and combinations thereof. One or more hosting sites 104-1 may participate in grid services, and some embodiments of the control systems and methods described herein may coordinate available energy storage and workload characteristics across these hosting sites 104-1.

[0019] In some embodiments, the system 100 for power management in a data center 102 includes at least a control service 108 that obtains or accesses a plurality of properties and / or telemetry of a utility (e.g., at a substation 106) and the data center 102 to provide instructions to one or more components of the data center 102. The instructions provided by the control service 108 allow the data center 102 to more efficiently make computing services available to users of the data center 102. The control service 108 may be located remotely from the data center and / or colocation (multiple) and obtain information about and communicate with components of the data center 102 over a network connection. In some embodiments, it may be beneficial for the control service 108 to have a response time to changing conditions of less than 5 milliseconds (ms), less than 2 ms, or less than 1 ms, and it may be beneficial for the control service 108 to be located on-site at the data center 102 to facilitate faster communication times. In some embodiments, the control service 108 is a service running on a control computing device in the data center that communicates with other components of the data center 102. In some embodiments, the control service 108 includes a dedicated processor, hardware storage, and / or computing device that executes the control service 108 .

[0020] In some embodiments, the control service 108 communicates data with an energy controller 110 of the host 104-1. For example, each host within a data center can have an energy controller that controls, allocates, manages the power supply infrastructure of the host, and combinations thereof. In some examples, the energy controller 110 is at least partially responsible for charging and / or discharging the long-term energy storage 112 for the host. In some examples, the energy controller is at least partially responsible for other hardware power supply and / or power storage operations. It should be understood that although Figure 1 Batteries are described, but other long term energy stores 112 or supplies may be used, such as hydrogen fuel cells, gravity storage, or other long term stable energy sources.

[0021] For example, the energy controller 110 can communicate data with one or more UPS 114 of the host hosting 104-1. In some embodiments, the host hosting has at least one UPS 114 for each server rack 116 of the host hosting. In some embodiments, the host hosting has a UPS 114 for each server rack 116 of the host hosting 104-1. In some embodiments, the host hosting has at least one UPS 114 that is configured to provide power to the server racks 116 of the host hosting 104-1. In some embodiments, at least one UPS 114 is configured to provide power to multiple server racks 116 of the host hosting 104-1. The energy controller can communicate with the UPS 114 to provide power or additional power to one or more server computers or other IT components in response to changes in the utility power supply of the host hosting 104-1 and / or the data center 102.

[0022] In another example, the energy controller 110 can communicate data with one or more generators 115 of the host hosting. In some embodiments, the host hosting has at least one generator 115 for each server rack 116 of the host hosting 104-1. In some embodiments, the host hosting has a generator 115 for each server rack 116 of the host hosting 104-1. In some embodiments, the host hosting 104-1 has at least one generator 115 that is configured to provide power to the server racks 116 of the host hosting 104-1. In some embodiments, the at least one generator 115 is configured to provide power to multiple server racks 116 of the host hosting 104-1. The energy controller 110 can communicate with the generator 115 to provide power or additional power to one or more server computers or other IT components in response to a change in the utility power supply of the host hosting 104-1 and / or the data center 102.

[0023] In another example, the energy controller 110 can communicate data with one or more long-term energy storages 112 (e.g., long-term batteries) of the hosting 104-1. In some embodiments, the hosting 104-1 has at least one long-term energy storage 112 for each server rack 116 of the hosting 104-1. In some embodiments, the hosting 104-1 has a long-term energy storage 112 for each server rack 116 of the hosting 104-1. In some embodiments, the hosting 104-1 has at least one long-term energy storage 112 that is configured to provide power to the server racks 116 of the hosting 104-1. In some embodiments, the at least one long-term energy storage 112 is configured to provide power to multiple server racks 116 of the hosting 104-1. The energy controller can communicate with the long-term energy storage 112 to provide power or additional power to one or more server computers or other IT components in response to a change in the utility power supply of the hosting 104-1 and / or the data center 102.

[0024] In some embodiments, the control service 108 communicates data with the workload controller 117. The workload controller 117 is responsible for performing workload operations and / or controls, such as power capping, server shutdown, VM allocation, process allocation, and workload migration. In some embodiments, the workload controller 117 responds to long-term (several minutes to several hours) grid service requests through a combination of power capping, server shutdown, VM allocation, process allocation, and workload migration. In some embodiments, the workload controller 117 performs one or more of power capping, server shutdown, VM allocation, process allocation, and workload migration based at least in part on hardware capabilities (e.g., whether it can be power capped / throttled), availability requirements (e.g., whether the software is redundant), utilization patterns, and potential impact of one or more of power capping, server shutdown, VM allocation, process allocation, and workload migration on workloads and / or processes.

[0025] In some embodiments, a determination and / or instruction to engage in one or more of power capping, server shutdown, VM allocation, process allocation, and workload migration is made at the control service 108. In some embodiments, the workload controller 117 provides the control service 108 with a list of viable options for workload management (e.g., which of power capping, server shutdown, VM allocation, process allocation, and workload migration is an available option, based at least in part on hardware capabilities, availability requirements, and current workload / process). In some embodiments, the control service 108 determines which option to engage in based at least in part on acquired information (e.g., common telemetry and infrastructure status). In some examples, the acquired information includes the amount of power that needs to be "recovered", latency requirements (e.g., a few seconds for unplanned events; advance notice for planned events), and a dynamic impact function defined by the workload. In some embodiments, parameters are obtained and / or calculated by the control service 108 periodically and / or on demand and provided to the workload controller 117, which determines workload management decisions.

[0026] In some embodiments, workload controller 117 monitors critical events (e.g., grid service requests or data center equipment status) and takes corresponding actions when any critical events are detected. In such embodiments, workload controller 117 is tasked with responding to potential critical events (e.g., insufficient backup battery), which can reduce the response time to specific critical events.

[0027] The control service 108 then uses these inputs and a set of heuristics or machine learning (ML) to decide whether to use hardware-based energy management to compensate for the power needs of the data center and / or host hosting, such as releasing long-term energy storage 112 or starting generators 115 via energy controller 110, and / or using workload controller 117 to reduce the power consumption of the data center and / or host hosting through software-defined mechanisms.

[0028] In some embodiments, the control service 108 also communicates with a second hosting 104-2 (or more) including a second energy controller and a second workload controller to manage the infrastructure power source and workload of the second hosting, respectively. The control service can coordinate the workload controller to migrate workloads or processes between hostings 104-1, 104-2, or coordinate the energy controller to allocate power from the infrastructure power source between hostings 104-1, 104-2.

[0029] Figure 22 is a flow chart illustrating an embodiment of a method 218 for power management. In some embodiments, the method includes obtaining at least one workload state of at least one server rack at 220, obtaining at least one infrastructure parameter at 222, and obtaining at least one common telemetry at 224 at a control service. In some embodiments, the control service obtains at least a portion of the workload state from a workload controller. In some embodiments, the control service obtains at least a portion of the workload state from one or more server computers. In some embodiments, the controller service obtains at least a portion of the workload state from a rack manager of a server rack hosted by a host. For example, a rack manager can communicate with one or more server computers in a server rack, and the rack manager can monitor the power consumption of (multiple) server computers. Power consumption is the amount of electrical power (from all power sources inside or outside the data center) required by (multiple) server computers to perform the current or requested workload. For example, the power consumption of a single server computer can be monitored, the power consumption of a rack of server computers can be monitored, or the power consumption of a host hosting of server computers can be monitored.

[0030] In some embodiments, the workload state includes one or more of VM allocation, process allocation, process priority list, process migration state, utilization mode, workload performance and availability requirements, failover capability, or other information related to the computing operation of the server computer hosted by the host. For example, the VM allocation information can inform the workload controller and / or control service of the number of VMs allocated to a specific server computer, server rack, or host. The number of VMs can inform the workload controller and / or control service of the computing capacity available on the allocated server and / or the maximum power consumption that the allocated server may require. The workload controller and / or control service can use the VM allocation information to help estimate the computing and power requirements of the host hosting and / or data center.

[0031] In some examples, process allocation information can notify a workload controller and / or control service of a particular process requested or currently being executed on at least some server computers in a server rack, a hosting, and / or a data center. In some embodiments, the workload controller and / or control service includes a process list to monitor the processes and their power consumption and computing requirements. For example, the workload controller and / or control service can determine that a first process assigned to a first server computer within a hosting has a first power consumption associated with the first process, and the workload controller and / or control service can determine that a second process assigned to a second server computer within the hosting has a second power consumption associated with the second process. The workload controller and / or control service can determine the current total power consumption or expected power consumption of the assigned processes based at least in part on the process list.

[0032] In some embodiments, the process assignment and / or process list has a process priority list that informs the workload controller and / or control service of the relative importance of processes currently executing or queued on the host. For example, a first process assigned to a first server computer may have a higher priority than a second process assigned to a second server computer, and the workload controller may power cap or throttle the second server computer to prioritize the performance of the first process of the first server computer.

[0033] In some examples, the process migration status can notify the workload controller and / or control service of the availability of computing resources for process migration between server computers and / or between hosting. In some examples, the process migration status can notify the workload controller and / or control service of the initiation and / or completion of process migration to allow the workload controller and / or control service to track the availability of computing resources.

[0034] In some embodiments, the utilization pattern can inform the workload controller and / or control service of the current or predicted future state of the workload on the host hosting based at least in part on historical data and resource utilization trends. For example, the utilization pattern can include process allocation, power consumption, and / or computational load, which is based at least in part on the time of day, day of the week, day of the year, or correlation with other events (such as weather, holidays, or periodic events). In some embodiments, the workload controller and / or control service can determine the trend or predicted future state of the workload based on the utilization pattern, and preemptively change or adjust the workload or power supply to at least partially compensate for the trend or predicted future state of the workload.

[0035] In some embodiments, the control service and / or the energy controller obtains infrastructure parameters. For example, the energy controller can obtain or store infrastructure parameters, and the control service can obtain infrastructure parameters from the energy controller. The infrastructure parameters include information related to the performance, history, or requirements of the hardware of the hosting and / or data center. In some embodiments, the infrastructure parameters may include battery / UPS charge status, battery / UPS degradation (e.g., degradation counters), generator capacity, component temperature, server computer power consumption, maintenance scheduling, and other measurements or properties of (multiple) energy sources and (multiple) energy sinks within the hosting and / or data center.

[0036] In some embodiments, the battery charge state or UPS charge state includes a percentage state of charge of a long-term battery and / or UPS, a nominal voltage of a long-term battery and / or UPS, or a nominal charge state of a long-term battery and / or UPS (e.g., measured in kilowatt-hours). The battery charge state or UPS charge state may inform the control service and / or energy controller of the duration that the battery / UPS may provide power or additional power to the server computer in the event of a utility failure or other event.

[0037] In some embodiments, the generator capacity allows the control service and / or energy controller to know how much peak power the generator can provide, how long the generator can provide power, and the total power the generator can provide. In some embodiments, the infrastructure parameters also include a start time for the generator, which can inform the control service and / or energy controller of the delay in starting the generator before the generator can begin providing power to the colocation and / or data center.

[0038] In some embodiments, the battery / UPS degradation includes total battery aging parameters for the battery and / or UPS, the number of charge / discharge cycles of the battery and / or UPS, or the available charge depth. For example, based on the depth of charge / discharge, a battery may have a limited number of charge cycles. A cost is associated with each discharge and charge cycle of the battery. In some embodiments, the age of the battery affects the capacity of the battery, which limits the amount of power that an older battery can provide to the host hosting relative to a newer battery. In some embodiments, the energy controller monitors the age of the battery (in terms of time, cycles, or capacity) and provides the age to the control service. In some embodiments, the control service monitors the age of the battery based at least in part on information provided by the energy controller.

[0039] In some embodiments, the energy controller and / or control service monitors component temperature because elevated temperatures may affect the efficiency and / or operating life of a battery, UPS, or generator in data communication with the energy controller and / or control service.

[0040] In some embodiments, infrastructure parameters include maintenance scheduling of one or more components of the hosting and / or data center. For example, some components may be unavailable due to planned maintenance. In other examples, planned maintenance may require additional power provided by a fuel cell, battery, other long-term energy storage, UPS, or generator, and the energy controller and / or control service can prepare the fuel cell, battery, other long-term energy storage, UPS, or generator in advance to provide the capacity required for the additional power.

[0041] In some embodiments, the control service obtains utility telemetry from a substation, utility lines, or other communications with a utility grid. Utility telemetry may include the frequency of power provided by a utility to a substation and / or through a substation, the carbon intensity of power provided by a utility to a substation and / or through a substation, and the power demand and response of the utility grid. The frequency of the power provided by the utility may vary with grid stability, supply, and demand. In a utility grid, operators attempt to maintain grid balance and reliability and keep their frequency within defined limits. Deviations from the nominal frequency, i.e., 50 Hz or 60 Hz, are caused by a mismatch between supply and demand (a phenomenon exacerbated by greater penetration of variable renewable energy sources, which is also associated with the carbon intensity of the power provided). In some embodiments, a control service according to the present disclosure may help operators regulate grid frequency by reducing or increasing power consumption of colocation and / or data centers. Frequency regulation may require fast-response energy storage, generators, or fast workload management.

[0042] Additionally, some embodiments of the control services according to the present disclosure may provide or assist in providing frequency suppression. In some examples, frequency suppression reserves (e.g., provided by long-term energy storage, a UPS, or a generator in communication with an energy controller) may provide a primary response to sudden frequency changes (typically low frequencies) caused by unexpected events in the utility grid or sudden drops in renewable energy. When system inertia is reduced in areas with high penetration of renewable energy, even faster-reacting reserves (e.g., batteries relative to generators) may be used to at least partially compensate for low inertia conditions. In some embodiments, fast-responding battery energy storage in a data center with limited energy storage duration may provide frequency suppression in response to acquired utility telemetry.

[0043] In some embodiments, the utility telemetry obtained includes carbon intensity related to the delivery of power and source provided by the utility grid. In some embodiments, the control service selectively charges or discharges batteries, or starts generators, based on the reduced carbon impact of the colocation and / or data center.

[0044] In some embodiments, Figure 2 The method also includes comparing at least one workload state and at least one utility telemetry at 226, and determining a workload demand based at least in part on a difference between the at least one workload state and the at least one utility telemetry at 228. Comparing at least one workload state and at least one utility telemetry allows the control service to determine the difference between the power provided by the utility grid and the power required for the current or predicted future state of the workload. For example, the workload state and utility telemetry may indicate a positive workload demand, which requires deployment of one or more infrastructure power sources, such as long-term batteries, UPS, or generators, to at least partially compensate for the positive workload demand. In some embodiments, the workload state and utility telemetry may indicate a negative workload demand, which provides an opportunity to charge a long-term battery or overclock process resources in a hosting and / or data center.

[0045] In some embodiments, the method further includes, at 230, changing at least one infrastructure parameter based at least in part on the workload demand and at least one infrastructure parameter. For example, changing at least one infrastructure parameter may include discharging a long-term battery. In some examples, changing at least one infrastructure parameter may include charging a long-term battery. In some examples, changing at least one infrastructure parameter may include discharging a UPS. In some examples, changing at least one infrastructure parameter may include starting a generator. In some embodiments, changing at least one infrastructure parameter based on the workload demand supplements the power received from the utility grid. In some embodiments, changing at least one infrastructure parameter based on the workload demand supplements the power received from the utility grid without outputting power from the data center (i.e., from the long-term energy storage and / or the generator) to the utility grid.

[0046] Workload state, infrastructure parameters, and common telemetry may be inputs to an ML model to determine outputs provided by a control service to a controller. In other embodiments, workload state, infrastructure parameters, and common telemetry may be inputs to one or more of heuristics, mathematical models, and algorithms to determine outputs provided by a control service to a controller.

[0047] Figure 3It is a schematic diagram of a machine learning (ML) model that can be used with one or more embodiments of the systems and methods described herein. As used herein, a "machine learning model" refers to a computer algorithm or model (e.g., a classification model, a regression model, a language model, an object detection model) that can be adjusted (e.g., trained) to approach an unknown function based on a training input. For example, a machine learning model can refer to a neural network or other machine learning algorithm or architecture that learns and approaches complex functions based on multiple inputs provided to the machine learning model, and generates an output. In some embodiments, the machine learning system, model, or neural network described herein is an artificial neural network. In some embodiments, the machine learning system, model, or neural network described herein is a convolutional neural network. In some embodiments, the machine learning system, model, or neural network described herein is a recurrent neural network. In at least one embodiment, the machine learning system, model, or neural network described herein is a Bayesian classifier. As used herein, a "machine learning system" can refer to one or more machine learning models that collaboratively generate one or more outputs based on corresponding inputs. For example, a machine learning system can refer to any system architecture with multiple discrete machine learning components that consider different types of information or inputs.

[0048] As used herein, an "instance" refers to an input object that can be provided as input to a machine learning system to generate outputs, such as utility telemetry, workload status, and infrastructure parameters. For example, an instance can refer to any event in which utility telemetry changes in a manner that affects the frequency of power provided by the utility grid. For example, low-frequency events can be associated with afternoons or evenings in areas with warmer weather, corresponding to increased demand on the utility grid. In some embodiments, low-frequency events can be at least partially compensated for with power capping or workload migration, while in other instances, low-frequency events can be at least partially compensated for with additional infrastructure power sources when workloads are high or process priority is high.

[0049] In some embodiments, the machine learning system has multiple layers, including an input layer 336 configured to receive at least one input training data set 332 or input training instance 334, and an output layer 340, with multiple additional or hidden layers 338 between the input layer and the output layer. The training data set can be input to the machine learning system to train the machine learning system and identify individual and combinations of labels or attributes of the training instances that allow the hosting and / or data center to participate in the grid service. In some embodiments, the input includes utility telemetry, workload status, infrastructure parameters, or a combination thereof.

[0050] In some embodiments, the machine learning system can concurrently receive multiple training data sets and learn from different training data sets simultaneously. For example, a training data set of common grid common telemetry changes includes different information and / or labels than a training data set including changes in workload states.

[0051] In some embodiments, the machine learning system includes multiple machine learning models running together. Each machine learning model has multiple hidden layers between the input layer and the output layer. The hidden layer has multiple input nodes (e.g., node 342), where each of the nodes operates on the input received from the previous layer. In a specific example, the first hidden layer has multiple nodes, and each of the nodes performs an operation on each instance from the input layer. Each node in the first hidden layer provides new inputs to each node in the second hidden layer, and each node in the second hidden layer in turn performs a new operation on each of these inputs. The nodes of the second hidden layer then pass the output (e.g., identified clusters 344) to the output layer.

[0052] In some embodiments, each node in nodes 342 has a linear function and an activation function. The linear function may attempt to optimize or approximate a solution with a best fit line, such as reduced power cost or reduced carbon intensity. The activation function is used as a test to check the effectiveness of the linear function. In some embodiments, the activation function produces a binary output that determines whether the output of the linear function is passed to the next layer of the machine learning model. In this way, the machine learning system can limit and / or prevent poor fits from propagating to the data and / or non-convergent solutions.

[0053] The machine learning model includes an input layer that receives at least one training data set. In some embodiments, at least one machine learning model uses supervised training. Supervised training allows input of multiple common grid or workload events with known responses from energy controllers and / or workload controllers, and allows the machine learning system of the control service to develop correlations between the inputs and responses to learn risk factors and combinations thereof. In some embodiments, at least one machine learning model uses unsupervised training. Unsupervised training can be used to draw inferences and find patterns or associations from (multiple) training data sets in the absence of known events. In some embodiments, unsupervised learning can identify clusters of similar labels or characteristics for various training instances, and allow the machine learning system to extrapolate safety and / or risk factors for instances with similar characteristics.

[0054] In some embodiments, semi-supervised learning can combine the advantages of supervised learning and unsupervised learning. As described herein, a machine learning system can identify associated labels or features between instances, which can allow for the fusion of a training data set with known events and a second training data set including more general input information. Unsupervised training can allow a machine learning system to cluster instances from a second training data set without known events and associate the clusters with known events from a first training data set.

[0055] Figure 4 is a flow chart illustrating an embodiment of a method for training an ML model 446, for example, Figure 3 In some embodiments, offline training of the ML model 446 may include an offline simulation environment (e.g., simulation environment 448) having a data center simulator 450 that receives input from an IT simulator 452 and a simulated data center power infrastructure 454. The simulation environment 448 outputs a state 458 to a reinforcement learning (RL) agent 456, the output state 458 having a reward 460 (positive or negative reward) associated with the state 458 of the data center simulator 450. The RL agent 456 may create a loop that provides further input to the simulation environment 448 to improve the response and / or output of the simulation environment 448 over time.

[0056] In some embodiments, the RL agent 456 provides information to the online ML model 446, which receives real-time inputs 462 including utility telemetry, workload status, and infrastructure parameters. The RL agent 456 and the simulation environment 448 may allow the ML model 446 to be pre-trained and / or continuously trained offline with additional scenarios, which are then fused with the real-time inputs 462 at the ML model 446. The simulation environment 448 may allow the ML model 446 to be trained more quickly without the data center and / or colocation experiencing adverse utility grid or infrastructure conditions simulated in the simulation environment 448.

[0057] Figure 5is a flow chart illustrating another embodiment of a method 564 for power management. In some embodiments, a method for power management in a data center includes making changes to a host-hosted workload and a host-hosted power infrastructure. A control service may communicate with an energy controller and a workload controller to provide short-term (e.g., fast response) and long-term grid services and workload balancing. For example, a control service may communicate with an infrastructure power source (e.g., a fuel cell, a battery, a UPS, a generator) to provide a fast response to a change in utility telemetry or workload state, and a control service may also communicate with a workload controller to migrate a workload to a second host-hosted and / or to power cap a first host-hosted to balance workload demands between available power and the cost or carbon intensity of the available power.

[0058] In some embodiments, the method includes, at a control service, obtaining at least one workload state of at least one server rack at 520, obtaining at least one infrastructure parameter at 522, and obtaining at least one common telemetry at 524. In some embodiments, the control service obtains at least a portion of the workload state from a workload controller. In some embodiments, the control service obtains at least a portion of the workload state from one or more server computers. In some embodiments, the controller service obtains at least a portion of the workload state from a rack hypervisor of a server rack in colocation.

[0059] In some embodiments, the method further includes inputting at least one common telemetry, at least one infrastructure parameter, and at least one workload state into an ML model, as described herein, at 566 to determine at least one change to each of the at least one infrastructure parameter and the at least one workload state. Figure 4 The content is outputted by a pre-trained ML model, and the method includes changing at least one infrastructure parameter at 568 and changing at least one workload state at 570.

[0060] In some examples, the control service may change at least one infrastructure parameter at 568 to charge the infrastructure power source of the utility grid based at least in part on the utility telemetry obtained at 524. In some embodiments, the control service may change at least one infrastructure parameter at 568 to charge the infrastructure power source of the utility grid based at least in part on the workload state obtained at 520.

[0061] For example, a positive workload demand may result from an infrastructure failure or a partial infrastructure failure. In some embodiments, a control service and / or workload controller may adjust the workload and / or process of a server computer based on a change in an infrastructure parameter. In at least one example, a change in an infrastructure parameter may be related to a limited availability of infrastructure power provided to the server computer. In such an example, a control service and / or workload controller may migrate the workload to another hosting to reduce the workload demand. In some examples, a control service and / or workload controller may cap or throttle the server computer power to reduce the workload demand.

[0062] Industrial Applicability

[0063] In some embodiments, systems and methods according to the present disclosure allow a data center or a host hosting within a data center to provide computing services more efficiently and / or faster while reducing the operating costs and / or carbon impact of the data center operation. In some embodiments, a control service or control plane of the data center communicates with a substation that provides power to the host hosting of server computers in the data center, and one or more controllers of the host hosting, to allow the control service to change the process allocation and power supply of the host hosting based on the utility availability at the substation. In at least one embodiment, the control service can change the virtual machine (VM) allocation within the host hosting and change or adjust at least one power source of the host hosting in response to telemetry received from the utility substation.

[0064] In some embodiments, one or more host-hosted data centers including server computers include infrastructure resources configured to provide high availability (e.g., via power supply devices such as uninterruptible power supplies (UPS)) and software controllers that enable efficient use of the data center. For example, the software controller can efficiently use computing and / or information technology (IT) resources in the data center through one or more of power capping, workload reduction, and active migration (e.g., before a planned maintenance event). In some embodiments, redundant computing and / or IT resources are used only in planned maintenance or power outage scenarios, and the redundant resources may not be used during normal data center operations. Therefore, redundant computing and / or IT resources can provide opportunities for various grid interactive services, such as frequency regulation, frequency suppression, and demand response.

[0065] Such example scenarios may require different reaction times and durations, and are limited by battery capacity and the need to maintain sufficient backup energy to support any data center failure events. In some embodiments, systems and methods of power management according to the present disclosure utilize a combination of energy storage for rapid reaction in short periods of time and workload management for long-term regulation. A hybrid approach combining on-site power sources with workload management (e.g., power capping, workload shifting, power-aware scheduling) can further reduce reliance on fossil fuel-based electricity.

[0066] In some embodiments, a data center site consists of one or more co-located data centers that drive power from the same high voltage utility substation. In some embodiments, the utility high voltage lines are injected into the substation, which in turn feeds multiple spaces (hosting) in one or more data centers through a set of medium voltage transformers. In at least one embodiment, an external utility (e.g., an electric utility) supplies power to multiple host hostings, and each host hosting can have its own transformer, UPS backup battery, generator, and combination thereof. One or more host hostings can participate in grid services, and some embodiments of the control systems and methods described herein can coordinate available backup batteries and workload characteristics across these host hostings.

[0067] In some embodiments, a system for power management in a data center includes at least a control service that obtains or accesses multiple attributes of utilities and data centers to provide instructions to one or more components of the data center. The instructions provided by the control service allow the data center to more efficiently make computing services available to users of the data center. The control service can be remote from the data center and / or (multiple) host hosting, and obtain information about components of the data center through a network connection and communicate with components of the data center. In some embodiments, it may be beneficial for the control service to have a response time of less than 5 milliseconds (ms), less than 2ms, or less than 1ms to changing conditions, and it may be beneficial to have the control service located on-site in the data center to facilitate faster communication times. In some embodiments, the control service is a service running on a control computing device in a data center that communicates with other components of the data center. In some embodiments, the control service includes a dedicated processor, a hardware storage device, and / or a computing device that executes the control service.

[0068] In some embodiments, the control service communicates data with an energy controller of the host hosting. For example, each host hosting in a data center can have an energy controller that controls, allocates, manages the power supply infrastructure of the host hosting, and combinations thereof. In some examples, the energy controller is at least partially responsible for charging and / or discharging batteries for the host hosting. In some examples, the energy controller is at least partially responsible for other hardware power supply and / or power storage operations.

[0069] For example, the energy controller can communicate data with one or more UPS hosted by the host. In some embodiments, the host hosts at least one UPS for each server rack hosted by the host. In some embodiments, the host hosts a UPS for each server rack hosted by the host. In some embodiments, the host hosts at least one UPS configured to provide power to the server rack hosted by the host. In some embodiments, at least one UPS is configured to provide power to multiple server racks hosted by the host. The energy controller can communicate with the UPS to provide power or additional power to one or more server computers or other IT components in response to changes in the utility power supply of the host and / or data center.

[0070] In another example, the energy controller can be in data communication with one or more generators hosted by the host. In some embodiments, the host hosts at least one generator for each server rack hosted by the host. In some embodiments, the host hosts a generator for each server rack hosted by the host. In some embodiments, the host hosts at least one generator configured to provide power to the server racks hosted by the host. In some embodiments, at least one generator is configured to provide power to multiple server racks hosted by the host. The energy controller can communicate with the generator to provide power or additional power to one or more server computers or other IT components in response to changes in the utility power supply of the host and / or data center.

[0071] In another example, the energy controller can communicate data with one or more long-term energy storages (e.g., fuel cells or long-term batteries) hosted by the host. In some embodiments, the host hosts at least one long-term energy storage for each server rack hosted by the host. In some embodiments, the host hosts a long-term energy storage for each server rack hosted by the host. In some embodiments, the host hosts at least one long-term energy storage configured to provide power to the server rack hosted by the host. In some embodiments, at least one long-term energy storage is configured to provide power to multiple server racks hosted by the host. The energy controller can communicate with the long-term energy storage to provide power or additional power to one or more server computers or other IT components in response to changes in the utility power supply of the host hosting and / or data center.

[0072] In some embodiments, the control service communicates data with the workload controller. The workload controller is responsible for performing workload operations and / or controls, such as power capping, shutting down servers, VM allocation, process allocation, and workload migration. In some embodiments, the workload controller responds to long-term (several minutes to several hours) grid service requests through a combination of power capping, shutting down servers, VM allocation, process allocation, and workload migration. In some embodiments, the workload controller performs power capping, shutting down servers, VM allocation, process allocation, and workload migration based at least in part on hardware capabilities (e.g., whether it can be power capped / throttled), availability requirements (e.g., whether the software is redundant), utilization patterns, and potential impacts of one or more of power capping, shutting down servers, VM allocation, process allocation, and workload migration on workloads and / or processes.

[0073] In some embodiments, a determination and / or instruction to engage in one or more of power capping, server shutdown, VM allocation, process allocation, and workload migration is made at a control service. In some embodiments, a workload controller provides a list of feasible options for workload management to the control service (e.g., which of power capping, server shutdown, VM allocation, process allocation, and workload migration is an available option, based at least in part on hardware capabilities, availability requirements, and current workload / process). In some embodiments, the control service determines which option to engage in based at least in part on information obtained (e.g., public telemetry and infrastructure status). In some examples, the information obtained includes the amount of power that needs to be "recovered", delay requirements (e.g., a few seconds for unplanned events; advance notice for planned events), and a dynamic impact function defined by the workload. In some embodiments, parameters are obtained and / or calculated by the control service periodically and / or on demand and provided to the workload controller, which determines workload management decisions.

[0074] In some embodiments, the workload controller monitors critical events (e.g., grid service requests or data center equipment status) and takes corresponding actions when any critical events are detected. In such embodiments, the workload controller is tasked with responding to potential critical events (e.g., insufficient backup battery), which can reduce the response time to specific critical events.

[0075] The platform then uses these inputs and a set of heuristics or machine learning (ML) to decide whether to use hardware-based energy management, such as discharging long-term energy storage or starting a generator via an energy controller, and / or use a workload controller through software-defined mechanisms to reduce power consumption.

[0076] In some embodiments, the method of power management includes obtaining at least one workload state of at least one server rack at a control service, obtaining at least one infrastructure parameter, and obtaining at least one common telemetry. In some embodiments, the control service obtains at least a portion of the workload state from a workload controller. In some embodiments, the control service obtains at least a portion of the workload state from one or more server computers. In some embodiments, the controller service obtains at least a portion of the workload state from a rack manager of a server rack hosted by a host. For example, the rack manager can communicate with one or more server computers in a server rack, and the rack manager can monitor the power consumption of (multiple) server computers. Power consumption is the amount of electrical power (from all power sources inside or outside the data center) required by (multiple) server computers to perform the current or requested workload. For example, the power consumption of a single server computer can be monitored, the power consumption of a rack of server computers can be monitored, or the power consumption of a host hosting of server computers can be monitored.

[0077] In some embodiments, the workload state includes one or more of VM allocation, process allocation, process priority list, process migration state, utilization mode, workload performance and availability requirements, failover capability, or other information related to the computing operation of the server computer hosted by the host. For example, the VM allocation information can inform the workload controller and / or control service of the number of VMs allocated to a specific server computer, server rack, or host. The number of VMs can inform the workload controller and / or control service of the computing capacity available on the allocated server and / or the maximum power consumption that the allocated server may require. The workload controller and / or control service can use the VM allocation information to help estimate the computing and power requirements of the host hosting and / or data center.

[0078] In some examples, process allocation information can notify a workload controller and / or control service of a particular process requested or currently being executed on at least some server computers in a server rack, a hosting, and / or a data center. In some embodiments, the workload controller and / or control service includes a process list to monitor the processes and their power consumption and computing requirements. For example, the workload controller and / or control service can determine that a first process assigned to a first server computer within a hosting has a first power consumption associated with the first process, and the workload controller and / or control service can determine that a second process assigned to a second server computer within the hosting has a second power consumption associated with the second process. The workload controller and / or control service can determine the current total power consumption or expected power consumption of the assigned processes based at least in part on the process list.

[0079] In some embodiments, the process assignment and / or process list has a process priority list that informs the workload controller and / or control service of the relative importance of processes currently executing or queued on the host. For example, a first process assigned to a first server computer may have a higher priority than a second process assigned to a second server computer, and the workload controller may power cap or throttle the second server computer to prioritize the performance of the first process of the first server computer.

[0080] In some examples, the process migration status can notify the workload controller and / or control service of the availability of computing resources for process migration between server computers and / or between hosting. In some examples, the process migration status can notify the workload controller and / or control service of the initiation and / or completion of process migration to allow the workload controller and / or control service to track the availability of computing resources.

[0081] In some embodiments, the utilization pattern can inform the workload controller and / or control service of the current or predicted future state of the workload on the host hosting based at least in part on historical data and resource utilization trends. For example, the utilization pattern can include process allocation, power consumption, and / or computational load, which is based at least in part on the time of day, day of the week, day of the year, or correlation with other events (e.g., weather, holidays, or periodic events). In some embodiments, the workload controller and / or control service can determine the trend or predicted future state of the workload based on the utilization pattern, and preemptively change or adjust the workload or power supply to at least partially compensate for the trend or predicted future state of the workload.

[0082] In some embodiments, the control service and / or the energy controller obtains infrastructure parameters. For example, the energy controller may obtain or store infrastructure parameters, and the control service may obtain infrastructure parameters from the energy controller. The infrastructure parameters include information related to the performance, history, or requirements of the hardware of the hosting and / or data center. In some embodiments, the infrastructure parameters may include battery charge status, UPS charge status, battery / UPS degradation (e.g., degradation counters), component temperature, server computer power consumption, maintenance scheduling, and other measurements or properties of (multiple) energy sources and (multiple) energy sinks within the hosting and / or data center.

[0083] In some embodiments, the battery charge state or UPS charge state includes a percentage state of charge of a long-term battery and / or UPS, a nominal voltage of a long-term battery and / or UPS, or a nominal state of charge of a long-term battery and / or UPS (e.g., measured in kilowatt-hours). The battery or UPS charge state can inform the control service and / or energy controller of the duration that the battery / UPS can provide power or additional power to the server computer in the event of a utility failure or other event.

[0084] In some embodiments, the generator capacity allows the control service and / or energy controller to know how much peak power the generator can provide, how long the generator can provide power, and the total power the generator can provide. In some embodiments, the infrastructure parameters also include a start time for the generator, which can inform the control service and / or energy controller of the delay in starting the generator before the generator can begin providing power to the colocation and / or data center.

[0085] In some embodiments, the battery / UPS degradation includes total battery aging parameters for the battery and / or UPS, the number of charge / discharge cycles of the battery and / or UPS, or the available charge depth. For example, based on the depth of charge / discharge, a battery may have a limited number of charge cycles. A cost is associated with each discharge and charge cycle of the battery. In some embodiments, the age of the battery affects the capacity of the battery, which limits the amount of power that an older battery can provide to the host hosting relative to a newer battery. In some embodiments, the energy controller monitors the age of the battery (in terms of time, cycles, or capacity) and provides the age to the control service. In some embodiments, the control service monitors the age of the battery based at least in part on information provided by the energy controller.

[0086] In some embodiments, the energy controller and / or control service monitors component temperature because elevated temperatures may affect the efficiency and / or operating life of a battery, UPS, or generator in data communication with the energy controller and / or control service.

[0087] In some embodiments, the infrastructure parameters include maintenance scheduling of one or more components of the hosting and / or data center. For example, some components may be unavailable due to planned maintenance. In other examples, planned maintenance may require additional power provided by batteries, fuel cells, other long-term energy storage, UPS, or generators, and the energy controller and / or control service can prepare the batteries, UPS, or generators in advance to provide the capacity required for the additional power.

[0088] In some embodiments, the control service obtains utility telemetry from a substation, utility lines, or other communications with a utility grid. Utility telemetry may include the frequency of power provided by a utility to a substation and / or through a substation, the carbon intensity of power provided by a utility to a substation and / or through a substation, and the power demand and response of the utility grid. The frequency of the power provided by the utility may vary with utility grid stability, supply, and demand. In a utility grid, operators attempt to maintain utility grid balance and reliability and keep their frequency within defined limits. Deviations from the nominal frequency, i.e., 50 Hz or 60 Hz, are caused by a mismatch between supply and demand (a phenomenon exacerbated by greater penetration of variable renewable energy sources, which is also associated with the carbon intensity of the power provided). In some embodiments, a control service according to the present disclosure may help operators adjust the utility grid frequency by reducing or increasing the power consumption of a hosting and / or data center. Frequency regulation may require fast-response energy storage, generators, or fast workload management.

[0089] Additionally, some embodiments of the control services according to the present disclosure may provide or assist in providing frequency suppression. In some examples, frequency suppression reserves (e.g., provided by long-term energy storage, a UPS, or a generator in communication with an energy controller) may provide a primary response to sudden frequency changes (typically low frequencies) caused by unexpected events in the utility grid or sudden drops in renewable energy. When system inertia is reduced in areas with high penetration of renewable energy, even faster-reacting reserves (e.g., batteries relative to generators) may be used to at least partially compensate for low inertia conditions. In some embodiments, fast-responding battery energy storage in a data center with limited energy storage duration may provide frequency suppression in response to acquired utility telemetry.

[0090] In some embodiments, the utility telemetry obtained includes carbon intensity related to the source and delivery of power provided by the utility grid.In some embodiments, the control service selectively charges or discharges batteries, or starts generators, based on the reduced carbon impact of the colocation and / or data center.

[0091] In some embodiments, the method further comprises comparing at least one workload state and at least one utility telemetry, and determining a workload demand based at least in part on a difference between the at least one workload state and the at least one utility telemetry. Comparing at least one workload state and at least one utility telemetry allows the control service to determine a difference between the power provided by the utility grid and the power required for the current or predicted future state of the workload. For example, the workload state and utility telemetry may indicate a positive workload demand, which may require deployment of one or more infrastructure power sources, such as long-term batteries, UPS, or generators, to at least partially compensate for the positive workload demand. In some embodiments, the workload state and utility telemetry may indicate a negative workload demand, which may provide an opportunity to charge long-term batteries or overclock process resources in a hosting and / or data center.

[0092] In some embodiments, the method further comprises changing at least one infrastructure parameter based at least in part on the workload demand and at least one infrastructure parameter. For example, changing at least one infrastructure parameter may include discharging a long-term energy storage. In some examples, changing at least one infrastructure parameter may include charging a long-term battery. In some examples, changing at least one infrastructure parameter may include discharging a UPS. In some examples, changing at least one infrastructure parameter may include starting a generator. In some embodiments, changing at least one infrastructure parameter based on the workload demand supplements the power received from the public grid. In some embodiments, changing at least one infrastructure parameter based on the workload demand supplements the power received from the public grid without outputting power from the data center (i.e., from the long-term energy storage and / or the generator) to the public grid.

[0093] Workload state, infrastructure parameters, and common telemetry may be inputs to an ML model to determine outputs provided by a control service to a controller. In other embodiments, workload state, infrastructure parameters, and common telemetry may be inputs to one or more of heuristics, mathematical models, and algorithms to determine outputs provided by a control service to a controller.

[0094] As used herein, "machine learning model" refers to a computer algorithm or model (e.g., a classification model, a regression model, a language model, an object detection model) that can be adjusted (e.g., trained) to approximate an unknown function based on training inputs. For example, a machine learning model can refer to a neural network or other machine learning algorithm or architecture that learns and approximates complex functions based on multiple inputs provided to the machine learning model, and generates an output. In some embodiments, the machine learning system, model, or neural network described herein is an artificial neural network. In some embodiments, the machine learning system, model, or neural network described herein is a convolutional neural network. In some embodiments, the machine learning system, model, or neural network described herein is a recurrent neural network. In at least one embodiment, the machine learning system, model, or neural network described herein is a Bayesian classifier. As used herein, "machine learning system" can refer to one or more machine learning models that collaboratively generate one or more outputs based on corresponding inputs. For example, a machine learning system can refer to any system architecture with multiple discrete machine learning components that consider different kinds of information or inputs.

[0095] As used herein, an "instance" refers to an input object that can be provided as input to a machine learning system to generate outputs, such as utility telemetry, workload status, and infrastructure parameters. For example, an instance can refer to any event in which utility telemetry changes in a manner that affects the frequency of power provided by the utility grid. For example, low-frequency events can be associated with afternoons or evenings in areas with warmer weather, corresponding to increased demand on the utility grid. In some embodiments, low-frequency events can be at least partially compensated for with power capping or workload migration, while in other instances, low-frequency events can be at least partially compensated for with additional infrastructure power sources when workloads are high or process priority is high.

[0096] In some embodiments, the machine learning system has multiple layers, including an input layer configured to receive at least one input data set or input instance, and an output layer with multiple additional or hidden layers between the input layer and the output layer. The training data set can be input into the machine learning system to train the machine learning system and identify individual and combinations of labels or attributes of the training instances that allow the hosting and / or data center to participate in the grid service. In some embodiments, the input includes utility telemetry, workload status, infrastructure parameters, or a combination thereof.

[0097] In some embodiments, the machine learning system can concurrently receive multiple training data sets and learn from different training data sets simultaneously. For example, a training data set of common grid common telemetry changes includes different information and / or labels than a training data set including changes in workload states.

[0098] In some embodiments, the machine learning system includes multiple machine learning models running together. Each machine learning model has multiple hidden layers between the input layer and the output layer. The hidden layer has multiple nodes, each of which operates on the input received from the previous layer. In a specific example, the first hidden layer has multiple nodes, and each node performs an operation on each instance from the input layer. Each node of the first hidden layer provides new inputs to each node of the second hidden layer, and each node of the second hidden layer in turn performs a new operation on each of these inputs. The nodes of the second hidden layer then pass the output (e.g., the identified clusters) to the output layer.

[0099] In some embodiments, each of the nodes has a linear function and an activation function. The linear function may attempt to optimize or approximate a solution with a best fit line, such as reduced power cost or reduced carbon intensity. The activation function is used as a test to check the effectiveness of the linear function. In some embodiments, the activation function produces a binary output that determines whether the output of the linear function is passed to the next layer of the machine learning model. In this way, the machine learning system can limit and / or prevent poor fits from propagating to the data and / or non-convergent solutions.

[0100] The machine learning model includes an input layer that receives at least one training data set. In some embodiments, at least one machine learning model uses supervised training. Supervised training allows input of multiple common grid or workload events with known responses from energy controllers and / or workload controllers, and allows the machine learning system of the control service to develop correlations between the inputs and responses to learn risk factors and combinations thereof. In some embodiments, at least one machine learning model uses unsupervised training. Unsupervised training can be used to draw inferences and find patterns or associations from (multiple) training data sets in the absence of known events. In some embodiments, unsupervised learning can identify clusters of similar labels or characteristics for various training instances, and allow the machine learning system to extrapolate safety and / or risk factors for instances with similar characteristics.

[0101] In some embodiments, semi-supervised learning can combine the advantages of supervised learning and unsupervised learning. As described herein, a machine learning system can identify associated labels or features between instances, which can allow for the fusion of a training data set with known events and a second training data set including more general input information. Unsupervised training can allow a machine learning system to cluster instances from a second training data set without known events and associate the clustering clusters with known events from a first training data set.

[0102] In some embodiments, offline training of the ML model may include a simulation environment having a data center simulator that receives input from an IT simulator and a simulated data center power infrastructure. The simulation environment outputs a state to a reinforcement learning (RL) agent, the output state having a reward (positive or negative) associated with the state of the data center simulator. The RL agent may create a loop that provides further input to the simulation environment to improve the response and / or output of the simulation environment over time.

[0103] In some embodiments, the RL agent provides information to an online ML model that receives real-time inputs including utility telemetry, workload status, and infrastructure parameters. The RL agent and simulation environment may allow the ML model to be pre-trained and / or continuously trained offline with additional scenarios, which are then fused with the real-time inputs at the ML model. The simulation environment may allow for faster training of the ML model without the data center and / or colocation experiencing adverse utility grid or infrastructure conditions simulated in the simulation environment.

[0104] In some embodiments, a method of power management in a data center includes making changes to a host-hosted workload and a host-hosted power infrastructure. A control service can communicate with an energy controller and a workload controller to provide short-term (e.g., fast response) and long-term grid services and workload balancing. For example, a control service can communicate with an infrastructure power source (e.g., a battery, UPS, generator) to provide a fast response to a change in utility telemetry or workload status, and the control service also communicates with a workload controller to migrate a workload to a second host-hosted and / or to power cap a first host-hosted to balance workload demand between available power and the cost or carbon intensity of the available power.

[0105] In some embodiments, the method includes obtaining, at a control service, at least one workload state of at least one server rack, obtaining at least one infrastructure parameter, and obtaining at least one common telemetry. In some embodiments, the control service obtains at least a portion of the workload state from a workload controller. In some embodiments, the control service obtains at least a portion of the workload state from one or more server computers. In some embodiments, the controller service obtains at least a portion of the workload state from a rack hypervisor of a server rack in a hosting operation.

[0106] In some embodiments, the method further comprises inputting at least one common telemetry, at least one infrastructure parameter, and at least one workload state into an ML model, as described herein, to determine at least one change to each of the at least one infrastructure parameter and the at least one workload state. Based on the output of the ML model, which may be pre-trained as described herein, the method comprises changing the at least one infrastructure parameter and the at least one workload state.

[0107] In some examples, the control service may change at least one infrastructure parameter based at least in part on utility telemetry to charge the infrastructure power source of the utility grid. In some embodiments, the control service may change at least one infrastructure parameter based at least in part on workload status to charge the infrastructure power source of the utility grid.

[0108] For example, a positive workload demand may result from an infrastructure failure or a partial infrastructure failure. In some embodiments, a control service and / or workload controller may adjust the workload and / or process of a server computer based on a change in an infrastructure parameter. In at least one example, a change in an infrastructure parameter may be related to a limited availability of infrastructure power provided to the server computer. In such an example, a control service and / or workload controller may migrate the workload to another hosting to reduce the workload demand. In some examples, a control service and / or workload controller may cap or throttle the server computer power to reduce the workload demand.

[0109] The present disclosure relates to systems and methods for power management in a data center, the systems and methods being at least according to the examples provided in the following sections:

[0110] [A1] In some embodiments, a method of power management in a data center includes obtaining at least one workload state of at least one server rack, obtaining at least one infrastructure parameter, obtaining at least one utility telemetry, and comparing at least one workload state and at least one utility telemetry. The method also includes determining a workload demand based at least in part on a difference between at least one workload state and at least one utility telemetry, and changing at least one infrastructure parameter based on the workload demand and at least one infrastructure parameter.

[0111] [A2] In some embodiments, at least one workload state of [A1] includes power consumption of at least one server rack.

[0112] [A3] In some embodiments, at least one workload state of [A1] or [A2] includes a virtual machine (VM) allocation to at least one server rack.

[0113] [A4] In some embodiments, at least one workload state of any one of [A1] to [A3] includes a predicted future state of at least one workload state.

[0114] [A5] In some embodiments, comparing at least one workload state and at least one public telemetry of any of [A1] to [A4] includes inputting at least one workload state and at least one public telemetry into an input layer of a machine learning (ML) model, and wherein the workload demand is an output of the ML model.

[0115] [A6] In some embodiments, at least one infrastructure parameter of any one of [A1] to [A5] includes generator capacity.

[0116] [A7] In some embodiments, at least one infrastructure parameter of any one of [A1] to [A6] includes a battery charge status.

[0117] [A8] In some embodiments, at least one infrastructure parameter of any one of [A1] to [A7] includes a battery aging parameter.

[0118] [A9] In some embodiments, at least one common telemetry of any one of [A1] to [A8] includes a frequency.

[0119] [A10] In some embodiments, at least one common telemetry of any of [A1] to [A9] includes a requirement.

[0120] [A11] In some embodiments, at least one common telemetry of any of [A1] to [A10] includes carbon intensity.

[0121] [A12] In some embodiments, changing at least one infrastructure parameter of any of [A1] to [A11] includes communicating with a workload controller to change the workload of at least one server rack and reduce the power consumption of at least one server rack.

[0122] [A13] In some embodiments, changing at least one infrastructure parameter of any of [A1] to [A12] includes communicating with an energy controller to change the power supply to at least one server rack.

[0123] [A14] In some embodiments, changing at least one infrastructure parameter of any one of [A1] to [A11] includes changing the power supply to at least one server rack, and changing the workload of at least one server rack.

[0124] [B1] In some embodiments, a system for controlling power supply in a data center includes a control service, an energy controller in data communication with the control service, and a workload controller in data communication with the control service. The control service is configured to obtain at least one workload state of at least one server rack, obtain at least one infrastructure parameter, obtain at least one common telemetry, and compare at least one workload state and at least one common telemetry. The control service is also configured to determine a workload demand based at least in part on a difference between at least one workload state and at least one common telemetry, and to change at least one infrastructure parameter based on the workload demand and at least one infrastructure parameter.

[0125] [B2] In some embodiments, the system of [B1] further includes at least one battery in data communication with the energy controller.

[0126] [B3] In some embodiments, the system of [B1] or [B2] further includes at least one generator in data communication with the energy controller.

[0127] [B4] In some embodiments, the energy controller of any one of [B1] to [B3] is a first energy controller associated with the first host hosting, and the workload controller is a first workload controller associated with the first host hosting. The system also includes a second energy controller associated with the second host hosting, wherein the control service is in data communication with the second energy controller, and the system includes a second workload controller associated with the second host hosting, wherein the control service is in data communication with the second workload controller.

[0128] [C1] In some embodiments, a method of power management in a data center includes obtaining at least one workload state of at least one server rack, obtaining at least one infrastructure parameter, obtaining at least one utility telemetry, and inputting the at least one utility telemetry, at least one workload state, and at least one infrastructure parameter into an ML model. The method also includes changing at least one infrastructure parameter based on the at least one utility telemetry, at least one workload state, and at least one infrastructure parameter, and changing at least one workload state based on the at least one utility telemetry, at least one workload state, and at least one infrastructure parameter.

[0129] [C2] In some embodiments, the ML model of [C1] is pre-trained using an offline simulation environment.

[0130] The articles "one", "an", and "the" are intended to mean that there are one or more elements in the foregoing. The words "comprising", "including", and "having" are intended to be inclusive, and mean that in addition to the listed elements, there may be other elements. Additionally, it should be understood that the reference to "one embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of other embodiments that also contain the features. For example, any element related to the embodiments described herein can be combined with any element of any other embodiment described herein. As will be understood by those of ordinary skill in the art covered by the embodiments of the present disclosure, the numerals, percentages, ratios, or other values ​​specified herein are intended to include the value, and other values ​​that are "about" or "close to" the value. Therefore, the specified value should be interpreted broadly enough to include at least a value close enough to the specified value to perform the desired function or achieve the desired result. The specified value includes at least the expected change in a suitable manufacturing or production process, and may include a value within 5%, within 1%, within 0.1%, or within 0.01% of the specified value.

[0131] Those of ordinary skill in the art should recognize that, in light of the present disclosure, equivalent constructions do not depart from the scope of the present disclosure, and various changes, substitutions, and modifications may be made to the embodiments disclosed herein without departing from the scope of the present disclosure. Equivalent constructions including functional "means-plus-function" clauses are intended to cover structures described herein that perform the functions described, including structural equivalence that operates in the same manner and equivalent structures that provide the same functions. It is the express intent of the applicant not to invoke means-plus-function or other functional claims for any claim, except for claims in which the term "means for..." appears together with the associated function. Every addition, deletion, and modification to the embodiments that falls within the meaning and scope of the claims will be encompassed by the claims.

[0132] It should be understood that any of the aforementioned directions or reference frames are only relative directions or movements. For example, any reference to "front" and "back" or "top" and "bottom" or "left" and "right" is only a description of the relative position or movement of the relevant elements.

[0133] The present disclosure may be embodied in other specific forms without departing from its characteristics. The described embodiments should be regarded as illustrative rather than restrictive. Therefore, the scope of the present disclosure is indicated by the appended claims rather than by the preceding description. Changes within the equivalent meaning and scope of the claims should be included within the scope of the claims.

Claims

1. A power management method in a data center, the method include: At Control Services: obtaining at least one workload status of at least one server rack; obtaining at least one infrastructure parameter; obtaining at least one public telemetry; comparing the at least one workload state and the at least one common telemetry; determining a workload demand based at least in part on a difference between the at least one workload state and the at least one common telemetry; and The at least one infrastructure parameter is changed based on the workload demand and the at least one infrastructure parameter. 2 . The method of claim 1 , wherein the at least one workload status comprises power consumption of the at least one server rack.

3. The method of claim 1 or claim 2, wherein the at least one workload state comprises a virtual machine (VM) allocation to the at least one server rack.

4. A method according to any preceding claim, wherein the at least one workload state comprises a predicted future state of the at least one workload state.

5. The method according to any of the preceding claims, wherein the at least one workload state is compared with the at least one common telemetry include: The at least one workload state and the at least one common telemetry are input into an input layer of a machine learning (ML) model, and wherein the workload demand is an output of the ML model.

6. A method according to any preceding claim, wherein the at least one infrastructure parameter comprises generator capacity.

7. A method according to any preceding claim, wherein the at least one infrastructure parameter comprises a battery charge state.

8. A method according to any preceding claim, wherein the at least one infrastructure parameter comprises a battery aging parameter.

9. A method according to any preceding claim, wherein the at least one common telemetry comprises a frequency.

10. A method according to any preceding claim, wherein the at least one common telemetry comprises a demand.

11. A method according to any preceding claim, wherein the at least one utility telemetry comprises carbon intensity.

12. The method of any preceding claim, wherein changing at least one infrastructure parameter comprises communicating with a workload controller to change the workload of the at least one server rack and to reduce power consumption of the at least one server rack.

13. The method of any preceding claim, wherein changing at least one infrastructure parameter comprises communicating with an energy controller to change the power supply to the at least one server rack.

14. The method of any preceding claim, wherein changing at least one infrastructure parameter comprises changing a power supply to the at least one server rack and changing a workload of the at least one server rack.

15. A system for controlling power supply, the system include: Control services; an energy controller in data communication with the control service; as well as a workload controller in data communication with the control service, Wherein the control service is configured to perform a method according to any preceding claim.

Citation Information

Cited By

  • Data center cooling system performance evaluation method and system

    CN120407364A