Real-time carbon footprint reduction controller

Through reinforcement learning frameworks and digital twin technology, the energy consumption and load transfer of data centers are optimized in real time, and the carbon footprint reduction problem under the complex interdependence of data centers is solved, achieving significant energy and carbon emission reduction.

CN119024734APending Publication Date: 2024-11-26HEWLETT PACKARD ENTERPRISE DEV LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410461620.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-10-23
Filing Date
2024-04-17
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The difficulty in real-time optimization of multiple power loss factors in data centers in the prior art makes it challenging to reduce carbon footprint, especially under complex interdependence and dynamic external factors, traditional methods lack real-time concurrency optimization capabilities.

Method used

Using reinforcement learning (RL) framework, leveraging short-term external factor forecasts through multiple agents (such as weather and grid carbon intensity) to optimize energy consumption, flexible load transfer and battery operation in real time, leveraging digital twins and Markov decision-making processes (MDP) Formula decouples complex problems, and the collaborative reward mechanism manages interdependence among subsystems.

Benefits of technology

Real-time and comprehensive carbon footprint optimization of data centers is achieved, energy consumption and carbon emissions are reduced, average energy costs are reduced by 8.04%, carbon emissions are reduced by 9.06%, and a scalable modular architecture is provided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024734A_ABST
    Figure CN119024734A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a real-time carbon footprint reduction controller. Systems and methods are provided for simultaneously optimizing energy consumption, flexible load transfer, and battery operation decisions in real time through reinforcement learning (RL). Examples include obtaining a state of a system including a plurality of subsystems and receiving, by an RL agent, a reward from a digital twin of the system, the reward including a plurality of rewards, each reward associated with a subsystem. The example also includes determining, by the RL agent, an action based on the state and each of the rewards. Each RL agent is associated with a subsystem and assigns a weight to an award corresponding to the associated subsystem, the weight being greater than the weights of awards assigned to other subsystems. The system may then be controlled to transition to an updated state according to the action.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 504,664, filed on May 26, 2023, the entire contents of which are incorporated herein by reference. Background Art

[0003] Carbon footprint serves as an indicator for comparing the total amount of greenhouse gases emitted from a system. The commonly used definition of carbon footprint is a measurement of the total amount of carbon dioxide (CO2) and methane (CH4) emissions of a defined system, taking into account all relevant sources, sinks and storage within the spatial and temporal boundaries of the system. Carbon footprints are usually reported in tons of emissions (CO2 equivalents) per comparative unit (such as per year, per month, etc.). For a system, a carbon footprint can include direct emissions, as well as indirect emissions caused by the system. Direct emissions refer to emissions directly from the sources of the system, and indirect emissions refer to emissions from upstream or downstream sources of the system originating from the activities of the system (whether or not controlled by the system). BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The present disclosure according to one or more embodiments is described in detail with reference to the accompanying drawings. The drawings are provided for purposes of illustration only and depict only typical or example embodiments.

[0005] Figure 1A and Figure 1B Depicted is the architecture of an example Data Center Carbon Footprint Reduction (DC-CFR) framework in accordance with implementations of the present disclosure.

[0006] Figure 2 It is a schematic diagram of the interdependencies and information exchanges between the subsystems of a data center.

[0007] Figure 3 is a schematic block diagram of an example process of reinforcement learning according to an implementation of the present disclosure.

[0008] Figure 4 are example computing components that may be used to implement various features of carbon footprint reduction in accordance with implementations disclosed herein.

[0009] Figure 5 is an example computer system that may be used to implement the various features of the carbon footprint reduction of the present disclosure.

[0010] The drawings are not exhaustive and do not limit the disclosure to the precise forms disclosed. DETAILED DESCRIPTION

[0011] Sustainability with reduced carbon footprint is becoming a priority for governments and businesses around the world. The current rise in global average temperatures is faster than previous changes, which is mainly caused by activities that burn fossil fuels to provide electricity and power. It is expected that by 2030, information and communication technology will account for 7% to 20% of global electricity demand. Data centers account for a large proportion of the global carbon footprint. It is estimated that the carbon footprint demand of these data centers will increase in the coming years, increasing 6 times every decade.

[0012] Achieving sustainability can involve optimizing various interdependent power consumption factors of a data center, such as the power consumed by operating the data center (e.g., in cooling and by information technology (IT) workloads), shifting flexible workloads based on the availability of renewable energy in the power grid, and utilizing power stored in batteries from the power grid. Flexible loads refer to certain IT workloads that are allowed to be shifted from one task to another or from one time to another to optimize other aspects of the data center. Static or inflexible workloads are those that are not allowed to be shifted and may be necessary to complete the tasks as scheduled.

[0013] Traditionally, optimization of these power loss factors is performed offline, where each factor is optimized separately and in isolation. Conventional systems lack the ability to concurrently optimize the various power loss factors of a data center in real time. This is because concurrent optimization is a difficult problem to solve due to the complex interdependencies between the various power loss factors to be optimized, as well as the dependence of the power loss factors on external factors (such as, but not limited to, weather and the availability of renewable energy in the power grid).

[0014] Analytical methods have been proposed. However, these proposed methods rely on offline static analysis and do not utilize external factors (such as weather forecasts) to optimize data center cooling (e.g., heating, ventilation, and air conditioning (HVAC) systems) to reduce total energy consumption simultaneously with comprehensive carbon reduction actions. In addition, static analytical methods collect day-ahead carbon intensity (CI) forecasts, train day-ahead energy supply and demand forecasting models, and use them to analytically find the optimal data center workload distribution to minimize the total carbon footprint. Since workloads move within 24 hours, these day-ahead analyses may ignore the net carbon footprint and therefore lack granularity for real-time implementation. In addition, due to changing weather patterns, relying on long-term forecast models may not be feasible. In addition, the complex interdependencies between energy consumption, load balancing, and energy storage, as well as the information exchange between the individual aspects, have hindered the previous development of a cohesive strategy to use all three opportunities in real time to reduce the carbon footprint simultaneously. Therefore, due to the complexity of these individual problems and the reliance on relatively long forecast horizons (e.g., 24 hours), it has proven challenging to achieve carbon footprint reduction for the entire system using these analytical methods.

[0015] In contrast, examples of the technology disclosed herein provide a framework for utilizing reinforcement learning (RL) to simultaneously optimize energy consumption, flexible load shifting, and battery operation decisions in real time, which overcomes the technical deficiencies of existing solutions. Such optimization can be based on forecasts of external factors using multiple agents (e.g., carbon intensity on the grid and weather conditions, to name a few examples). Such forecast ranges may be relatively short, for example, less than a 24-hour forecast, less than 12 hours, less than 1 hour, etc. In some cases, the forecast range of the present disclosure may be very short, for example, a 1-minute range or less. Therefore, the implementation disclosed herein can be based on forecasts of any short-term range as desired by the desired application. Therefore, the technology disclosed herein can effectively alleviate the shortcomings of existing isolated approaches by providing a framework for managing complex interdependencies and information exchanges between individual optimization strategies adopted by competing data center subsystems.

[0016] Carbon intensity (CI) as used herein refers to the carbon dioxide (CO2) emissions generated by the power grid per unit of electricity consumed from the grid, which varies based on the source of electricity (e.g., fossil fuels, renewable energy, etc.) and the party consuming the energy at a given point in time. Therefore, a large consumption of non-renewable energy from the grid would be considered a high CI period, while a lower CI means that the energy can be obtained from greener energy sources.

[0017] A framework according to the present disclosure provides a digital twin of a real-world data center that simulates the operation of the data center. The framework uses RL to comprehensively optimize power loss factors (such as energy loss, load transfer, and energy storage operations) for the digital twin in real time, and learns the optimization strategy used to control the operation of the real-world data center. The digital twin can be notified by the data center of real-time changes in operating conditions, which can dynamically adapt the digital twin to the changing conditions. The framework simultaneously optimizes the power loss of the digital twin through RL informed by the changing operating conditions. An example of the disclosed technology solves the exponential growth of power loss requirements by decoupling the problem into separate sub-problems through the RL Markov decision process (MDP) formula. The disclosed technology simultaneously solves the RL MDP formula to develop a real-time carbon footprint optimization strategy customized for sustainability and carbon footprint reduction. The optimized strategy can then be used to configure and control the operation of the real-world data center.

[0018] In an example implementation, the framework includes multiple RL agents that are trained on historical data to form an adaptive policy that is optimized to address complex and interrelated power loss factors in real time. The framework manages the complex interdependencies and information exchange between these RL agents through MDP formulation to provide an integrated real-time solution for carbon footprint reduction.

[0019] According to the examples disclosed herein, from a control perspective, a data center can be divided into multiple subsystems, such as a cooling subsystem (e.g., an HVAC system), a load transfer subsystem for transferring flexible workloads of the data center, and an energy storage subsystem including an energy storage device (e.g., a battery of a UPS system) that can be charged and discharged from the power grid to achieve uninterrupted power supply. Each subsystem can be simulated by a corresponding subsystem model (also referred to as a subsystem digital twin), which can be included as part of a system-wide digital twin. An RL agent in a plurality of RL agents can be associated with each subsystem and interfaced with the corresponding subsystem model. For example, an energy optimization RL agent can be provided to interface with the cooling subsystem model and configured to determine an optimized strategy for controlling the cooling subsystem to minimize the total energy loss caused by IT workload and cooling. The RL agent can further optimize energy reduction using weather forecasts. A load transfer RL agent can be provided to interface with the load transfer subsystem model and configured to determine a strategy for transferring flexible loads from a time period of low renewable energy availability on the power grid to a high availability time period (e.g., high to low carbon intensity (CI) on the grid). The load shifting RL agent may also utilize future predictions of CI (e.g., high CI may occur at noon when there is a high IT load) and weather forecast data. An energy storage RL agent is also provided to interface with the energy storage subsystem model and is configured to determine strategies for energy storage in the batteries of the energy storage subsystem during low CI periods and for discharging the stored energy from the batteries during high CI periods to assist in optimizing the data center energy consumption requirements. The energy storage RL agent may also account for the nonlinear charging and discharging patterns of the batteries.

[0020] According to various implementations, the framework can manage complex interdependencies and information exchanges between multiple RL agents through collaborative rewards. The collaborative reward includes a reward for each subsystem model, and each reward is uniquely weighted for each RL agent. That is, for example, each subsystem model can output a reward (e.g., the cooling subsystem model outputs a cooling reward, the load transfer subsystem model outputs a load transfer reward, etc.). By making the reward output by the subsystem model associated with a given RL agent heavier than other rewards, individual rewards can be combined into a collaborative reward customized for each corresponding RL agent. For example, in the case of an energy-optimizing RL agent, the reward output by the cooling subsystem model can be weighted as 0.8, while other rewards are weighted as 0.1. In this way, each agent can understand all other rewards and can determine actions that optimize the entire system, rather than selfish individual optimizations. This is in contrast to traditional non-RL-based systems, in which each subsystem does not know or consider the considerations and actions of other subsystems in real time.

[0021] Based on the individually customized collaboration rewards provided in the disclosed technology and the real-time operating conditions of the data center, each RL agent can determine actions for controlling its corresponding subsystem, thereby optimizing not only the individual subsystem but also the system-wide data center operation. The individually customized collaboration rewards enable each individual agent to account for the various interdependencies between the subsystems.

[0022] Therefore, the technology disclosed herein provides a carbon emission-aware real-time holistic control framework based on multi-agent RL for optimizing sustainable data centers through efficient cooling, redistribution of flexible server workloads, and battery storage for auxiliary energy supply. The technology disclosed herein can help solve complex interdependencies between control RL agents with dynamic external dependencies and produce performance gains relative to static optimizers, as described above. In addition, the technology disclosed herein can optimize data centers for multiple goals of carbon footprint reduction, energy consumption, and energy costs. The optimization by the disclosed technology can be performed using industry standard simulators and models across multiple geographic locations, where data center construction is feasible, with different weather patterns, and across multiple seasons. For example, based on industry standards developed by the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) and the Residential Building Council (ASHRAE), the disclosed technology can achieve an average 8.04% reduction in energy costs compared to traditional controls. In addition, the technology according to the present disclosure can achieve additional emission reductions through battery and load transfer optimization, resulting in an average reduction of, for example, 9.06% of carbon emissions between different data center configurations.

[0023] In addition, the framework including the data center digital twin, the control RL agent, and the RL interface provides a scalable modular architecture that can be extended to additional optimization controllers and can help democratize carbon reduction efforts by the ecosystem. For example, implementations of the present disclosure can be scaled across multiple physical parts of a data center (e.g., multiple rooms, multiple cities, multiple geographic regions, etc.). Scalability can be achieved through digital twins that can model each subsystem as a whole, or divide the subsystem into smaller parts (e.g., each room, each geographic region, etc.).

[0024] It should be noted that the terms "optimize," "optimal," and the like as used herein may be used to mean making or achieving as efficient or perfect a performance as possible. However, as one of ordinary skill in the art reading this document will recognize, perfection is not always achieved. Thus, these terms may also encompass making or achieving as good or efficient a performance as possible or practical under given circumstances, or making or achieving better performance than could be achieved using other settings or parameters.

[0025] Figure 1A and Figure 1B A system architecture 100 of an example data center carbon footprint reduction (DC-CFR) framework according to implementations of the present disclosure is depicted. The system architecture 100 includes a framework 110 communicatively coupled to a physical system 160 via a network 190. As used herein, the term "framework" refers to a combination of components (e.g., a combination of microprocessors, cores of multi-core microprocessors, microcontrollers, programmable integrated circuits, programmable gate arrays, etc.), and the manner in which those components communicate with each other.

[0026] In this example, physical system 160 is provided as a physical, real-world data center simulated by framework 110. Framework 110 includes digital twin 114, which is a digital model of data center 176 configured to simulate the operation of data center 176. Digital twin 114 may be a digital recreation of physical system 160 that is implemented as a computer program stored on a memory and executed by a processor (e.g., such as a processor in conjunction with a processor). Figure 5 Executable code executed by the computer system 500 described above.

[0027] The physical system 160 may include various sensors and the Internet of Things for sensing operating conditions of the physical system 160 under which the real-world data center 176 operates. The operating conditions may be transmitted to the framework 110 as input data through the network 190. When the operating conditions are sensed at the physical system 160, the input data may be supplied to the framework 110 in real time. The framework 110 may receive the input data and inject the input data as a state into the digital twin 114. By applying the state to the digital twin 114, the operation of the physical system 160 may be simulated under the operating conditions represented by the input data.

[0028] The framework 110 may be configured to execute a machine learning (ML) model based on the simulation to make decisions on actions to transform the operating state of the framework 110 that optimize the framework 110 for multiple goals of carbon footprint reduction, energy consumption, and energy cost. The determined actions to optimize the multiple goals may be transmitted to the framework 110 over the network 190 as instructions for controlling the data center 176 according to the determined actions. As a result, the physical system 160 may then be controlled in a manner that optimizes the data center 176 to achieve multiple goals of carbon footprint reduction, energy consumption, and energy cost across the physical system 160.

[0029] As mentioned above, the digital twin 114 is a model that can simulate the operation of the entire data center 176. The digital twin 114 includes multiple subsystem digital twins or models, each of which simulates the operation of a discrete subsystem of the data center 176. The subsystem models can be executed by applying states and transitioning the model to the next state and action. Each model simulates the operation (e.g., action) of the corresponding subsystem according to the state. A state can refer to a set of information that defines the operating conditions to be simulated at a given point in time. The state can be based on information obtained from the physical system 160. In this example, the digital twin 114 includes a cooling subsystem model 120, an energy storage subsystem model 130, and a load (e.g., workload) transfer model subsystem model 132.

[0030] The cooling subsystem model 120 may mimic or otherwise simulate data center temperature regulation through, for example, an HVAC system and temperature monitoring through a thermostat 122. That is, the cooling subsystem model 120 may include an HVAC system model that may be simulated to control the temperature of the digital twin 114 through monitoring of the simulated temperature through the thermostat 122. In this case, the actions input into the cooling subsystem model 120 may include a temperature set point for the data center that is set within the model and used to control the operation of the HVAC system model.

[0031] In some embodiments, Figure 1A As shown, the cooling subsystem model 120 may include one or more partitions 124a-124n that may represent different subsets of a data center. For example, an enterprise may operate multiple discrete data centers distributed across various regions. Different regions experience different weather conditions and other changes in operating conditions that affect the temperature regulation of each data center. In order to address such a configuration, the cooling subsystem model 120 may include multiple separate discrete partitions 124a-124n, each of which is configured to model the cooling subsystem for that partition. Each partition 124a-124n accordingly includes a corresponding HVAC system model 126a-126n and a data center model 128a-128n for modeling each partition separately. The thermostat 122 operates to monitor the simulated temperature of each partition according to a temperature set point. The temperature set point may be defined before the simulation step. Although the examples provided herein are for partition differences, other divisions of the data center are also within the scope of the present disclosure.

[0032] The energy storage subsystem model 130 may mimic or otherwise simulate energy storage performed by the data center at the battery subsystem. For example, the energy storage subsystem model 130 may include a UPS system model that may be simulated for energy storage and discharge from the battery. The energy storage subsystem model 130 may simulate charging of the battery subsystem for energy storage and discharging for energy use based on a simulation of the data center.

[0033] The load transfer model subsystem model 132 can mimic or otherwise simulate the transfer of flexible workloads performed during the simulation of the data center. For example, operations performed by the data center may require the utilization of computing resources that load the data center. The load transfer model subsystem model 132 can be configured to simulate the transfer of workloads between available resources for completing tasks. Certain workloads can be considered flexible loads, which can be allowed to be shifted to optimize other aspects of the data center based on operating conditions (such as CI and weather forecasts). For example, it may be optimal to shift flexible workloads to low CI periods and / or low weather forecasts, which may reduce carbon emissions performed by the data center.

[0034] The framework 110 also includes a control agent 116 configured for instructions to control the digital twin 114 in the form of actions. The control agent 116 includes a plurality of RL agents 134-138, each corresponding to a subsystem of the data center. Each RL agent 134-138 can be provided as a machine learning algorithm that receives a state 118 in the form of input data from a physical system 160, such as, but not limited to, weather information (e.g., weather forecasts); power grids, energy and carbon distributions, computed workloads, and other operating conditions. Each RL agent 134-138 can also receive a state of a simulation in the form of state information 150 (also referred to as observations) from the digital twin 114, which represents the current state of the subsystem model (e.g., simulated operating conditions).

[0035] The machine learning algorithm of each RL agent 134-138 can be a reinforcement learning algorithm that trains each respective RL agent according to a corresponding policy configured to optimize an individually customized collaboration reward (as will be described in more detail below). The individually customized collaboration reward can represent the effectiveness of an action based on the input state 118, which can be used to inform the policy for determining future actions. The policy applied by each RL agent attempts to optimize its own associated subsystem model while considering optimal system-wide operation.

[0036] Each subsystem model computes and outputs a corresponding reward that is included in reward information 148 provided to control agent 116. The individual corresponding rewards may be aggregated and weighted in a manner customized for each subsystem individually so that the RL agents 134-138 may determine an action that is optimal for both its own subsystem model and the system as a whole.

[0037] exist Figure 1A In the example of FIG. 1 , the control agent 116 includes an energy consumption RL agent 134, a load transfer RL agent 136, and an energy storage RL agent 138. The energy consumption RL agent 134 is configured to optimize the energy consumption of the real-world cooling subsystem by interacting with the cooling subsystem model 120. The load transfer RL agent 136 is configured to optimize the workload transfer of the real-world load transfer subsystem by interacting with the load transfer model subsystem model 132. The energy storage RL agent 138 is configured to optimize the energy storage of the real-world battery subsystem by interacting with the energy storage subsystem model 130.

[0038] Data centers, such as data center 176, may represent a significant portion of global energy consumption, and efforts to curb carbon emissions have intensified. One approach is carbon-aware workload scheduling (CAS), which takes advantage of flexible workloads and reduces carbon emissions by rescheduling flexible workloads to periods of low CI. Although not all workloads are flexible (e.g., latency tolerant), there are some applications (such as data processing) that can defer a significant portion of the workload.

[0039] Control agents 116, such as, for example, load shifting RL agents 136, can be used to optimize workload scheduling in a data center. By training the load shifting RL agent 136 to make intelligent decisions about when to allocate workloads, a reduction in energy consumption and an increase in thermal efficiency can be achieved. The framework 110 can be implemented to simultaneously manage workload scheduling through the load shifting RL agent 136 and manage the cooling system through the energy consumption RL agent 134 to improve overall efficiency. The control agent 116 can utilize an event-driven RL approach that reduces energy consumption by performing workload scheduling and cooling operations after specific events (such as rack overheating). The control agent 116 can also be utilized to predict thermal dynamics and reduce energy consumption through workload shifting. In addition, the control agent 116 can be implemented to optimize the above while also addressing the source of energy and optimization for carbon emissions.

[0040] Framework 110 also includes RL interface 112, which provides a plurality of wrappers for interfacing with each of RL agents 134-138. Figure 1AIn the example of , the RL interface 112 includes an energy consumption wrapper 140, which may be provided to encapsulate the cooling subsystem model 120 and convert data into a format that can be applied to and processed by the cooling subsystem model 120. For example, the energy consumption wrapper 140 may be provided to convert the action 146 into a format executable by the cooling subsystem model 120. The RL interface 112 also includes a load transfer model wrapper 142 and an energy storage model wrapper 144, which may be provided to encapsulate the load transfer model subsystem model 132 and the energy storage subsystem model 130, respectively, and convert data into a format that can be applied to and processed by each subsystem model. The RL interface 112 may also include an RL agent wrapper 152, which is provided to encapsulate the control agent 116 and convert data (e.g., state 118, reward information 148, and state information 150) into a format that can be processed by the RL agent.

[0041] refer to Figure 1B , physical system 160 depicts an example schematic block diagram of a real-world data center 176 and the environment in which the data center 176 operates. For example, the data center 176 can receive energy from a power grid 162 via infrastructure 164. The power grid 162 includes multiple energy sources, such as fossil fuel sources, renewable energy sources, etc. Figure 1B In the example shown, the power grid 162 includes non-renewable energy sources such as, but not limited to, coal, natural gas, oil, nuclear energy, to name a few. The power grid 162 also includes renewable energy sources such as, but not limited to, solar, hydroelectric, clean nuclear energy, wind, hydroelectric, to name a few.

[0042] The power grid 162 also includes one or more sensors 168 configured to monitor energy levels on the grid 162 from various energy sources and determine a measure of CI based on the energy levels on the power grid 162. In one example, the CI can be measured as the average CO2 emissions generated by the power grid 162 per unit of electricity consumed from the grid 162 by energy consumers including the data center 176. In an example, the CI measurement is an external factor that can be supplied as input data to the framework 110. In some examples, the sensor 168 can be, but is not limited to, an energy meter that uses energy information to estimate the grid CI.

[0043] The physical system 160 may also include a weather station sensor 166 or environmental monitoring device configured to measure and quantify weather data to determine both current and predicted weather forecasts. Examples of environmental monitoring devices 166 include, but are not limited to, piezoelectric sensors and / or diaphragm sensors for detecting atmospheric pressure; temperature sensors for measuring temperature (e.g., thermocouples, resistance temperature detectors (RTDs), thermistors, etc.); humidity sensors for measuring ambient humidity (e.g., hygrometers, etc.); wind speed and direction sensors (e.g., anemometers, wind vanes, etc.); rainfall sensors (e.g., rain gauges, etc.). In an example, weather data may be an external factor that may be supplied to the framework 110 as input data.

[0044] Datacenter 176 receives workload data from workload request 174. A workload may define a task and request datacenter resources for completing the task. In an example, a workload may be measured as a CPU utilization percentage (CPU%) determined based on a percentage required for the workload to be executed. As suggested above, a workload may be a flexible load or a non-flexible load.

[0045] As described above, the framework 110 can optimize policies that can be provided to the data center 176 as optimization policies for controlling the operating conditions of the data center 176. Figure 1B As shown, the framework 110 can supply control instructions including optimized strategies to the subsystem 170. The control instructions can be provided as signals transmitted via the network 190. As described above, the subsystem 170 can include multiple subsystems, such as but not limited to a cooling subsystem 182 (e.g., an HVAC system), a load transfer subsystem 184, and a battery subsystem 186. The cooling subsystem 182 can set a temperature set point according to the control instructions from the framework 110 and control the operation of the HVAC system so as not to exceed the temperature set point. The load transfer subsystem 184 can receive control instructions to optimize the current workload distribution to manage computing resources. The battery subsystem 186 can receive control instructions from the framework 110 and charge and / or discharge the battery according to the instructions.

[0046] Data center 176 outputs performance metrics 180 based on the operating conditions of subsystem 170. Output performance metrics 180 may include, but are not limited to, carbon footprint, energy consumption (e.g., actual energy consumed measured in terms of BTU, kwh, etc.), and energy cost (e.g., the product of energy consumption multiplied by a value to determine the cost per unit of energy, specified as unit cost / kwh or $ / kwh) based on the current operating conditions. Output performance metrics 180 are fed back to framework 110 as input data, which may define states for further refining the RL training of control agent 116.

[0047] The network 190 may be a wired or wireless network that provides a communication exchange of information and data between the framework 110 and the physical system 160. The network 190 may also be a public or private network, such as the Internet or other communication network that allows connectivity between the framework 110 and the physical system 160. The network 190 may include third-party telecommunication lines, such as telephone lines, broadcast coaxial cables, fiber optic cables, satellite communications, cellular communications, etc. The network 190 may include any number of intermediate network devices, such as switches, routers, gateways, servers, and / or controllers. The network 190 may include various servers or other computer systems, such as Figure 5 A computer system 500.

[0048] Figure 2 yes Figure 1A Schematic diagram of interdependencies and information exchange between RL agents 134-138. Figure 2 Shows about Figure 1A The internal and external dependencies of the actions determined by the RL agents 134-138 are shown in FIG. 1 . Arrows pointing into a given RL agent represent external or internal dependencies that act as input states that influence the actions determined by the given RL agent. Arrows leaving a given RL agent represent actions determined by the given RL agent, such as Figure 2 As shown, it can serve as a representation of the input state for another RL agent when making its own action determination.

[0049] As mentioned above, Figure 2 As shown, due to the complex interdependencies and balancing objectives of competing RL agents, traditional carbon reduction methods lack real-time operation capabilities and cannot effectively combine multiple control strategies. The technology disclosed in this article partially decouples carbon footprint reduction into sub-problems. Each RL agent solves an individual Markov decision process (MDP) formula while aggregating individual rewards and overlapping state variables in a multi-agent setting to solve complex dependencies in real time. This design can provide a comprehensive, real-time carbon footprint optimizer for a sustainable society, surpassing existing strategies to provide adaptable and robust solutions.

[0050] MDP is a mathematical framework that provides a formal way to model decision making in a dynamic environment, such as framework 110, where the actions of an RL agent affect the state of the environment, and the state of the environment determines the reward and subsequent state. Formally, an MDP can be defined as a tuple (S, A, P, R, γ). S is a state space containing all possible states of the environment, which can be viewed as framework 110 in the disclosed technology. A is an action space, which contains all possible actions that a given agent can provide to the environment to transition between states. P is a transition probability function, which defines the probability of moving from one state to another given an action. R is a reward function, which calculates the reward that the agent receives for taking an action in a given state. γ is a discount factor, which determines the importance of future rewards relative to immediate rewards.

[0051] At each time step t, the agent observes the current state s t ∈S, and based on the strategy π(a t |s t )Select action a t ∈A. The action changes the state of the environment (e.g., transitions), and the agent responds based on the transition probability P(s t +1|s t ,a t ) receives reward r from the environment t =R(s t ,a t ). Where s t +1 indicates that the RL agent is in state s t Action taken when a t The next state of the environment. In some examples, s t +1 can be represented as s t '.

[0052] To solve the MDP, RL algorithms can be used to optimize the agent’s policy π(a t |s t ). The policy is a function that maps states to actions and determines the agent's behavior in the environment. During the training phase, the agent interacts with the environment, observing the current state and choosing actions to optimize the reward. The agent's actions affect the state of the environment, which in turn affects the reward and subsequent states. The agent learns from the feedback provided by the environment in the form of rewards and adjusts its policy accordingly.

[0053] While the above discussion of the general MDP approach provides a single agent approach, the implementation disclosed herein formulates the energy and carbon footprint reduction problem for data centers using MDPs by considering three interdependent MDPs that account for workload shifting, energy consumption through cooling optimization, and supplemental energy supply using energy storage systems, respectively. To achieve this, three MDPs are constructed, as described in Table 1 below, one MDP for each RL agent 134-138, having the above Figure 2 The interdependencies summarized in: (1) Load transfer MDP (MDP LS ), which can be solved by shifting data center workloads to low-carbon hours and reducing energy consumption to reduce carbon footprint; (2) energy consumption MDP (MDP E ), which can be solved by optimizing cooling set points to reduce data center energy usage; and (3) energy storage MDP (MDP BAT ), which can be solved by charging and discharging batteries based on the grid carbon intensity to reduce the carbon footprint. The energy consumption RL agent 134 can be used to solve the MDP E The role of load transfer RL agent 136 can solve the MDP LS and the energy storage RL agent 138 can play a role in solving the MDP BAT role.

[0054] MDP E The state space S may include the physical and thermal operating conditions of a data center (e.g., data center 176), represented by s t ∈S. These may include external factors such as, but not limited to, relative outdoor temperature, indoor temperature, cooling set points, HVAC energy consumption, IT energy (ITE) consumption, total energy consumption, and day of year and time of day. For example, external factors may be obtained from the physical system 160 via various sensors and monitoring devices, as described above in conjunction with Figure 1A-1B The state space S may also include states simulated by the cooling subsystem model 120, which may depend on Figure 2 The action space A can be provided as a discrete set of incremental actions representing changes in the cooling set point of the cooling subsystem, represented by a t ∈A. Example incremental actions may include, but are not limited to, {-5, -2, -1, -0.5, 0, 0.5, 1, 2, 5} provided in degrees Fahrenheit, Celsius, Kelvin, etc. The conversion function P(s t+1 |s t ,a t) can be modeled by learning the physical and thermal properties of the data center, the cooling subsystem, and the external weather forecast. The set point can be simulated by the cooling subsystem model 120 and represents the action a t Then from state s t Transition to state s t+1 probability.

[0055] Reward function r(s t ,a t ) can motivate the energy consumption RL agent 134 to minimize the total cost of energy consumption and electricity by multiplying the negative normalized total power consumption by the electricity price. This reward function can be referred to as the energy usage reward (r E ). It is defined as r(s t ,a t )=-c(s t )·e(s t ,a t ), where c(s t ) is the electricity cost, and e(s t ,a t ) is taking action a t After the state t In other words, r E = -(TotalEnergyConsumpt x CostperkWh). In some examples, the discount factor (γ) may be set to 0.99, meaning that future rewards are considered important. The goal of the energy consumption RL agent 134 is to learn a policy π(a t |s t ). To optimize the agent's policy, the Proximal Policy Optimization (PPO) algorithm etc. can be used.

[0056] During training, the energy consumption RL agent 134 is calculated by observing the current state s t and select action a t Interact with a simulated environment (e.g., cooling subsystem model 120) to optimize energy consumption and reduce cost. Actions taken by the energy-consuming RL agent 134 can affect the state of the environment, which in turn affects rewards and subsequent states. The agent learns from the feedback provided by the cooling subsystem model 120 and adjusts its policy and value function accordingly.

[0057] For MDP LS, the disclosed technique formulates the problem as a sequential MDP where the load shifting RL agent 136 has access to limited information. This is in contrast to traditional approaches that rely on the assumption of having access to carbon intensity and workload information 24 hours in advance and utilize a static greedy algorithm for workload allocation. In this case, the load shifting RL agent 136 has access to N hours of future carbon forecasts. The load shifting RL agent 136 is given a certain amount of flexible workload at the beginning of each day and chooses the optimal time slot before the end of the day to assign the workload. Data center capacity may be limited, so workloads cannot be assigned in all time slots.

[0058] As an example, a carbon-aware workload optimized MDP LS can be defined as follows. The state space S can be provided as S:{DC t ,LoadLeft t ,CI t ,...,CI t+N}, where DC t is a real-world data center workload (eg, obtained from workload request 174), LoadLeft t is the amount of flexible workload assigned to the data center resources within the time limit N, and CI is the current grid carbon intensity and its forecast until time step t+N (e.g., obtained from one or more sensors 168). The action space A can be considered as a binary discrete action space, where a t ∈{0,1}, about which the load transfer RL agent 136 makes a decision to assign a certain amount of workload to the data center at the current time step t or keep it idle, which workload cannot exceed the maximum capacity.

[0059] As an example, the reward function can be defined as r t =-(r carbon,t +r penalty ), where r carbon,t is the net carbon footprint of the data center energy consumption at time step t (e.g., the CO2 footprint at time step t) and r penalty is the load shift penalty at the last time step of the day if the agent has not been assigned a predefined workload in a given time frame, and is 0 otherwise. The load shift penalty (or LS Penalty ) may be a scalar value of the unassigned flexible workload, which is selected to cause the assignment of the unassigned flexible workload. This reward function may be referred to as a load transfer reward function (r LS ), which can be described as r LS =-(CO2Footprint+LS Penalty). To obtain the net carbon footprint for time step t, the net energy drawn from the grid (ng t ) can be calculated based on the control decision and then multiplied by the corresponding carbon intensity of the grid at time step t (e.g., CI t ). The conversion function p(s t ,a t ,s t+1 ) can be based on a deterministic model. A base workload can be provided that is inflexible at each time step. The base workload can be increased by assigning new workloads from the flexible budget at each time step t until the flexible workload budget is empty or the data center is at maximum capacity. In an example implementation, the workload scheduling time frame can be limited to a 24-hour window with a forecast horizon of N=4.

[0060] To optimize battery performance, MDP BAT can be provided as follows. The state space can be provided as S: {Load dc ,Battery SOC ,CI t ,...,CI t+N}, where Load dc is based on the E (e.g., energy consumption RL agent 134) and the actions taken by agent A LS (e.g., load shifting RL agent 136) the instantaneous data center load of the load shifted, Battery SOC is the battery state of charge (SoC) and CI is the current CI and its forecast up to time step t+N. Here, N is a parameter that can be chosen depending on the reliability of the forecast model. In an example implementation, a conservative estimate of N=4 hours can be utilized. The action space A can be provided as {i t ,...,i t+N}, where for each i of τ∈(t,t+N) τ The possible actions at each of these time steps are to charge, discharge, or keep the battery idle. Thus, at each time step τ, the energy storage RL agent 138 will plan the next N steps. However, when sending the actions to the actual battery system included in the data center 176, the action at time step τ=t may be used.

[0061] The reward might be based on the net carbon footprint of the above sequence of actions of length N used for the battery over N steps. Thus, at each time step, the net energy (ng τ , τ∈(t,t+N)) can be calculated based on the action sequence and multiplied by the corresponding grid carbon intensity CI at those moments t ,...,CIt+N For example, the reward (r t ) can be calculated based on the vector of net energy drawn from the grid and the vector of carbon intensity on the grid (CI τ ) (e.g., This reward function can be called the energy storage reward function (r BAT ), which can be described as r BAT = -(CO2Footprint). The action space can be implemented as a multi-discrete space in the environment. As is known in the art, the transformation function p(s t ,a t ,s t+1 ) is based on a linear battery degradation model.

[0062] Additionally, according to some examples, actual charge and discharge rate limits may be imposed to make battery operation more realistic. This may be accomplished by deriving the charge and discharge rates from sigmoid curves for the battery storage subsystem. These curves generate linear charge / discharge rates at nominal battery SoC. The charge rate is highest at low SoC and vice versa. Similarly, the discharge rate may be lowest at low SoC and vice versa. For the data center demand curve, the instantaneous data center demand may be used, which may be based on the proxy A E (eg, energy consumption RL agent 134). The carbon intensity value may be estimated based on the instantaneous availability of different energy sources in the grid.

[0063] Table 1 below summarizes the above MDP formula.

[0064] Table 1

[0065]

[0066] As can be seen from the MDP description in Table 1, there is a dependency chain from the load transfer RL agent 136, through the energy consumption RL agent 134, and then through the energy storage RL agent 138. The energy consumption RL agent 134 decides whether to transfer the load based on the current workload, the grid CI, the current power consumption of the data center, and the current thermal state of the data center. The resulting workload is used by the energy consumption RL agent 134 along with other variables (such as weather forecast and current battery SoC) to optimize the energy operation for a given temperature set point. The cooling subsystem model 120 is then used to estimate the energy consumption in the next interval. This information, along with the grid CI, is considered by the energy storage RL agent 138 to use the grid to charge, discharge (or keep idle) the battery, supply auxiliary power to reduce grid dependence.

[0067] The multiple steps and circular dependencies in the above problem formulation create opportunities for increasing energy and carbon footprint savings, while having some challenges regarding training convergence for RL agents. Similar to multi-agent reinforcement learning (MARL), a collaborative reward framework can be adopted to help RL agents assign common state variables obtained from other MDPs to the agent's reward. The collaborative reward framework (an example is provided in Table 1) aggregates the reward functions of individual MDPs into a collaborative reward function customized for each MDP. For example, for a given MDP formulation, the reward function is the sum of the individual reward functions of each MDP formula, where each individual reward function is weighted according to a preset weight. The weight applied to the individual reward function corresponding to the given MDP formula is greater than the weight applied to other reward functions. For example, as shown in Table 1, for MDP LS , is applied to r LS The weight is 0.8 and is applied to r BAT and r E The weight of is 0.1. Similar weights are used for MDP E and MDP BAT However, these are just illustrative examples. Any weights desired can be applied so that the collaboration reward is w1*r LS +w2*r E +w3*r BAT , where w1+w2+w3=1.

[0068] Furthermore, the lower-weighted reward functions do not need to have the same weight. For example, in an MDP LS In the case of LS +w2*r E +w3*r BAT , where w1 is greater than w2 and w3. In this case, w2 and w3 can be the same value or different values, and w2 can be greater than or less than w3. As an example, for MDP LS The collaboration reward can be 0.7*r LS +.1*r E +.2*r BAT The weights for other MDPs can be implemented according to similar principles.

[0069] Figure 3 is a schematic block diagram of an example process 300 of RL according to an implementation of the present disclosure. Process 300 illustrates a generalized multi-agent reinforcement learning algorithm (MARL) process performed by framework 110 using the above MDP formulation: MDP LS 、MDP E and MDP BATThe goal of process 300 is for RL agents 134 - 136 to learn a strategy for optimally reducing the overall carbon footprint of a data center (e.g., data center 176 ) in real time by solving multiple MDPs in digital twin 114 simultaneously.

[0070] Process 300 provides an example of reinforcement learning performed by control agent 116 to learn an optimal policy for maximizing an individually customized collaborative reward function based on discrete rewards received from digital twin 114. For example, control agent 116 includes RL agents 134-138 that may be provided as ML algorithms, which are accessed via RL agent wrapper 152 ( Figure 3 14 (not shown) interacts with the corresponding subsystem model of the digital twin 114 in discrete time steps. At each time t, each RL agent receives current state information 150 and reward information 148 from the corresponding subsystem model of the digital twin 114. The state information 150 may also include external information obtained from the real-world environment, such as weather forecasts, CI forecasts, and workload forecasts.

[0071] Each RL agent 134-138 selects an action 146 from the set of available actions, which is sent to the corresponding subsystem model of the digital twin 114. The digital twin 114 moves to a new state (e.g., each subsystem transitions to a new state based on the received action) and determines updated reward information associated with the transition. For example, the RL agents 134-138 compute the reward information that is sent to the cooling subsystem model 120, the energy storage subsystem model 130, and the load transfer model subsystem model 132 accordingly. Figure 3 ), which causes each subsystem model to transition to a new state and calculate a discrete reward (e.g., r LS 、r E and r BAT ). The updated state and reward information is then provided back to the RL agents 134-138, each of which computes an individually customized collaborative reward as described above and updates its policy. The RL agents 134-138 then compute the next action for the next time step t+N. The goal of the RL agents 134-138 is to learn a policy that maximizes its corresponding collaborative reward function.

[0072] In more detail, a system implementation of multiple agents can be performed that takes into account Figure 2 The interdependencies between each agent action and the next state are depicted in FIG. The load transfer RL agent 136, the energy consumption RL agent 134, and the energy storage RL agent 138 can be initialized simultaneously and used accordingly for optimal load transfer (MDP LS ), Optimize Energy (MDP E ) and optimal battery operation (MDP BAT). At this point, the digital twin 114 can be initialized, including the cooling subsystem model 120, the energy storage subsystem model 130, and the load transfer model subsystem 132. State variables (such as data center temperature, total IT workload, unassigned flexible load, data center energy usage, and battery SoC information) can be obtained from the digital twin 114 and / or the data center 176 and stored as state information 150 to the database 314. The state information can be derived by the cooling subsystem model 120, the energy storage subsystem model 130, and the load transfer model subsystem model 132 and provided to the database 314 for storage. Grid data (such as but not limited to carbon intensity, weather conditions, and data center workload data) can also be stored in the database 314 as state information 150 for each moment. The state information 150 can therefore be shared between the RL agents 134-138. In real time, each RL agent 134-138 interacts with its corresponding MDP to collect rollout information (S t , A t , S t '、R t+1 ,γ,done). Note that S t 'Can be with S t+1 Interchange, as described above. "Done" represents a Boolean flag used in RL to indicate the end of an epoch, which can be the duration of a pushout and can be determined by the environment / MDP. Examples of pushout information are provided in Table 1 above. As used herein, "interaction" may refer to an RL agent performing an action (not solving) on ​​an MDP to obtain pushout information as output. The term "solving" may have a separate meaning in RL parlance, where in the RL methods used herein, an RL agent may not solve the MDP directly, but rather indirectly by interacting with the MDP and forming an internal representation of the MDP.

[0073] In operation, the load shifting RL agent 136 queries the state information 150 for time and grid carbon intensity. The load shifting RL agent 136 receives the MDP LS Get unassigned flexible load; Energy consumption RL agent 134 from MDP E The data center temperature, current IT load, and data center energy usage are obtained; and the energy storage RL agent 138 obtains the MDP BAT The battery SoC information is obtained. The load transfer RL agent 136 uses this information to decide its action on whether to reassign the flexible load from this time step to another time step or keep it idle. The resulting IT load information is passed to the energy consumption RL agent 134. The energy consumption RL agent 134 uses the time and the MDP EThe data center temperature, data center energy consumption, and temperature set points obtained at the previous time step are used to determine the set point for the current time step. The cooling subsystem model 120 of the digital twin 114 calculates the change in energy consumption and data center temperature, and the cooling subsystem model 120 transmits the change back to the MDP E The energy storage RL agent 138 obtains time, E The data center energy usage, current battery charge, and carbon intensity information are used to decide whether to charge the battery from the grid, discharge the battery to supplement the energy needs of the digital twin 114, or keep it idle.

[0074] In the example, RL agents 134-138 each determine their respective actions and issue these actions to the environment at the same time step. However, these actions may be performed on the environment according to a sequence. In an example implementation, the sequence may be a load transfer subsystem, a cooling subsystem, and a battery storage subsystem. In another example, the sequence may be different depending on the desired application.

[0075] In another example implementation, at each rollout step (e.g., each time step), the RL agent can complete one step of interacting with its corresponding MDP, effectively providing asynchronous concurrent rollout. From an implementation perspective according to this example, individual RL agents do not need to receive a rollout information tuple immediately after taking an action. They may wait until it is their turn to determine and issue an action again. This waiting can allow the effects of actions previously taken by other RL agents to be incorporated into all MDPs, which can be used to inform each RL agent's individually customized collaborative rewards. That is, at each time step, each RL agent can interact with its corresponding MDP to calculate an individually customized collaborative reward, which can be used to inform the action for the next time step. Therefore, the individually customized collaborative reward takes into account the impact of actions from all RL agents. This hierarchical formula is also shown in Table 1, which enables RL agents to consider the impact of their individual actions across interdependent MDPs.

[0076] At regular intervals, the data stored in the storage buffers 334, 336, and 338 can be used to update the policy of each corresponding agent. In an example implementation, the policy can be updated using the PPO algorithm to learn a decentralized policy and a decentralized critic (value function). In this case, each RL agent interacts through its individually customized collaborative reward function and shared state information, but is unaware of the actions taken by other agents. However, any other heterogeneous policy or policy gradient algorithm can be used. For example, some implementations can use a multi-agent deep deterministic policy gradient (MADDPG) algorithm, which extends the deep deterministic policy gradient (DDPG) algorithm to a multi-agent policy gradient algorithm, in which decentralized agents learn centralized critics based on the observations and actions of all agents. Here, a critic refers to a part of a class of RL algorithms that includes an actor and a critic. The actor (or policy) determines the action, and the critic provides a measurement of the goodness of the action (e.g., a reward) to inform the policy. In this example case, the learned policy uses the local information of each corresponding actor during test time, rather than the information of other actors. However, each agent can use the observations and actions of each of the three agents.

[0077] An example algorithm for training RL agents 134-138 is provided below in pseudocode:

[0078]

[0079]

[0080] Figure 4 Example computing components that can be used to achieve a reduction in the carbon footprint of a system according to various embodiments are shown. Figure 4 , computing component 400 may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 4 In the example implementation of , computing component 400 includes a hardware processor 402 and a machine-readable storage medium 404 .

[0081] Hardware processor 402 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for fetching and executing instructions stored in machine-readable storage medium 404. Hardware processor 402 may fetch, decode, and execute instructions, such as instructions 406-412, to control a process or operation for carbon footprint reduction. As an alternative or in addition to fetching and executing instructions, hardware processor 402 may include one or more electronic circuits including electronic components for performing the functions of one or more instructions, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other electronic circuits.

[0082] A machine-readable storage medium, such as machine-readable storage medium 404, may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, machine-readable storage medium 404 may be, for example, a random access memory (RAM), a non-volatile RAM (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a storage device, an optical disk, or the like. In some embodiments, machine-readable storage medium 404 may be a non-transitory storage medium, wherein the term "non-transitory" does not encompass transient propagating signals. As described in detail below, machine-readable storage medium 404 may be encoded with executable instructions, such as instructions 406-412.

[0083] Hardware processor 402 may execute instructions 406 to obtain state data representing a state of the system. The system includes a plurality of subsystems.

[0084] For example, the system may be a data center including a cooling subsystem, a load transfer subsystem, and an energy storage subsystem. The state data may be obtained from the physical data center as an operating condition, such as from a sensor monitoring the operating condition of the data center. According to some examples, the state data may include an internal state of the system and an external state for the environment of the system. The internal state may include one or more of an allocated workload, an unallocated workload, a temperature of the system, an energy consumption of the system, and energy stored by the system. The external state may include one or more of a grid carbon intensity, weather, and a time of day.

[0085] Hardware processor 402 may execute instructions 408 to receive, by a plurality of RL agents, reward data from a digital twin of a system simulating operation of the system. The reward data includes a plurality of rewards, each reward being associated with a subsystem of the plurality of subsystems.

[0086] For example, a digital twin could be a simulation of a data center, such as the one above combined with Figure 1A and Figure 1B As described. The digital twin may receive state data that configures the digital twin to simulate operating conditions of a current state of a data center. The digital twin may also receive action data that simulates transitions between states of the digital twin. Based on the transitions, the digital twin may generate reward data that represents the effectiveness of the action. The digital twin may provide reward data to multiple RL agents for training the RL agents.

[0087] Each subsystem of the digital twin can generate a corresponding reward, which can be included in the reward data provided to each of the multiple RL agents. For example, a first reward corresponding to the cooling subsystem in the multiple rewards can be based on a simulation of energy consumed by the cooling subsystem model (e.g., a digital twin of the cooling subsystem) and a simulation of energy costs. A second reward corresponding to the load transfer subsystem model in the multiple rewards can be based on a carbon footprint generated by a simulation of the load transfer subsystem model (e.g., a digital twin of the load transfer subsystem) and unallocated flexible workloads. A third reward corresponding to the energy storage subsystem in the multiple rewards can be based on a simulation of the carbon footprint of the energy storage subsystem by the energy storage subsystem model (e.g., a digital twin of the energy storage subsystem).

[0088] The hardware processor 402 may execute instructions 410 to determine a plurality of actions performed by the plurality of RL agents based on the state data and each of the plurality of rewards. Each of the plurality of RL agents may be associated with a subsystem of the plurality of subsystems, and a weight is assigned to a reward of the plurality of rewards corresponding to the associated subsystem that is greater than a weight assigned to rewards corresponding to other subsystems. Thus, individual rewards from each subsystem model may be aggregated into a collaborative reward that is individually customized for each respective RL agent. The individually customized collaborative rewards may represent the effectiveness of the actions taken by each RL agent not only for the corresponding system, but also for the entire system by taking into account rewards associated with other subsystems.

[0089] As above combined Figure 1A and Figure 1B As described, the plurality of RL agents may include energy consumption RL agents, load transfer RL agents, and energy storage RL agents corresponding to the cooling subsystem, the load transfer subsystem, and the energy storage subsystem. The various subsystems are interdependent with each other, so the RL agent is configured to determine actions depending on both previous rewards and actions of other subsystems.

[0090] In some examples, the action determined by the reinforcement learning agent associated with the cooling subsystem may be a cooling set point for the cooling subsystem. The action determined by the reinforcement learning agent associated with the load transfer subsystem may be a distribution of workload. The action determined by the reinforcement learning agent associated with the energy storage subsystem may be an amount to charge or discharge the energy storage subsystem.

[0091] In the example, the reinforcement learning agents determine their corresponding actions and issue these actions at the same time step. These actions can be performed on the environment in the order of the load transfer subsystem, the cooling subsystem, and the battery storage subsystem.

[0092] The hardware processor 402 can execute instructions 412 to convert the system to an updated state based on multiple actions. For example, control instructions can be generated by an RL agent, which control instructions represent the optimal strategy learned from RL based on individually customized collaborative rewards. The control instructions can be transmitted to a system (e.g., a data center) for configuring the operating conditions of the system to convert the system to an updated state that optimizes the carbon footprint of the system based on the current internal and external states. Therefore, flexible workloads can be allocated based on the grid CI and the battery storage system SoC, as well as cooling strategies based on upcoming weather. In addition, the cooling subsystem can be controlled to adjust the temperature based on the allocated workload (both flexible and non-flexible), and taking into account the current grid CI and battery SoC. In addition, the battery storage system can be optimized to charge the battery during low grid CI and discharge the battery during high grid CI depending on the power consumption requirements of the data center.

[0093] Figure 5 A block diagram of an example computer system 500 is depicted in which various embodiments described herein may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, one or more hardware processors 504 coupled with bus 502 for processing information. Hardware processor(s) 504 may be, for example, one or more general purpose microprocessors.

[0094] The computer system 500 also includes a main memory 506, such as a random access memory (RAM), a cache, and / or other dynamic storage device, coupled to the bus 502 for storing information and instructions to be executed by the processor 504. The main memory 506 can also be used to store temporary variables or other intermediate information during the execution of instructions executed by the processor 504. Such instructions, when stored in a storage medium accessible to the processor 504, present the computer system 500 as a special-purpose machine that is customized to perform the operations specified in the instructions. The main memory 506 can store information containing machine-readable instructions for providing the functionality of the framework 110, such as the digital twin 114 and the subsystem models therein, the control agent 116, and other components of the framework 110.

[0095] Computer system 500 also includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to bus 502 for storing information and instructions.

[0096] The computer system 500 may be coupled to a display 512, such as a liquid crystal display (LCD) (or touch screen), via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor controller 516, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. In some embodiments, the same direction information and command selections as cursor control may be achieved via receiving touches on a touch screen without a cursor.

[0097] In some implementations, the input device 514 may include a port or other input interface for receiving input data from a real-world system. Thus, the input device 514 may receive input data including internal states and external factors, such as those described above in conjunction with Figure 1A and Figure 1B as described.

[0098] The computing system 500 may include a user interface module for implementing a GUI, which may be stored in a mass storage device as executable software code executed by the computing device(s). As examples, this module and other modules may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and variables.

[0099] In general, the terms "component", "engine", "system", "database", "data store", etc., as used herein, may refer to logic embodied in hardware or firmware, or to a collection of software instructions, possibly with entry and exit points, written in a programming language (such as, for example, Java, C or C++). Software components may be compiled and linked into executable programs, installed in dynamic link libraries, or may be written in interpreted programming languages ​​(such as, for example, BASIC, Perl or Python). It will be understood that software components may be called from other components or from themselves, and / or may be called in response to detected events or interrupts. Software components configured for execution on a computing device may be provided on a computer-readable medium, such as a compact disc, digital video disc, flash drive, disk, or any other tangible medium, or as a digital download (and may initially be stored in a compressed or installable format that requires installation, decompression or decryption prior to execution). Such software code may be stored in part or in whole on a memory device of an executing computing device for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It will also be understood that hardware components may include connected logic units (such as gates and flip-flops), and / or may include programmable units (such as programmable gate arrays or processors).

[0100] Computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes or programs computer system 500 into a special purpose machine. According to one embodiment, the techniques herein are performed by computer system 500 in response to processor(s) 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor(s) 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0101] As used herein, the term "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cartridges, and network versions thereof.

[0102] Non-transient media are distinct from transmission media, but may be used in conjunction with transmission media. Transmission media participate in the transfer of information between non-transient media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 502. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0103] Computer system 500 also includes a network interface 518 coupled to bus 502. Network interface 518 provides bidirectional data communication coupled to one or more network links connected to one or more local networks. For example, communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connection to a telephone line of a corresponding type. As another example, network interface 518 can be a local area network (LAN) card to provide data communication connection to a compatible LAN (or a WAN component communicating with a WAN). A wireless link can also be implemented. In any such implementation, network interface 518 sends and receives electrical, electromagnetic or optical signals carrying digital data streams representing various types of information.

[0104] Communication interface 518 may be an input interface for receiving input data from a real-world system (such as physical system 160 and sensors / IoT devices therein) via a network (e.g., network 190). Thus, network interface 518 may receive input data including internal states and external factors, as described above in conjunction with Figure 1A and 1B As described.

[0105] A network link typically provides data communications to other data devices through one or more networks. For example, a network link can provide a connection through a local network to a host computer or to data devices operated by an Internet Service Provider (ISP). The ISP, in turn, provides data communications services through a worldwide packet data communications network now commonly referred to as the "Internet." Both local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link and through the communication interface 518 are example forms of transmission media that carry digital data to and from the computer system 500.

[0106] Computer system 500 can send messages and receive data, including program code, through the network(s), network links and communication interface 518. In the Internet example, a server may transmit the requested code for an application through the Internet, an ISP, a local network and communication interface 518.

[0107] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510 or other non-volatile storage for later execution.

[0108] Each of the processes, methods, and algorithms described in the foregoing sections may be embodied in a code component executed by one or more computer systems or a computer processor including computer hardware, and fully or partially automated by the code component. One or more computer systems or computer processors may also operate to support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). These processes and algorithms may be implemented in part or in whole in a dedicated circuit system. The various features and processes described above may be used independently of each other, or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of the present disclosure, and certain methods or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith may be executed in other appropriate orders, or may be executed in parallel or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The execution of certain operations or processes may be distributed between computer systems or computer processors, not only residing in a single machine, but also deployed on multiple machines.

[0109] As used herein, circuits can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form circuits. In implementation, the various circuits described herein can be implemented as discrete circuits, or the functions and features described can be shared in part or in whole between one or more circuits. Although various features or functional elements can be individually described or required as separate circuits, these features and functions can be shared between one or more common circuits, and such descriptions should not require or imply that separate circuits are required to implement such features or functions. In the case where the circuit is implemented in its entirety or in part using software, such software can be implemented to operate together with a computing or processing system (such as computer system 500) that can perform the functions described therein.

[0110] As used herein, the term "or" may be interpreted as inclusive or exclusive. In addition, descriptions of resources, operations, or structures in the singular should not be understood to exclude the plural. Conditional language, such as "can," "could," "possibly," or "may," among others, unless otherwise specifically stated or otherwise understood in the context of use, is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include certain features, elements, and / or steps.

[0111] Unless expressly stated otherwise, the terms and phrases used in this document and variations thereof should be interpreted as open ended and not restrictive. Adjectives such as "conventional," "traditional," "normal," "standard," "known," and terms of similar meaning should not be interpreted as limiting the items described to those available in a given time period or at a given time, but should be understood to cover conventional, traditional, normal, or standard technology that may be available or known at any time now or in the future. In certain cases, the presence of broad words and phrases such as "one or more," "at least," "but not limited to," or other similar phrases should not be understood to mean that a narrower case is intended or required where such an expanded phrase may not be present.

Claims

1. A method comprising: obtaining state data representing a state of a system, the system comprising a plurality of subsystems; receiving, by a plurality of reinforcement learning agents, reward data from a digital twin of the system, the digital twin simulating operation of the system, the reward data comprising a plurality of rewards, each reward associated with a subsystem of the plurality of subsystems; determining, by the plurality of reinforcement learning agents, a plurality of actions based on the state data and each of the plurality of rewards, wherein each of the plurality of reinforcement learning agents is associated with a subsystem of the plurality of subsystems and assigns a weight to a reward of the plurality of rewards corresponding to the associated subsystem of the plurality of subsystems that is greater than weights assigned to rewards corresponding to other subsystems of the plurality of subsystems; as well as The system is transitioned to an updated state based on the plurality of actions. The method of claim 1 , wherein the system is a data center. 3 . The method of claim 1 , wherein the plurality of subsystems include a cooling subsystem, a load transfer subsystem, and an energy storage subsystem.

4. A method according to claim 3, wherein a first reward among the multiple rewards corresponding to the cooling subsystem is based on a measurement of energy consumed by the system and an energy cost, wherein a second reward among the multiple rewards corresponding to the load transfer subsystem is based on a carbon footprint of the load transfer subsystem and an unallocated workload, and wherein a third reward among the multiple rewards corresponding to the energy storage subsystem is based on a carbon footprint of the energy storage subsystem.

5. The method of claim 3, wherein an action determined by a reinforcement learning agent associated with the cooling subsystem is dependent upon an action determined by a reinforcement learning agent associated with the load transfer subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem.

6. The method of claim 3, wherein the action determined by the reinforcement learning agent associated with the cooling subsystem is a cooling set point for the system, wherein the action determined by the reinforcement learning agent associated with the load transfer subsystem is a distribution of workload, and wherein the action determined by the reinforcement learning agent associated with the energy storage subsystem is a charge or discharge amount.

7. A method according to claim 3, wherein the action determined by the reinforcement learning agent associated with the load transfer subsystem is dependent on the action determined by the reinforcement learning agent associated with the cooling subsystem and the action determined by the reinforcement learning agent associated with the energy storage subsystem.

8. A method according to claim 3, wherein the action determined by the reinforcement learning agent associated with the energy storage subsystem is dependent on the action determined by the reinforcement learning agent associated with the cooling subsystem and the action determined by the reinforcement learning agent associated with the load transfer subsystem.

9. A method according to claim 1, wherein the state data includes an internal state of the system and an environmental state for the environment of the system, wherein the internal state includes one or more of the following: allocated workload, unallocated workload, temperature of the system, energy consumption of the system, and energy stored by the system, and wherein the environmental state includes one or more of the following: grid carbon intensity, weather, and time.

10. A system for reducing the carbon footprint of a system, comprising: Memory, which stores instructions; as well as one or more processors coupled to the memory and configured to execute the instructions to: obtaining status data representing a status of the system, the system comprising a plurality of subsystems; receiving reward data from a digital twin of the system, the digital twin simulating operation of the system, the reward data comprising a plurality of rewards, each reward associated with a subsystem of the plurality of subsystems; determining, by a plurality of reinforcement learning agents, a plurality of actions based on the state data and each of the plurality of rewards, wherein each of the plurality of reinforcement learning agents is associated with a subsystem of the plurality of subsystems and assigns a weight to a reward of the plurality of rewards corresponding to the associated subsystem of the plurality of subsystems that is greater than weights assigned to rewards corresponding to other subsystems of the plurality of subsystems; as well as The system is transitioned to an updated state based on the plurality of actions.

11. The system of claim 10, wherein the system is a data center, and wherein the plurality of subsystems include a cooling subsystem, a load transfer subsystem, and an energy storage subsystem.

12. The system of claim 11, wherein an action determined by a reinforcement learning agent associated with the cooling subsystem is dependent upon an action determined by a reinforcement learning agent associated with the load transfer subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem.

13. The system of claim 11, wherein the action determined by the reinforcement learning agent associated with the cooling subsystem is a cooling set point for the system, wherein the action determined by the reinforcement learning agent associated with the load transfer subsystem is a distribution of workload, and wherein the action determined by the reinforcement learning agent associated with the energy storage subsystem is a charge or discharge amount.

14. The system of claim 11, wherein an action determined by a reinforcement learning agent associated with the load transfer subsystem is dependent upon an action determined by a reinforcement learning agent associated with the cooling subsystem and an action determined by a reinforcement learning agent associated with the energy storage subsystem.

15. The system of claim 11, wherein an action determined by a reinforcement learning agent associated with the energy storage subsystem is dependent upon an action determined by a reinforcement learning agent associated with the cooling subsystem and an action determined by a reinforcement learning agent associated with the load transfer subsystem.

16. A system according to claim 10, wherein the state data includes an internal state of the system and an environmental state for the environment of the system, wherein the internal state includes one or more of the following: allocated workload, unallocated workload, temperature of the system, energy consumption of the system, and energy stored by the system, and wherein the environmental state includes one or more of the following: grid carbon intensity, weather, and time.

17. A system comprising: a digital twin of a data center, the digital twin comprising a plurality of subsystem models corresponding to a plurality of subsystems of the data center and configured to simulate a state for each subsystem model and determine a reward for each subsystem model; as well as A plurality of reinforcement learning agents are interfaced with the digital twin and configured to: receiving said state and said reward of the simulation from the digital twin, aggregating said rewards, and determining an action for each subsystem model based on the aggregated reward, and A control signal is generated for each subsystem according to the determined action, wherein the data center is configured based on the control signal.

18. The system of claim 17, wherein the plurality of subsystem models include a cooling subsystem model, a load transfer subsystem model, and an energy storage subsystem model.

19. The system of claim 17, wherein each reinforcement learning agent of the plurality of reinforcement learning agents is associated with a subsystem model of the plurality of subsystem models, and wherein aggregating the rewards comprises: A weight is assigned, by each reinforcement learning agent of the plurality of reinforcement learning agents, to rewards among the rewards determined by an associated subsystem model that is greater than weights assigned to rewards determined by other subsystem models.

20. The system of claim 17, further comprising: receiving an environmental status for an environment of the data center, the environmental status comprising one or more of: grid carbon intensity, weather, and time, The simulated state includes one or more of the following: allocated workload, unallocated workload, temperature of the data center, energy consumption of the data center, and energy stored by the data center.