Data machine room energy-saving system based on reinforcement learning

Abstract

The invention relates to the technical field of data machine room energy conservation, which comprises a machine room environment modeling module, a reinforcement learning environment construction module, a reinforcement learning energy-saving algorithm module and an algorithm comparison module, and is characterized in that the machine room environment modeling module comprises an actual machine room environment unit and a TRASYS simulation machine room environment unit; the reinforcement learning environment construction module comprises a state action space design unit and a reward function design unit, and the reinforcement learning energy-saving algorithm module comprises a single-agent machine room energy-saving algorithm implementation unit and a multi-agent machine room energy-saving algorithm implementation unit. By constructing a data machine room energy consumption simulation model, simulating a running environment and performing energy consumption behavior optimization model training, a reinforcement learning model is utilized to optimize machine room energy consumption based on data, factors such as air conditioner and cabinet arrangement and outdoor temperature are considered, a state space, an action space and a reward function are designed, and the application effect and feasibility of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data room energy saving, and more specifically, to a data room energy saving system based on reinforcement learning. Background Art

[0002] With the development of cloud computing and related technologies, the number of data centers has increased, and their energy consumption problem has become increasingly serious. Data shows that the annual growth rate of power consumption in data centers exceeds 12%, and it is expected to reach 38 billion watt-hours by 2025, and will account for 8% of the country's electricity consumption in 2030. The energy consumption of data centers mainly includes data equipment systems (about 50%), air conditioning systems (about 37%), UPS power supply systems (about 10%), and other systems (about 3%). The key to reducing energy consumption is to reduce major energy consumption factors such as equipment, air conditioning, and power supply systems. The PUE value is an indicator to measure the energy efficiency of data centers. The PUE value of most data centers in China is greater than 2, and the goal is to reduce it to between 1.5 and 2.

[0003] Traditional energy consumption optimization methods rely on rules and static strategies. Most data centers are under manual control all year round, and the air conditioners in the data centers are set to fixed values ​​all year round. Most of the time, the air conditioners are always running, which leads to a waste of power consumption. When the air supply and return air temperature of the air conditioning system increases by 1°C, the energy efficiency of the air conditioner increases by about 2.5%, and the cooling capacity of the air conditioner increases by about 5%. The present invention designs a data center energy-saving system based on reinforcement learning. The system controls the switch of the air conditioner and sets the temperature through the reinforcement learning algorithm to achieve energy saving in the data center from the air conditioning system. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a data center energy-saving system based on reinforcement learning, which has the advantage of achieving energy saving in the center by controlling the switch of the air conditioner and setting the temperature through the air conditioning system through a reinforcement learning algorithm.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a data center energy-saving system based on reinforcement learning, comprising a center environment modeling module, a reinforcement learning environment construction module, a reinforcement learning energy-saving algorithm module and an algorithm comparison module, wherein the center environment modeling module comprises an actual center environment unit and a TRASYS simulation center environment unit, the reinforcement learning environment construction module comprises a state-action space design unit and a reward function design unit, the reinforcement learning energy-saving algorithm module comprises a single-agent center energy-saving algorithm implementation unit and a multi-agent center energy-saving algorithm implementation unit, and the algorithm comparison module comprises a comparison unit between each reinforcement learning algorithm and a comparison with other traditional and advanced control methods.

[0006] As a preferred technical solution of the present invention, the TRASYS simulation room environment unit uses TRASYS software to model the data room, complete the simulation and generate actual simulation data.

[0007] As a preferred technical solution of the present invention, the reinforcement learning environment construction module is implemented using an energy-saving algorithm based on a deep Q network. The state-action space design unit constructs the state space of the deep Q network with the outdoor temperature of the computer room. All executable actions and maintenance actions of the air conditioner are included in the action space of the deep Q network. The reward function design unit comprehensively considers the COP value of the air conditioner and whether the temperature of each data point in the data room exceeds the set range and the degree of over-temperature to set the reward function of the air conditioner and train the deep Q network.

[0008] As a preferred technical solution of the present invention, the single-agent computer room energy-saving algorithm implementation unit adopts a SAC-based reinforcement learning computer room energy-saving algorithm. The SAC-based reinforcement learning computer room energy-saving algorithm can handle discrete and continuous action space problems. The algorithm encourages exploration and assigns approximately equal probabilities to actions with similar Q values, avoiding the air conditioner from repeatedly performing the same action and falling into suboptimal conditions. By maximizing rewards, low reward strategies are abandoned.

[0009] As a preferred technical solution of the present invention, the multi-agent computer room energy-saving algorithm implementation unit regards all air-conditioning systems in the above-mentioned computer room as an intelligent agent as a whole. In future planning, each air conditioner in the computer room will be regarded as an independent intelligent agent and controlled by agents respectively to optimize their different operations. The goal of each agent is to maximize the shared overall reward value.

[0010] As a preferred technical solution of the present invention, the algorithm comparison module uses multiple evaluation indicators to evaluate performance, including energy-saving efficiency, system stability, response time, and algorithm convergence speed. Through these indicators, the performance of different reinforcement learning algorithms in practical applications can be comprehensively compared, and optimization suggestions can be provided based on the evaluation results to guide further adjustment and improvement of the system.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] 1. The present invention constructs a simulation model for the energy consumption of a data room, which can simulate the actual operating environment and energy consumption behavior of the data room. Through simulation, the model can be better trained and optimized, and a reinforcement learning model suitable for the data room can be constructed. Based on the collected data, a reinforcement learning model suitable for optimizing the energy consumption of the data room is designed. According to the characteristics of the data room, such as the arrangement and combination of the air conditioners and cabinets in the room, and the outdoor temperature, a suitable state space, action space and reward function are designed to improve the effect and feasibility of the reinforcement learning model in practical applications.

[0013] 2. The present invention realizes energy saving in the computer room by controlling the switching of the air conditioner and setting the temperature through the air conditioning system through the reinforcement learning algorithm. The data show that when the supply and return air temperature of the air conditioning system increases by 1°C, the energy efficiency of the air conditioner increases by about 2.5%. At the same time, the cooling capacity of the air conditioner increases by about 5%. The switching and setting temperature of the precision air conditioner in the computer room are controlled by the reinforcement learning algorithm, which reduces the energy consumption of the air conditioner without exceeding the specified temperature. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall framework structure diagram of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] like Figure 1 As shown, the present invention provides a data center energy-saving system based on reinforcement learning, including a center environment modeling module, a reinforcement learning environment construction module, a reinforcement learning energy-saving algorithm module and an algorithm comparison module, wherein the center environment modeling module includes an actual center environment unit and a TRASYS simulation center environment unit, the reinforcement learning environment construction module includes a state-action space design unit and a reward function design unit, the reinforcement learning energy-saving algorithm module includes a single-agent center energy-saving algorithm implementation unit and a multi-agent center energy-saving algorithm implementation unit, and the algorithm comparison module includes comparison units between various reinforcement learning algorithms and comparisons with other traditional and advanced control methods.

[0017] The TRASYS simulation room environment unit uses TRASYS software to model the data room, complete the simulation and generate actual simulation data.

[0018] Through the modular dynamic simulation software TRANSYS, which has accurate HVAC mathematical models and control system models, it is easy to build a centralized air-conditioning control system. The software opens module codes to facilitate users to customize system models and control algorithms. By using TRASYS software to model the actual data computer room, the simulation of the data computer room can be completed well. Compared with the current actual computer room, the actual simulation data of the computer room can be generated more quickly, which can be better combined with the subsequent reinforcement learning algorithm, facilitating the subsequent experiments.

[0019] Among them, the reinforcement learning environment construction module is implemented using an energy-saving algorithm based on a deep Q network. The state-action space design unit constructs the state space of the deep Q network with the outdoor temperature of the computer room. All executable actions and maintenance actions of the air conditioner are included in the action space of the deep Q network. The reward function design unit comprehensively considers the COP value of the air conditioner and whether the temperature of each data point in the data room exceeds the set range and the degree of over-temperature, so as to set the reward function of the air conditioner and train the deep Q network.

[0020] Through the state space S design, the return air temperature, compressor status, set temperature, arrangement of the air conditioner, the air inlet and outlet temperature of the cabinet, the arrangement, the central wall temperature of the computer room, and the outdoor temperature of the computer room are used as the state space of DQN. All executable actions + hold actions of all air conditioners are used as the action space of DQN

[10] . The reward function of the air conditioner is set by comprehensively considering the cop value of the air conditioner and whether each data point in the data room is over-temperature and the degree of over-temperature. Finally, the deep Q network is trained.

[0021] Among them, the single-agent computer room energy-saving algorithm implementation unit adopts a SAC-based reinforcement learning computer room energy-saving algorithm. The SAC-based reinforcement learning computer room energy-saving algorithm can handle discrete and continuous action space problems. The algorithm encourages exploration and assigns approximately equal probabilities to actions with similar Q values, avoiding the air conditioner from repeatedly performing the same action and falling into suboptimal conditions. By maximizing rewards, low reward strategies are abandoned.

[0022] The reinforcement learning SAC algorithm can solve the reinforcement learning problems in discrete action space and continuous action space, and improve the stability and performance of Off-policy RL. The SAC algorithm uses maximum entropy. In addition to maximizing the cumulative reward, it also requires that the entropy value of each output action of the policy is the largest, that is, the action is more random and not concentrated on a certain action.

[0023] Compared with the reinforcement learning algorithm that randomizes the strategy, it also encourages exploration and assigns almost equal probabilities to actions with similar Q values. It does not assign very high probabilities to any action within the action range of the data center air conditioner, thus avoiding the air conditioner repeatedly choosing the same action and falling into suboptimal behavior. At the same time, by maximizing the reward, it abandons strategies that are obviously hopeless (abandoning low-reward strategies).

[0024] The reinforcement learning algorithm based on SAC can handle discrete and continuous action space problems, and improves stability and performance. SAC not only pursues the maximization of cumulative rewards, but also requires the action entropy value of the strategy output to be maximized to increase the randomness of the action. The algorithm encourages exploration and assigns approximately equal probabilities to actions with similar Q values ​​to avoid the air conditioner repeatedly performing the same action and falling into suboptimal behavior. At the same time, it abandons low-reward strategies by maximizing rewards.

[0025] Among them, the multi-agent computer room energy-saving algorithm implementation unit regards all air-conditioning systems in the above-mentioned computer room as an intelligent agent as a whole. In future planning, each air conditioner in the computer room will be regarded as an independent intelligent agent and controlled by agents respectively to optimize their different operations. The goal of each agent is to maximize the shared overall reward value.

[0026] By applying multi-agent reinforcement learning algorithms to the computer room energy-saving system, more refined control and more efficient energy utilization can be achieved. In the multi-agent system, each agent (i.e., each air conditioner) has its own state and action space, and they can communicate and coordinate with each other to achieve the overall energy saving goal. The multi-agent system learns how to minimize the overall energy consumption while ensuring the stability of the computer room temperature by constantly interacting with the environment. Each agent selects the optimal action based on its own state and environmental information, and achieves the energy-saving goal through coordination with other agents. For example, when an air conditioner detects that the temperature in the central area of ​​the computer room is high, it may lower its own temperature setting and notify other air conditioners to increase the wind speed to help lower the temperature in the central area.

[0027] In addition, the multi-agent system can further improve energy efficiency by simulating and predicting future environmental changes and making adjustments in advance. For example, the system can predict the temperature change trend of the computer room based on historical data and adjust the working status of the air conditioner in advance to avoid unnecessary energy consumption.

[0028] Among them, the algorithm comparison module uses a variety of evaluation indicators to evaluate performance, including energy-saving efficiency, system stability, response time, and algorithm convergence speed. Through these indicators, the performance of different reinforcement learning algorithms in practical applications can be comprehensively compared, and optimization suggestions can be provided based on the evaluation results to guide further adjustment and improvement of the system.

[0029] By comparing the energy-saving effects of different reinforcement learning algorithms under the same conditions, we can evaluate which algorithm is more efficient in practical applications. Through comparative analysis, we can find the advantages of reinforcement learning algorithms in dealing with complex environments and dynamic changes, as well as possible deficiencies in practical applications, thereby providing a basis for further optimization of the system.

[0030] The working principle and usage process of the present invention are as follows: first, a centralized air-conditioning control system model is constructed using TRANSYS software, and users are allowed to customize models and algorithms, simulate a data room to generate data, and support reinforcement learning training. Next, a DQN-based energy-saving algorithm is used to construct a reinforcement learning environment, and the outdoor temperature and air-conditioning action are included in the state and action space of DQN. The reward function considers the COP value and temperature range, and DQN is trained. The single-agent energy-saving algorithm adopts SAC to deal with action space problems, and pursues the maximization of rewards and action entropy. The multi-agent energy-saving algorithm regards all air conditioners as one agent. In the future, it is planned to control each air conditioner independently to optimize the operation. Finally, the algorithm performance is evaluated through indicators such as energy-saving efficiency, stability, response time and convergence speed, and optimization suggestions are provided.

[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0032] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data center energy-saving system based on reinforcement learning, comprising a center environment modeling module, a reinforcement learning environment construction module, a reinforcement learning energy-saving algorithm module and an algorithm comparison module, characterized in that: The computer room environment modeling module includes an actual computer room environment unit and a TRASYS simulated computer room environment unit, the reinforcement learning environment construction module includes a state-action space design unit and a reward function design unit, the reinforcement learning energy-saving algorithm module includes a single-agent computer room energy-saving algorithm implementation unit and a multi-agent computer room energy-saving algorithm implementation unit, and the algorithm comparison module includes comparison units between various reinforcement learning algorithms and comparisons with other traditional and advanced control methods.

2. According to claim 1, a data center energy saving system based on reinforcement learning is characterized in that: The TRASYS simulation room environment unit uses TRASYS software to model the data room, complete the simulation and generate actual simulation data.

3. According to claim 1, a data center energy saving system based on reinforcement learning is characterized in that: The reinforcement learning environment construction module is implemented using an energy-saving algorithm based on a deep Q network. The state-action space design unit uses the outdoor temperature of the computer room to form the state space of the deep Q network. All executable actions and maintenance actions of the air conditioner are included in the action space of the deep Q network. The reward function design unit comprehensively considers the COP value of the air conditioner and whether the temperature of each data point in the data room exceeds the set range and the degree of over-temperature to set the reward function of the air conditioner and train the deep Q network.

4. The data center energy saving system based on reinforcement learning according to claim 1 is characterized in that: The single-agent computer room energy-saving algorithm implementation unit adopts a SAC-based reinforcement learning computer room energy-saving algorithm. The SAC-based reinforcement learning computer room energy-saving algorithm can handle discrete and continuous action space problems. The algorithm encourages exploration and assigns approximately equal probabilities to actions with similar Q values, avoiding the air conditioner from repeatedly performing the same action and falling into suboptimal conditions. By maximizing rewards, low reward strategies are abandoned.

5. According to claim 1, a data center energy saving system based on reinforcement learning is characterized in that: The multi-agent computer room energy-saving algorithm implementation unit regards all air-conditioning systems in the above-mentioned computer room as an intelligent agent as a whole. In future planning, each air conditioner in the computer room will be regarded as an independent intelligent agent and controlled by agents respectively to optimize their different operations. The goal of each agent is to maximize the shared overall reward value.

6. The data center energy saving system based on reinforcement learning according to claim 1 is characterized in that: The algorithm comparison module uses a variety of evaluation indicators to evaluate performance, including energy efficiency, system stability, response time, and algorithm convergence speed. Through these indicators, the performance of different reinforcement learning algorithms in practical applications can be comprehensively compared, and optimization suggestions can be provided based on the evaluation results to guide further adjustment and improvement of the system.

Images