Cooling control method and device of data center based on safe deep reinforcement learning

Abstract

The embodiment of the invention discloses a cooling control method and device for a data center based on safety deep reinforcement learning, and the method comprises the steps: defining a to-be-processed cooling control problem of the data center as a constrained Markov decision process, the target of the constrained Markov decision process is to minimize the total energy consumption of the data center to be processed on the premise that a preset temperature safety threshold # imgabs0 # is not exceeded; solving the constrained Markov decision process through a deep reinforcement learning algorithm to determine an optimal strategy parameter; based on the optimal strategy parameter, through a preset state transition model, determining an area temperature of the optimal strategy parameter at the end of a prediction control period; and determining a control action of the to-be-processed data control center based on the area temperature of the optimal strategy parameter at the end of the prediction control period.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology and cooling control technology, and in particular to a cooling control method and device for a data center based on secure deep reinforcement learning. Background Art

[0002] As an important part of modern information technology infrastructure, data centers are becoming increasingly large and complex with the rapid development of new technologies such as cloud computing, big data, and big models. When data center servers and equipment are running, they will generate a lot of heat. If it cannot be dissipated in time, local hot spots will be generated, resulting in decreased server performance and shortened equipment life, which will in turn affect the stability and security of the data center. Therefore, the cooling problem of data centers is particularly important.

[0003] In the existing technology, the cooling control of the data center cannot take into account both the energy saving and thermal safety constraints of the data center at the same time. Therefore, it is impossible to solve the constrained Markov decision problem of the cooling control of the data center while ensuring that the thermal safety constraints are met and optimizing the energy efficiency. Therefore, how to improve the efficiency and safety of the cooling control of the data center as much as possible while ensuring that the thermal safety constraints are met is one of the problems that need to be solved urgently.

[0004] Based on this, the embodiments of this specification provide a cooling control method and device for a data center based on secure deep reinforcement learning. Summary of the invention

[0005] The embodiments of this specification provide a cooling control method and device for a data center based on secure deep reinforcement learning, which is used to solve the following technical problems: In the prior art, the cooling control of a data center cannot take into account both the energy saving and thermal safety constraints of the data center. Therefore, it is impossible to solve the constrained Markov decision problem of the cooling control of the data center while ensuring that the thermal safety constraints are met and optimizing the energy efficiency. Therefore, how to improve the efficiency and safety of the cooling control of the data center as much as possible while ensuring that the thermal safety constraints are met is one of the problems that need to be solved urgently.

[0006] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0007] The embodiment of this specification provides a cooling control method for a data center based on secure deep reinforcement learning, including:

[0008] The cooling control problem of the data center to be processed is defined as a constrained Markov decision process, the goal of which is: not to exceed a preset temperature safety threshold Under the premise of minimizing the total energy consumption of the data center to be processed;

[0009] Solving the constrained Markov decision process by a deep reinforcement learning algorithm to determine optimal policy parameters;

[0010] Based on the optimal strategy parameters, determining the regional temperature of the optimal strategy parameters at the end of the prediction control cycle through a preset state transition model;

[0011] Based on the regional temperature of the optimal strategy parameters at the end of the predicted control period, a control action of the data control center to be processed is determined.

[0012] The embodiment of this specification also provides a cooling control device for a data center based on secure deep reinforcement learning, including:

[0013] A custom module that defines the cooling control problem of the data center to be processed as a constrained Markov decision process, the goal of which is to not exceed a preset thermal safety limit Under the premise of minimizing the total energy consumption of the data center to be processed;

[0014] A determination module, solving the constrained Markov decision process through a deep reinforcement learning algorithm to determine optimal policy parameters;

[0015] A prediction module, based on the optimal strategy parameters, determines the regional temperature of the optimal strategy parameters at the end of the prediction control cycle through a preset state transition model;

[0016] The correction module determines the control action of the data center to be processed based on the regional temperature of the optimal strategy parameters at the end of the predicted control cycle.

[0017] The embodiments of this specification provide a cooling control method for a data center based on secure deep reinforcement learning. By combining imitation learning with a post-correction method designed based on the thermodynamic principles of a data center, SARF can avoid violating thermal safety constraints in hot zones of the data center during the learning process while pursuing energy conservation in the data center. It can effectively optimize the energy efficiency of the cooling system while ensuring the thermal safety of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1A schematic diagram of the system architecture of a cooling control method for a data center based on secure deep reinforcement learning provided in an embodiment of this specification;

[0020] Figure 2 A flowchart of a cooling control method for a data center based on secure deep reinforcement learning provided in an embodiment of this specification;

[0021] Figure 3 A schematic diagram of a cooling control device for a data center based on secure deep reinforcement learning provided in an embodiment of this specification. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0023] The Secure Adaptive Reinforcement Learning Framework (SARF) is designed to solve the Constrained Markov Decision Process (CMDP) problem of data center cooling control by ensuring that energy efficiency is optimized while meeting thermal safety constraints. At each step of the algorithm, the algorithm not only considers how to maximize the cumulative reward, but also evaluates whether taking a certain action will lead to a violation of safety constraints. If a potential unsafe behavior is detected, the algorithm will take measures to correct the strategy or directly prevent the behavior from occurring. Based on this, it is possible to ensure that efficient learning is maintained while also meeting the necessary safety requirements.

[0024] The embodiments of this specification provide a cooling control method for a data center based on secure deep reinforcement learning, which optimizes energy efficiency while satisfying thermal safety constraints and realizes cooling control of the data center.

[0025] Figure 1 A schematic diagram of the system architecture of a cooling control method for a data center based on secure deep reinforcement learning provided in an embodiment of this specification. Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various client applications may be installed on the terminal devices 101, 102, 103. For example, a dedicated program such as a cooling control method for a data center based on secure deep reinforcement learning.

[0027] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various dedicated or general electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or they may be implemented as a single software or software module.

[0028] The server 105 may be a server that provides various services, such as a backend server that provides services for client applications installed on the terminal devices 101, 102, and 103. For example, the server may perform cooling control of a data center based on secure deep reinforcement learning, so that the control result of the cooling control of the data center based on secure deep reinforcement learning is displayed on the terminal device servers 101, 102, and 103. The server may also perform cooling control of a data center based on secure deep reinforcement learning, so that the control result of the cooling control of the data center based on secure deep reinforcement learning is displayed on the terminal devices 101, 102, and 103.

[0029] The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 105 is software, it may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or as a single software or software module.

[0030] Figure 2 The following is a flowchart of a cooling control method for a data center based on secure deep reinforcement learning provided in an embodiment of this specification. From a program perspective, the execution subject of the process can be a program installed on an application server or application terminal. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the cooling control method includes:

[0031] Step S201: The cooling control problem of the data center to be processed is defined as a constrained Markov decision process, wherein the goal of the constrained Markov decision process is: not to exceed a preset temperature safety threshold Under the premise of minimizing the total energy consumption of the data center to be processed.

[0032] In the embodiment of the present specification, the data center to be processed is an intelligent data center, and the data center to be processed is an intelligent data center that needs to perform cooling process control.

[0033] In the embodiment of this specification, the constrained Markov decision process is specifically:

[0034]

[0035] in,

[0036] θ * represents the optimal strategy parameters;

[0037] It means finding the parameter θ that maximizes the objective function;

[0038] μ θ Represents the strategy μ θ ;

[0039] Indicates the outdoor air temperature T0 and strategy μ θ The expected sum of rewards r[k] under

[0040] P IT Indicates the power consumption of IT equipment;

[0041] T0 represents the outdoor air temperature T0;

[0042] k represents the kth time step;

[0043] γ represents the discount factor;

[0044] ∈ represents the probability threshold for allowing thermal safety constraint violation;

[0045] Indicates the preset temperature safety threshold;

[0046] T z [k] represents the regional temperature at the kth time step;

[0047] Represents a constraint.

[0048] In a specific embodiment, the preset temperature safety threshold is set according to the upper temperature limit determined by the equipment conditions, the environmental conditions of the data center to be processed, the system redundancy and emergency strategy, the balance between energy consumption and safety, historical data and simulation results. In one embodiment of the present specification, the preset temperature safety threshold is 32°C.

[0049] The equipment conditions refer to the temperature recommendations provided by the server equipment manufacturer.

[0050] Environmental conditions refer to the geographical location and climatic conditions of the data center to be processed.

[0051] System redundancy and emergency strategy means that if the data center has sufficient cooling system redundancy, you can consider setting a higher temperature safety threshold; conversely, if the cooling system has less redundancy or the equipment is highly dependent on the cooling effect, you may need to set a lower temperature safety threshold to ensure safety.

[0052] The balance between energy consumption and safety means that when setting the temperature safety threshold, it is also necessary to weigh the relationship between energy consumption and equipment safety. A higher temperature safety threshold may reduce the energy consumption of the cooling system, but it also increases the risk of equipment overheating.

[0053] Historical data and simulation results refer to that by analyzing historical operation data and simulation results, it is possible to determine at what temperature excessive safety violations occur, thereby optimizing the setting of the temperature safety threshold.

[0054] The core of the SARF algorithm is to combine deep reinforcement learning (DRL) with physical laws-guided safety control to ensure that thermal safety constraints are strictly adhered to while optimizing the energy efficiency of the data center cooling system. In deep reinforcement learning, the parameters of the policy network determine the policy itself, that is, how to choose the best action based on the current state.

[0055] The optimal policy parameter θ * The optimal policy parameter θ is optimized and learned by deep reinforcement learning algorithm to solve the Markov decision process (MDP) problem in data center cooling control. * Mainly responsible for the optimization of the following aspects: (1) Control decision of cooling system: optimal strategy parameter θ * Determines the control actions of cooling systems (such as computer room air conditioning CRAC units), including the supply air temperature T in and the set point of air flow f. These parameters directly affect the temperature and energy consumption level inside the data center. (2) Thermal safety: Optimal strategy parameter θ * It is also necessary to ensure the thermal safety of the data center to prevent the equipment temperature from exceeding the safety upper limit. In DRL, when the temperature exceeds the safe range (i.e. T z [k] not in [TL ,T U ]), the reward function will punish the strategy, thereby pushing the strategy to pay more attention to thermal safety in subsequent choices. (3) Energy efficiency optimization: The optimal strategy parameter θ * One of the optimization goals is to reduce the total energy consumption of the data center DC , which includes the power consumption P of IT equipment IT and the power consumption of the cooling system P c By learning the optimal policy in the DRL process, θ * Actions that maintain the temperature near the set point while reducing overall energy consumption will be chosen.

[0056] In the embodiments of this specification, the discount factor γ is a key hyperparameter that determines the degree of emphasis the algorithm places on future rewards when optimizing strategies. Specifically, the value range of the discount factor γ is between 0 and 1, which represents the trade-off between the importance of current rewards and future rewards. In the cooling control problem of a data center, future environmental conditions and IT loads have a greater impact on current decisions, so a higher discount factor γ needs to be selected. In practical applications, the value of the discount factor is usually set experimentally by continuously adjusting parameters, such as a fixed value between [0.9, 0.99], to ensure proper consideration of future rewards.

[0057] In the embodiments of this specification, ∈ represents the probability threshold for allowing thermal safety constraint violation. Specifically, ∈ defines the tolerance for thermal safety constraints during the optimization process. The specific selection depends on the safety requirements of the application scenario. If the system has very strict requirements for thermal safety, ∈ will be set to a very small value, such as 0.001 or lower, which means that the risk of temperature exceeding the limit is extremely low. However, in some scenarios, a slightly higher ∈ value may be allowed in exchange for improvements in other aspects of system performance (such as energy efficiency).

[0058] Step S203: Solve the constrained Markov decision process by a deep reinforcement learning algorithm to determine the optimal strategy parameters.

[0059] Deep Reinforcement Learning (DRL) is a branch of Reinforcement Learning (RL), which combines deep learning techniques to solve complex decision-making problems.

[0060] In the embodiment of this specification, the deep reinforcement learning algorithm is a DDPG algorithm, and the constrained Markov decision process is solved by the deep reinforcement learning algorithm to determine the optimal strategy parameters, specifically including:

[0061] Solving the constrained Markov decision process via a pre-trained DDPG model to determine the optimal policy parameters;

[0062] in,

[0063] The loss function of the pre-trained DDPG model is the imitation loss function, and the expression of the imitation loss function is:

[0064]

[0065] Loss(θ) represents the imitation loss function;

[0066] μ θ (s safe [m]) indicates that the state is based on s safe [m] the action output by the pre-trained DDPG model;

[0067] a safe [m] indicates the action performed by the traditional safety controller in the same state;

[0068] M represents the number of samples in the training dataset.

[0069] In the embodiments of this specification, the purpose of the simulation loss function is to train the DDPG model so that the control action output by the DDPG model is as close as possible to the action performed by the traditional safety controller. Specifically, the goal of the simulation loss function is to minimize the error between the action output by the model and the action performed by the traditional controller.

[0070] In the embodiments of this specification, the traditional safety controller is a controller used in the prior art to implement data center cooling control. The traditional safety controller can be a DRL-based controller or a large model-based controller. In short, the specific type of the traditional safety controller does not constitute a limitation on this application.

[0071] DDPG (Deep Deterministic Policy Gradient) is a deep reinforcement learning algorithm that is particularly suitable for solving problems in continuous action spaces. It combines the Actor-Critic architecture and the policy gradient method, and uses a deep neural network to approximate the policy and value functions.

[0072] Before obtaining the pre-trained DDPG model, data collection and data processing are required. Specifically, data collection includes: historical operation data, including the status under the control of traditional safety controllers (such as data hall temperature, IT equipment power consumption, cooling system power consumption, etc.) and corresponding control actions (such as CRAC unit supply air temperature and mass flow set point).

[0073] After collecting data, in order to ensure the consistency and accuracy of the data, the data needs to be preprocessed, including data cleaning (removing noise, outliers, etc.), data format unification, etc. The data is divided into training sets and validation sets for model training and validation.

[0074] In a specific embodiment, the data format unification includes: consistent data types and data normalization / standardization.

[0075] Specifically, data type consistency means that all input data (such as temperature, power consumption, air flow, etc.) need to be converted into numerical data and ensure that they have the same measurement unit, such as Celsius for temperature and Watts for power.

[0076] Data normalization / standardization means that in order to ensure the stability and efficiency of model training, all input variables may need to be normalized or standardized. For example, all input features are scaled to the range of 0 to 1 or converted to a standard normal distribution with zero mean and unit variance.

[0077] In an embodiment of the present specification, the pre-trained DDPG model includes an actor network and a critic network, wherein the actor network is used to output a control action according to the current state of the data center to be processed, and the critic network is used to evaluate the value of the control action output by the actor network, and the current state of the data center to be processed includes IT power consumption, cooling power consumption, cold channel temperature, regional temperature, and outdoor air temperature of the data center to be processed.

[0078] Since DDPG is a policy gradient algorithm, it maximizes the action value given by the critic network by continuously adjusting the network parameters, so that the control actions output by the actor network can optimize the energy consumption of the data center while ensuring thermal safety. This process ensures that the actor network can learn an effective strategy and make the best control decision based on the current state, thereby improving the performance of the entire system.

[0079] In the embodiments of this specification, a pre-trained DDPG model is obtained through an offline imitation learning method, which can provide a good starting point for online deep reinforcement learning, significantly reduce unsafe attempts in the online learning stage, accelerate the learning process, and improve the performance and safety of the final control strategy.

[0080] In order to further ensure that subsequent cooling control actions will not cause unsafe conditions, the pre-trained DDPG model obtained offline is used as a DRL agent during the application process. After deployment, it is also necessary to perform online post-correction. Specifically, during the online operation of the DRL agent, each recommended action will be predicted by a state transition model based on the laws of physics to predict the regional temperature at the end of the control cycle. If the prediction result shows that the action will cause the system to enter an unsafe state (the predicted temperature exceeds the safety threshold), a minimum correction action is found by solving the optimization problem to ensure that the corrected action will not cause an unsafe state. Specifically, a preset state transition model can be used for implementation.

[0081] Step S205: Based on the optimal strategy parameters, determine the regional temperature of the optimal strategy parameters at the end of the prediction control cycle through a preset state transition model.

[0082] In the embodiments of this specification, the preset state transition model is a model based on the laws of thermodynamics, which can accurately predict the state changes such as the temperature of the data hall caused by different control actions. The model is obtained by fitting historical safety data and can be extrapolated to the state explored by DRL but not appearing in the historical data.

[0083] In an embodiment of the present specification, the preset state transition model is a state transition model based on physical laws, and the preset state transition model is used to determine the regional temperature of the optimal strategy parameters at the end of the predicted control cycle. The preset state transition model includes: a steady-state state transition model (SARF-1), and / or a transient state transition model (SARF-2), and\or an integrated predicted IT power trajectory model (SARF-3).

[0084] In order to further understand the preset state transition model, the preset state transition model will be described in detail below.

[0085] The steady-state state transfer model is based on the simplified assumption that at the end of the control cycle, the data center system has reached a steady state and the main control of the cooling components has no steady-state control error. It uses a simple algebraic equation to predict the zone temperature at the end of the next control cycle. This model is suitable for scenarios where the accuracy of temperature changes within the control cycle is not very high, or when the control cycle is short and the system can quickly reach a steady state.

[0086] The transient state transfer model abandons the steady-state assumption in SARF-2 and uses a transient thermodynamic model based on the law of conservation of energy to predict changes in regional temperature. The model takes into account the continuous changes in supply air temperature and mass flow rate during the control cycle, providing a more accurate prediction of future states. It is suitable for situations where high temperature control accuracy is required, or when the control cycle is long and the system cannot quickly reach a steady state.

[0087] The Integrated Predicted IT Power Trajectory Model (SARF-3) further improves on SARF-2 by integrating predicted IT power trajectories to optimize the prediction of state transitions. It uses historical data to construct a maximum rise trajectory for IT power and uses this trajectory in the forecast to more accurately predict the thermal load of the data center. This model is suitable for scenarios where the IT load varies greatly or there is a clear rise / fall pattern, especially when the predictability of the IT load in the data center is high.

[0088] Step S207: Determine the control action of the data control center to be processed based on the regional temperature at the end of the predicted control period according to the optimal strategy parameters.

[0089] In the embodiment of this specification, the determining of the control action of the data center to be processed based on the regional temperature at the end of the predicted control cycle by the optimal strategy parameter specifically includes:

[0090] If the regional temperature of the optimal strategy parameter at the end of the prediction control cycle is less than the preset temperature safety threshold Then the optimal strategy parameters are used as the control action of the data center to be processed;

[0091] If the regional temperature of the optimal strategy parameter at the end of the prediction control cycle is greater than or equal to the preset temperature safety threshold Based on the type of the preset state transition model, the control action corresponding to the optimal strategy parameter is corrected to determine the control action of the data center to be processed.

[0092] In order to further understand the correction process of the control action, it will be described in detail below in combination with the states of the preset state transition model.

[0093] In the embodiment of this specification, the preset state transition model is a steady-state state transition model, and the steady-state state transition model is:

[0094]

[0095] in,

[0096] k represents the kth time step;

[0097] (k-1) represents the (k-1)th time step;

[0098] T z [k] represents the regional temperature at the kth time step;

[0099] Indicates the set cold channel temperature;

[0100] represents the cold channel temperature set at the (k-1)th time step;

[0101] Indicates the total volume flow rate of the set cooling supply air;

[0102] represents the total volume flow rate of the cooling supply air set at the (k-1)th time step;

[0103] α represents the system-related parameter related to the heat capacity of air;

[0104] ρ represents the density of air;

[0105] P IT Indicates the power consumption of IT equipment;

[0106] P IT [k] represents the power consumption of IT equipment at the kth time step;

[0107] Then the predicted temperature of the next time step (k+1) after the kth time step is:

[0108]

[0109] in,

[0110] represents the predicted temperature at the (k+1)th time step;

[0111] μ[k] represents the control action recommended by the deep reinforcement learning algorithm at the kth time step;

[0112] h(·) represents the steady-state state transition model;

[0113] like Greater than the preset temperature safety threshold Based on the optimal strategy parameters, the regional temperature of the optimal strategy parameters at the end of the prediction control cycle is determined through a preset state transition model, and the optimization is as follows:

[0114]

[0115] in,

[0116] represents the corrected supply air temperature set point;

[0117] represents the cold channel temperature set at the kth time step;

[0118] λ represents the Lagrange multiplier;

[0119] represents the corrected air flow set point;

[0120] P IT [k+1] represents the power consumption of IT equipment at the (k+1)th time step;

[0121] α represents the system-related parameter related to the heat capacity of air;

[0122] represents the volume flow rate of the cooling supply air set at the kth time step;

[0123] The corrected action As a control action of the data center to be processed.

[0124] In a specific embodiment, the value of α is not pre-set, but is determined by fitting the actual system operation data. In actual operation, by collecting data of the traditional controller under steady-state conditions (including temperature, air flow, IT load, etc.), the most appropriate α value can be fitted using these data. Specifically, using the known input-output pairs in the historical data, α is adjusted to minimize the error between the model prediction value and the actual measurement value.

[0125] In the embodiment of this specification, the preset state transition model is a transient state transition model, and the transient transition model uses the current regional temperature as the initial condition to calculate the temperature change within the control period:

[0126]

[0127] in,

[0128] T z (t) represents the temperature of the current region in period t;

[0129] t represents the control time;

[0130] f(t) represents the flow rate of cold air supplied by the cooling system of the data center to be processed to the data center at time t

[0131] ρ represents the density of air;

[0132] V s Indicates the volume of the data hall of the data center to be processed;

[0133] Tin (t) represents the air supply temperature of the cooling system of the data center to be processed at time t;

[0134] Q(t) represents the heat generated by the data center to be processed at time t;

[0135] Determining a temperature average value within the control period based on the temperature change within the control period;

[0136] If the temperature average value within the control period is greater than the preset temperature safety threshold, a grid search method is used to determine a candidate corrective action;

[0137] Based on the candidate corrective action, the temperature average value in the corresponding control cycle is predicted. If the predicted temperature average value in the corresponding control cycle is not greater than the preset temperature safety threshold, the candidate corrective action is used as the control action of the data center to be processed.

[0138] In the embodiment of this specification, the preset state transition model is an integrated predicted IT power trajectory model;

[0139] The preset state transition model predicts the growth trajectory of IT power in the current control cycle using a maximum rise function;

[0140] Determining a predicted trajectory of the zone temperature z(t) based on the predicted growth trajectory of IT power during the current control period and a preset set point;

[0141] Determining an average temperature within the control period based on the predicted trajectory of the regional temperature z(t);

[0142] If the temperature average value within the control period is greater than the preset temperature safety threshold, a grid search method is used to determine a candidate corrective action;

[0143] Based on the candidate corrective action, the temperature average value in the corresponding control cycle is predicted. If the predicted temperature average value in the corresponding control cycle is not greater than the preset temperature safety threshold, the candidate corrective action is used as the control action of the data center to be processed.

[0144] In the embodiment of this specification, the maximum rise function is obtained based on historical IT power data to construct a maximum rise function representing IT power growth. This function reflects the power from a specific starting Starting with the maximum possible trajectory of IT power growth over time Δt.

[0145] In the embodiment of the present specification, the prediction algorithm predicts the IT power change in the next period based on the IT power IT[t] of the current period at the beginning of each control period τ.

[0146] In the embodiment of this specification, the preset state transition model predicts the growth trajectory of IT power in the current control cycle with the maximum rise function, specifically including:

[0147] Use the maximum rise function to predict the growth trajectory of IT power within the current control period

[0148] In the embodiment of the present specification, the preset set point is the set point T of the supply air temperature set by integrating the predicted IT power trajectory model at the beginning of the control cycle t=τ. in [t] and the set point m[t] for the total mass flow rate.

[0149] In the embodiment of the present specification, based on the predicted growth trajectory of IT power in the current control period and the preset set point, the predicted trajectory of the regional temperature z(t) is determined, specifically including:

[0150] Based on the predicted growth trajectory of IT power in the current control cycle and the preset set point, the following formula is used for calculation:

[0151]

[0152] in,

[0153] T z (t) represents the temperature of the current region in period t;

[0154] t represents the control time;

[0155] f(t) represents the flow rate of cold air supplied by the cooling system of the data center to be processed to the data center at time t

[0156] ρ represents the density of air;

[0157] V s Indicates the volume of the data hall of the data center to be processed;

[0158] T in (t) represents the air supply temperature of the cooling system of the data center to be processed at time t;

[0159] Q(t) represents the heat generated by the data center to be processed at time t;

[0160] To determine the predicted trajectory of the regional temperature z(t).

[0161] In the embodiment of the present specification, the IT power Q(t) predicted by the integrated predicted IT power trajectory model is set to a conservative high value, which helps prevent the occurrence of unsafe situations such as overheating.

[0162] The cooling control method of a data center based on secure deep reinforcement learning provided in the embodiments of this specification defines the cooling control problem of the data center to be processed as a constrained Markov decision process, wherein the goal of the constrained Markov decision process is: not to exceed a preset thermal safety upper limit Under the premise of minimizing the total energy consumption of the data center to be processed; solving the constrained Markov decision process through a deep reinforcement learning algorithm to determine the optimal strategy parameters; based on the optimal strategy parameters, determining the regional temperature of the optimal strategy parameters at the end of the predicted control cycle through a preset state transition model; based on the regional temperature of the optimal strategy parameters at the end of the predicted control cycle, determining the control action of the data center to be processed, while pursuing energy saving in the data center, avoiding violation of thermal safety constraints in the hot zone of the data center during the learning process, and effectively optimizing the energy efficiency of the cooling system under the premise of ensuring the thermal safety of the data center.

[0163] The above content describes in detail a cooling control method for a data center based on secure deep reinforcement learning. Correspondingly, this specification also provides a cooling control device for a data center based on secure deep reinforcement learning, such as Figure 3 shown. Figure 3 A schematic diagram of a cooling control device for a data center based on secure deep reinforcement learning provided in an embodiment of this specification, the cooling control device comprising:

[0164] Custom module 301 defines the cooling control problem of the data center to be processed as a constrained Markov decision process, wherein the goal of the constrained Markov decision process is: not to exceed a preset thermal safety upper limit Under the premise of minimizing the total energy consumption of the data center to be processed;

[0165] A determination module 303 solves the constrained Markov decision process by a deep reinforcement learning algorithm to determine optimal policy parameters;

[0166] The prediction module 305 determines the regional temperature of the optimal strategy parameter at the end of the prediction control cycle through a preset state transition model based on the optimal strategy parameter;

[0167] The correction module 307 determines the control action of the data center to be processed based on the regional temperature of the optimal strategy parameters at the end of the predicted control cycle.

[0168] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0169] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0170] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have beneficial technical effects similar to those of the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0171] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0172] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0173] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0174] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0175] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0176] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0179] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0180] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0181] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0182] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0183] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0184] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] The above is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A cooling control method for data centers based on secure deep reinforcement learning, It is characterized in that The cooling control method comprises: The cooling control problem of the data center to be processed is defined as a constrained Markov decision process, the goal of which is: not to exceed a preset temperature safety threshold Under the premise of minimizing the total energy consumption of the data center to be processed; Solving the constrained Markov decision process by a deep reinforcement learning algorithm to determine optimal policy parameters; Based on the optimal strategy parameters, determining the regional temperature of the optimal strategy parameters at the end of the prediction control cycle through a preset state transition model; Based on the regional temperature of the optimal strategy parameters at the end of the predicted control period, a control action of the data control center to be processed is determined.

2. The cooling control method according to claim 1, It is characterized in that The constrained Markov decision process is specifically: in, θ * represents the optimal strategy parameters; It means finding the parameter θ that maximizes the objective function; μ θ Represents the strategy μ θ ; Indicates the outdoor air temperature T 0 and strategy μ θ The expected sum of rewards r[k] under P IT Indicates the power consumption of IT equipment; T 0 Indicates the outdoor air temperature T 0 ; k represents the kth time step; γ represents the discount factor; ∈ represents the probability threshold for allowing thermal safety constraint violation; Indicates the preset temperature safety threshold; T z [k] represents the regional temperature at the kth time step; Represents a constraint.

3. The cooling control method according to claim 1, It is characterized in that The deep reinforcement learning algorithm is a DDPG algorithm, and the constrained Markov decision process is solved by the deep reinforcement learning algorithm to determine the optimal strategy parameters, specifically including: Solving the constrained Markov decision process via a pre-trained DDPG model to determine the optimal policy parameters; in, The loss function of the pre-trained DDPG model is the imitation loss function, and the expression of the imitation loss function is: Loss(θ) represents the imitation loss function; μ θ (s safe [m]) indicates that the state is based on s safe [m] the action output by the pre-trained DDPG model; a safe [m] indicates the action performed by the traditional safety controller in the same state; M represents the number of samples in the training dataset.

4. The cooling control method according to claim 3, It is characterized in that The pre-trained DDPG model includes an actor network and a critic network, wherein the actor network is used to output a control action according to the current state of the data center to be processed, and the critic network is used to evaluate the value of the control action output by the actor network, and the current state of the data center to be processed includes IT power consumption, cooling power consumption, cold channel temperature, regional temperature, and outdoor air temperature of the data center to be processed.

5. The cooling control method according to claim 1, It is characterized in that The preset state transition model is a state transition model based on physical laws. The preset state transition model is used to determine the regional temperature of the optimal strategy parameters at the end of the predicted control cycle. The preset state transition model includes: a steady-state state transition model, and / or a transient state transition model, and\or an integrated predicted IT power trajectory model.

6. The cooling control method according to claim 1, It is characterized in that The step of determining the control action of the data center to be processed based on the regional temperature at the end of the predicted control cycle by the optimal strategy parameter specifically includes: If the regional temperature of the optimal strategy parameter at the end of the prediction control cycle is less than the preset temperature safety threshold Then the optimal strategy parameters are used as the control action of the data center to be processed; If the regional temperature of the optimal strategy parameter at the end of the prediction control cycle is greater than or equal to the preset temperature safety threshold Based on the type of the preset state transition model, the control action corresponding to the optimal strategy parameter is corrected to determine the control action of the data center to be processed.

7. The cooling control method according to claim 1, It is characterized in that The preset state transition model is a steady-state state transition model, and the steady-state state transition model is: in, k represents the kth time step; (k-1) represents the (k-1)th time step; T z [k] represents the regional temperature at the kth time step; Indicates the set cold channel temperature; represents the cold channel temperature set at the (k-1)th time step; Indicates the total volume flow rate of the set cooling supply air; represents the total volume flow rate of the cooling supply air set at the (k-1)th time step; α represents the system-related parameter related to the heat capacity of air; ρ represents the density of air; P IT Indicates the power consumption of IT equipment; P IT [k] represents the power consumption of IT equipment at the kth time step; Then the predicted temperature of the next time step (k+1) after the kth time step is: in, represents the predicted temperature at the (k+1)th time step; μ[k] represents the control action recommended by the deep reinforcement learning algorithm at the kth time step; h(·) represents the steady-state state transition model; like Greater than the preset temperature safety threshold Based on the optimal strategy parameters, the regional temperature of the optimal strategy parameters at the end of the prediction control cycle is determined through a preset state transition model, and the optimization is as follows: in, represents the corrected supply air temperature set point; represents the cold channel temperature set at the kth time step; λ represents the Lagrange multiplier; represents the corrected air flow set point; P IT [k+1] represents the power consumption of IT equipment at the (k+1)th time step; α represents the system-related parameter related to the heat capacity of air; represents the volume flow rate of the cooling supply air set at the kth time step; The corrected action As a control action of the data center to be processed.

8. The cooling control method according to claim 1, It is characterized in that The preset state transfer model is a transient state transfer model, which uses the current regional temperature as the initial condition and calculates the temperature change within the control period: in, T z (t) represents the temperature of the current region in period t; t represents the control time; f(t) represents the flow rate of cold air supplied by the cooling system of the data center to be processed to the data center at time t ρ represents the density of air; V s Indicates the volume of the data hall of the data center to be processed; T in (t) represents the air supply temperature of the cooling system of the data center to be processed at time t; Q(t) represents the heat generated by the data center to be processed at time t; Determining a temperature average value within the control period based on the temperature change within the control period; If the temperature average value within the control period is greater than the preset temperature safety threshold, a grid search method is used to determine a candidate corrective action; Based on the candidate corrective action, the temperature average value in the corresponding control cycle is predicted. If the predicted temperature average value in the corresponding control cycle is not greater than the preset temperature safety threshold, the candidate corrective action is used as the control action of the data center to be processed.

9. The cooling control method according to claim 1, It is characterized in that The preset state transition model is an integrated predicted IT power trajectory model; The preset state transition model predicts the growth trajectory of IT power in the current control cycle using a maximum rise function; Determining a predicted trajectory of the zone temperature z(t) based on the predicted growth trajectory of IT power during the current control period and a preset set point; Determining an average temperature within the control period based on the predicted trajectory of the regional temperature z(t); If the temperature average value within the control period is greater than the preset temperature safety threshold, a grid search method is used to determine a candidate corrective action; Based on the candidate corrective action, the temperature average value in the corresponding control cycle is predicted. If the predicted temperature average value in the corresponding control cycle is not greater than the preset temperature safety threshold, the candidate corrective action is used as the control action of the data center to be processed.

10. A cooling control device for a data center based on secure deep reinforcement learning. It is characterized in that The cooling control device comprises: A custom module that defines the cooling control problem of the data center to be processed as a constrained Markov decision process, the goal of which is to not exceed a preset thermal safety limit Under the premise of minimizing the total energy consumption of the data center to be processed; A determination module, solving the constrained Markov decision process through a deep reinforcement learning algorithm to determine optimal policy parameters; A prediction module, based on the optimal strategy parameters, determines the regional temperature of the optimal strategy parameters at the end of the prediction control cycle through a preset state transition model; The correction module determines the control action of the data center to be processed based on the regional temperature of the optimal strategy parameters at the end of the predicted control cycle.

Images