Distributed intelligent control method for data center air conditioning

Abstract

The distributed intelligent control method of data center air conditioning establishes a Markov game model inside the data center computer room, and uses a multi-agent reinforcement learning algorithm to solve the model to control the air supply speed and air supply temperature of the air conditioner. The Markov game model consists of five parts: the number of agents, the environment state, the action space, the reward function, and the discount factor. The optimal strategy is determined in the process of continuous interaction with the environment, and the optimal strategy maximizes the cumulative reward of the model. The algorithm will explore and learn the environment, and adjust its own strategy according to the reward signal fed back. The reward signal will take into account the power consumption generated by the air conditioning refrigeration and the temperature condition of the room equipment. Under the control of the multi-agent reinforcement learning algorithm, multiple air conditioners can share their control strategies, thereby realizing collaborative refrigeration and improving the refrigeration efficiency of the data center. Compared with other data center refrigeration control schemes, the present invention has strong scalability, easy deployment, and can achieve better refrigeration effects.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic control, and in particular relates to a distributed intelligent control method for air conditioning in a data center. Background Art

[0002] In data centers, in addition to the necessary energy consumption generated by IT system operations, the refrigeration system used to cool down the equipment in the data center also consumes a huge amount of energy. Therefore, there is still a lot of room for improvement in the energy saving of data center refrigeration systems.

[0003] At present, the energy consumption optimization problem of the cooling system in the data center can be solved by cooling control through the optimization model method. However, the model obtained by the model predictive control method will have errors, and when the model is rolled forward, the model error will continue to accumulate, making the final solution of the model lose the optimal solution. In addition, MPC needs to continuously iterate to solve the optimization problem, and the solution process is relatively time-consuming. In addition, the calibrated model also needs to be recalibrated when applied to a new environment.

[0004] Deep reinforcement learning can avoid the problems encountered in model optimization by using the "trial and error" optimization method through interaction with the environment, and has a good performance in cooling management. The application of deep reinforcement learning control methods in cooling system applications has shown certain effectiveness, but ordinary deep reinforcement learning methods usually only focus on smaller-scale cooling control problems. For example, when only one controller can be used to achieve control. In distributed data centers, when the number of cooling equipment increases, using only one controller will increase the dimension of the action space of reinforcement learning, resulting in the problem of dimensional explosion. In addition, in large-scale data centers, the temperature of the computer room at the next moment is affected by the air supply temperature and air supply speed of all air conditioners. Due to the dynamic nature of the temperature and air flow distribution in the data center, the environment will become unstable from the perspective of each intelligent agent at this time, and in an unstable environment, the reinforcement learning algorithm may not converge. In order to achieve better control effects, the control of multiple air conditioners requires a coordination mechanism to enable the air conditioners to cooperate with each other to achieve energy consumption optimization. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a distributed intelligent control method for data center air conditioners, which can intelligently adjust the air supply speed and air supply temperature of the computer room air conditioners by observing the computer room environment, so that the equipment can operate at a safe temperature. And through the cooperative cooling between air conditioners, the energy consumption of the computer room refrigeration system is reduced, thereby improving the energy efficiency of the refrigeration system.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A distributed intelligent control method for data center air conditioning comprises the following steps:

[0008] Step 1: monitor the air inlet temperature of each rack in the data center and collect all monitoring data so that each air conditioner can observe the air inlet temperature of all racks;

[0009] Step 2: Take each air conditioner as an intelligent agent and model the data center cooling control problem as a Markov game model. The model includes the number of intelligent agents n, the environment state S, the action space A, the reward function R, and the discount factor γ. The intelligent agent represents the decision maker in the Markov game model, that is, the air conditioner of the data center. Therefore, the number of air conditioners in the present invention is also n. The environment state S is used to describe the environment of the current model. The size of the discount factor γ determines the size of the impact of the expected result on the current decision.

[0010] The state space of the i-th agent is represented by S i =(tran 1 ,tran 2 ,...,tran m ), m is the number of racks, tran m is the air inlet temperature of the mth rack;

[0011] The action space of the i-th agent is represented as A i , when executing an action, select action a from the action space i =(f i ,t i ),a i ∈A i , f i and t i represent the air supply speed and air supply temperature of the i-th air conditioner, respectively, and f i and t i Adjustable within the set range;

[0012] The reward function R of the i-th agent i The temperature default penalty function cost and the corresponding air conditioning power consumption power i composition;

[0013] The actions between agents should avoid conflicts and obtain greater long-term rewards through cooperation. The optimization problem is expressed as

[0014]

[0015] Among them, γ t is the discount factor of the reward after t steps, t means t steps have been executed backward from the start of the algorithm, R t+1 Represents the reward obtained by interacting with the environment at time t+1;

[0016] By solving the model optimization problem, the optimal action is obtained, that is, the optimal set point of the air supply speed and air supply temperature of the air conditioner, with the goal of reducing the cooling power consumption of the data center room as the optimization goal, and while reducing the cooling power consumption, ensure that the equipment can operate normally at a safe temperature;

[0017] Step 3, solve the model, explore and learn the rack's air inlet temperature and air conditioning energy consumption, dynamically adjust the air supply speed and air supply temperature of the air conditioner, and obtain an optimal strategy that can both reduce air conditioning energy consumption and meet cooling needs, and implement data center cooling based on this strategy.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention models the complex thermal effect model of the data center, uses a multi-agent reinforcement learning algorithm, and adaptively adjusts the air supply speed and air supply temperature of the air conditioner through interaction with the environment. The algorithm can be deployed by adding a control module to the original refrigeration equipment. Compared with other data center refrigeration control solutions, the present invention has strong scalability, easy deployment, and can achieve better refrigeration effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Design and deploy diagrams for data center environments.

[0021] Figure 2 This is the algorithm flow chart. DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0023] The present invention is a distributed intelligent control method for data center air conditioners. A Markov game model is established for the complex thermal effects inside the data center computer room, and the air supply speed and air supply temperature of the air conditioner are controlled by a multi-agent reinforcement learning solution model. The optimal strategy is determined in the process of continuous interaction with the environment, and the optimal strategy maximizes the cumulative reward of the model. The algorithm will explore and learn the environment, and adjust its own strategy based on the reward signal fed back. The reward signal will take into account the power consumption generated by the air conditioning refrigeration and the temperature conditions of the room equipment. In addition, under the control of the multi-agent reinforcement learning algorithm, multiple air conditioners can share their control strategies, thereby achieving collaborative refrigeration and improving the refrigeration efficiency of the data center.

[0024] Figure 1This is a detailed deployment implementation diagram of the present invention. There are m racks in the data center room. Sensors are arranged at the air inlet of each rack to monitor the rack air inlet temperature, that is, the air intake temperature of each rack, and n air conditioners are set up for room cooling. The cold air produced by the air conditioner will be sent to the vicinity of the rack through the cold channel. The rack air inlet inhales cold air, and the hot air is discharged from the back of the rack and recovered through the hot channel. The cooling capacity in the cold channel is determined by the air supply temperature and air supply speed of the air conditioner, and the cold channel is not closed.

[0025] The distributed intelligent control method of the air conditioner of the present invention runs on the PC side, and the PC side is connected to the air conditioner through a microcontroller, so as to modify the air supply speed and air supply temperature of the air conditioner. The sensor readings on the rack can be directly transmitted to the PC side, and the PC side can collect all monitoring data, so that each air conditioner can observe the air inlet temperature of all racks. It is easy to understand that the distributed intelligent control method of the air conditioner of the present invention can also be run on a mobile terminal with sufficient computing power, or a remote cloud.

[0026] The control method specifically includes the following parts:

[0027] 1. Take each air conditioner as an intelligent agent and model the data center cooling control problem as a Markov game model. The model includes the number of intelligent agents n, the environmental state S, the action space A, the reward function R and the discount factor γ. The intelligent agent represents the decision maker in the Markov game model, that is, the air conditioner of the data center. Therefore, the number of air conditioners in the present invention is also n. The environmental state S is used to describe the environment of the current model. The size of the discount factor γ can determine the impact of the expected results on the current decision. The tasks processed by the model are classified into fully cooperative tasks, fully competitive tasks and mixed tasks. Obviously, the present invention models the above problem as a Markov game model of a fully cooperative task. In the cooperative problem, the actions between intelligent agents should avoid conflicts and obtain greater long-term returns through collaboration.

[0028] Specifically, for the i-th agent, its modeling is as follows:

[0029] State space: The temperature of the rack air inlet directly affects the cooling effect of the rack equipment. Taking the sensor reading as the state, the state space of the i-th agent can be expressed as S i =(tran 1 ,tran 2 ,...,tran m ), tran represents the temperature reading of the sensor, tran mis the inlet temperature of the mth rack. The temperature of the rack air inlet is modeled as a state, and the temperature of the air inlet is obtained using a temperature sensor. By collecting all sensor monitoring data, each air conditioner can observe the temperature of the air inlet of all racks.

[0030] Action space: The cooling power of the air conditioner is controlled by controlling the air supply temperature and air supply speed, so the air supply temperature and air supply speed are used as the action space of the Markov game. Each air conditioner has its own action space, and the action space of the i-th agent is represented by A i , which starts from A when executing the action i Select action a i =(f i ,t i ),a i ∈A i .f i and t i represent the air supply speed and air supply temperature of the i-th air conditioner, respectively, and f i and t i It is adjustable within a certain range, so the action space A i is continuous within this range. Let f i low and f i high are the minimum and maximum air supply speeds of the i-th air conditioner, respectively. and are the minimum and maximum air supply temperatures of the i-th air conditioner, respectively, then:

[0031] f i low ≤f i ≤f i high

[0032]

[0033] Reward function: The goal of reinforcement learning is to obtain the maximum cumulative reward, while the goal of the cooling system is to minimize the power consumption of the cooling system. Therefore, in order to facilitate learning, the reward function is set to negative. The reward function is divided into the power consumption of the cooling system and the penalty function when the rack air inlet temperature exceeds the threshold temperature. The i-th air conditioner performs action a i =(f i ,t i ) after which it consumes power i Related to its action, it is expressed as:

[0034] power i =f(a i )

[0035] Function f represents the linear relationship between the air supply speed and air supply temperature of the air conditioner and the power consumption generated by the air conditioner.

[0036] When the room temperature exceeds the safety threshold temperature of the equipment, it will affect the normal operation of the equipment. For actions that exceed the safety threshold temperature, a corresponding penalty should be set, which is defined as a temperature default penalty function. The temperature default penalty function cost is only related to the size of the rack inlet temperature exceeding the threshold temperature, so the temperature default penalty function cost of each air conditioner is the same. The larger the cost value, the more times the temperature default occurs and the greater the degree of default. The cost is expressed as:

[0037]

[0038] t threshold Indicates the safety threshold temperature of the device, tran i Indicates the temperature detected by the sensor on a rack after the air conditioner performs the current action, that is, the inlet air temperature of the rack. o <=t threshold When , it means that the inlet air temperature of the oth rack has not violated the contract, and the penalty function is 0. o >t threshold When the temperature of the air inlet of the oth rack is too high and exceeds the safe temperature of the equipment, the penalty is tran o -t threshold , the accumulation of all rack default penalties is the penalty function.

[0039] Therefore, the reward function R of the i-th agent of the present invention is i The temperature default penalty function cost and the corresponding air conditioning power consumption power i Composition, expressed as:

[0040] R i =-[αcost+(1-α)power i ×w]

[0041] α represents cost and power i The weight in the reward, in order to ensure power consumption i The power consumption of the air conditioner is in the same order of magnitude as the cost value, using w as the scaling factor. i to zoom in or out.

[0042] Furthermore, the joint action value function Q(s,a) can be calculated based on the reward function R:

[0043]

[0044] where s nextIndicates the state at the next moment, a is the action a of each air conditioner i The joint action composed of can use Q(s,a) to guide the update of the policy network.

[0045] In the above problem model, air conditioners, as the main cooling equipment in data centers, consume a lot of energy when cooling other equipment. Therefore, the algorithm mainly optimizes the air supply temperature and air supply speed of air conditioners for energy saving. By solving the problem model, the optimal action is obtained, that is, the optimal set point of air supply speed and air supply temperature of the air conditioner, with the goal of reducing the cooling power consumption of the data center room. And while reducing the cooling power consumption, it is necessary to ensure that the equipment can operate normally at a safe temperature. The air conditioner reduces power consumption and causes the temperature of the equipment to rise. When the temperature exceeds the preset safety threshold temperature, the system will be punished accordingly. Therefore, the optimization problem can be expressed as:

[0046]

[0047] Among them, γ t is the discount factor of the reward after t steps, t means t steps have been executed backward from the start of the algorithm, R t+1 Represents the reward obtained by interacting with the environment at time t+1;

[0048] 2. Solve the model, explore and learn the rack's air inlet temperature and air conditioning energy consumption, dynamically adjust the air supply speed and air supply temperature of the air conditioner, and obtain an optimal strategy that can both reduce air conditioning energy consumption and meet cooling needs, and implement data center cooling based on this strategy.

[0049] To solve this model, a multi-agent reinforcement learning algorithm can be used to interact with the environment and convert the obtained sample transition (s, a, r, s next ) is stored, a neural network is used to approximate Q(s,a), and finally the behavior is output through the policy network.

[0050] The multi-agent reinforcement learning algorithm of the present invention adopts the Actor-Critic framework, and two neural networks are used to approximate the Actor network and the Critic network respectively. The Critic network is used to output the Q value, and the Actor network is used to output the behavior. Each agent follows the ε-greedy algorithm, and its respective Actor network takes the current state as input and outputs the corresponding action. Since the Actor outputs a certain action, the Actor network outputs the corresponding air supply speed and air supply temperature a of the corresponding air conditioner in the current state with a probability of (1-ε). i =(f i ,t i ), and generate random actions a for exploration with probability ε i '=(f i',t i '); The air conditioner executes the action generated by the Actor network and obtains the next state S next , according to a i =(f i ,t i ) and S next Calculate the instantaneous reward, and finally use its current state s, joint action a = {a 1 ,a 2 ,...a i ...,a n}、Next state s next The instantaneous reward r is stored in the experience replay pool, and the current state is transferred to the next state for the next round of sampling.

[0051] That is, the Actor network takes the state as input and outputs the corresponding air supply temperature and air supply speed of the air conditioner, and then obtains the environmental state and reward composition sample transition (s, a, r, s next ), and store it in the cache, and use the neural network to approximate the action value function, that is, the neural network updates its parameters by sampling from the cache, and outputs the behavior through the policy network. The algorithm is designed as follows:

[0052] The online network and the target network are introduced. The online network and the target network have the same structure but different parameters. Through these two networks with the same structure but different parameters, the targetQ value is kept fixed for a period of time, thereby reducing the correlation between the onlineQ value and the targetQ value and improving the stability of the algorithm. The online network is represented by μ and the target network is represented by μ'.

[0053] When a certain number of samples are accumulated, training begins. During training, each agent extracts a certain number of sample transitions (s, a, r, s next ). The loss of the Critic network and the Actor network is calculated through samples and the parameters of the online network are updated. The update formula of the Critic network for the loss function is:

[0054]

[0055] θ i represents the parameters of the Critic network of the i-th agent, P represents the number of samples, j represents the j-th sample sampled from the experience replay pool during training, and s j represents the state in the jth sample, represents the cooling strategy adopted by the i-th air conditioner in the j-th sample, n represents the number of air conditioners; μ represents the use of the online network, represents the onlineQ value output by the i-th air conditioner; y j Represents the targetQ value obtained by the decayed value function of the current instantaneous reward and the next moment, y j The calculation formula is:

[0056]

[0057] represents the reward obtained by the i-th air conditioner after executing the cooling strategy in the j-th sample, μ' represents the use of the target network, represents the targetQ value output by the i-th air conditioner; s' j Indicates the next state, and the next action is output by the target Actor network of each air conditioner, that is Represents the global state. s' j The physical meaning of is: the air inlet temperature monitored by the rack sensor after the air conditioner executes its own cooling strategy (i.e., the next state), and μ' is the neural network used by the air conditioner to generate the corresponding cooling strategy in the next state, which is the target network here.

[0058] Taking the air conditioner power consumption and rack inlet temperature exceeding the limit as rewards, the neural network used by the air conditioner to generate the cooling strategy updates the neural network by finding the gradient of its rewards. That is, the Actor network updates the parameters by finding the gradient of the cumulative rewards. The cumulative rewards can be expressed as

[0059]

[0060] The cumulative reward is that the state follows a certain distribution ρ π When , the expected value of the reward obtained by executing the action is obtained, and the gradient of the strategy parameters is calculated to obtain the update formula of the Actor network:

[0061]

[0062] in represents the parameters of the Actor network, P represents the number of samples, μ i Used according to the status The act of producing air conditioning.

[0063] The algorithm of the present invention uses a centralized training method to update the strategy of each air conditioner. Through the multi-agent reinforcement learning algorithm, the present invention can centrally learn the refrigeration strategies of all air conditioners. That is, during the air conditioner training, the refrigeration strategy of each air conditioner will be saved, and each air conditioner will obtain the refrigeration strategies of other air conditioners. That is to say, when each air conditioner learns its own refrigeration strategy, it will take into account the refrigeration strategies of all air conditioners and then learn its own refrigeration strategy, and update its own strategy through the joint behavior composed of all air conditioner refrigeration strategies. In this way, when adjusting their own strategies, the air conditioners will take into account the strategies of other air conditioners, thereby realizing the collaboration of refrigeration of multiple air conditioners.

[0064] like Figure 2 , each air conditioner Actor network will generate its own corresponding air supply temperature and air supply speed according to the current room temperature (sensor reading). When each air conditioner calculates the Q value of the current behavior, the behavior of other air conditioners is also considered, so the joint behavior is used to calculate the Q value, and then the joint Q value is used to update each Actor network.

[0065] The operating logic of the multi-agent reinforcement learning algorithm is as follows:

[0066] The first step is to initialize the data center computer room environment, and the air conditioner runs with the safest cooling strategy. Initialize the neural network that generates the air conditioner strategy.

[0067] In the second step, the air conditioner executes the cooling strategy and interacts with the computer room to obtain the temperature at the air inlet of the rack, which is the status.

[0068] The third step is that each air conditioner generates its own cooling strategy (the cycle starts and is cycled for each air conditioner).

[0069] Step 4: Use the rack air inlet temperature as the neural network input and output the cooling strategy a of each air conditioner i . (End of loop)

[0070] Step 5: Execute air conditioning cooling strategy a = {a 1 ,a 2 ,...a i ...,a n}.

[0071] Step 6: The sensor deployed at the air inlet of the rack obtains the temperature reading, which is the next state s next And calculate the instantaneous reward r.

[0072] Step 7: Transition (s, a, r, s next ) is stored in the experience replay pool D.

[0073] Step 8. The environmental state is transferred to the next state, and the air conditioner regenerates the cooling strategy according to the state.

[0074] Step 9. When a certain amount of experience is generated by interacting with the data center environment, the neural network for generating cooling strategies for the air conditioner is trained (the cycle starts and is cycled for each air conditioner).

[0075] Step 10: Randomly sample P samples from the experience replay pool D.

[0076] Step 11: Input the next state into the air conditioner's strategy network to generate the next cooling strategy.

[0077] Step 12: According to the current state, the cooling strategies of all current air conditioners and the cooling strategies of all air conditioners in the next state and the next moment, obtain the corresponding current Q value and target Q value.

[0078] Step 13. Calculate the loss function of the neural network used by the conditioning to generate the behavior.

[0079] Step 14: Update the parameters of the neural network (end of the loop).

[0080] In summary, the present invention establishes a Markov game model inside the computer room of the data center, and uses a multi-agent reinforcement learning algorithm to solve the model to control the air supply speed and air supply temperature of the air conditioner. The Markov game model consists of five parts: the number of agents, the state of the environment, the action space, the reward function, and the discount factor. The optimal strategy is determined in the process of continuous interaction with the environment, and the optimal strategy maximizes the cumulative reward of the model. The algorithm will explore and learn the environment, and adjust its own strategy according to the reward signal fed back. The reward signal will take into account the power consumption generated by the air conditioning refrigeration and the temperature condition of the room equipment. Under the control of the multi-agent reinforcement learning algorithm, multiple air conditioners can share their control strategies, thereby realizing collaborative refrigeration and improving the refrigeration efficiency of the data center. Compared with other data center refrigeration control schemes, the present invention has strong scalability, is easy to deploy, and can achieve better refrigeration effects.

Claims

1. A distributed intelligent control method for data center air conditioning, characterized in that: The steps include: Step 1: monitor the air inlet temperature of each rack in the data center and collect all monitoring data so that each air conditioner can observe the air inlet temperature of all racks; Step 2: Take each air conditioner as an agent and model the data center cooling control problem as a Markov game model. The model includes the number of agents n, the environment state S, the action space A, the reward function R, and the discount factor γ. The agent represents the decision maker in the Markov game model, that is, the air conditioner in the data center. The environment state S is used to describe the environment of the current model. The size of the discount factor γ determines the impact of the expected result on the current decision. The state space of the i-th agent is represented by S i =(tran1,tran2,...,tran m ), m is the number of racks, tran m is the air inlet temperature of the mth rack; The action space of the i-th agent is represented by A i , when executing an action, select action a from the action space i =(f i ,t i ),a i ∈A i , f i and t i represent the air supply speed and air supply temperature of the i-th air conditioner, respectively, and f i and t i Adjustable within the set range; Action space A i Continuous within the stated setting range, expressed as: f i low ≤f i ≤f i high f i low and f i high are the minimum and maximum air supply speeds of the i-th air conditioner, respectively. and are the minimum and maximum air supply temperatures of the i-th air conditioner respectively; The reward function R of the i-th agent i The temperature default penalty function cost and the corresponding air conditioning power consumption power i The temperature default penalty function cost is the corresponding penalty set for the action exceeding the safety threshold temperature. The temperature default penalty function cost of each air conditioner is the same. The larger the cost value, the more times the temperature default occurs and the greater the degree of default. The cost is expressed as: t threshold Indicates the safety threshold temperature, tran i Indicates the air inlet temperature of a rack after the air conditioner executes the current action. o <=t threshold When , it means that the inlet air temperature of the oth rack has not violated the contract, and the penalty function is 0. o >t threshold When , it means that the air inlet temperature of the oth rack exceeds the safety threshold temperature, and the penalty is tran o -t threshold , the sum of all rack default penalties is the penalty function; The air conditioning power consumption i The i-th air conditioner is performing action a i =(f i ,t i ) after the power consumption is expressed as: power i =f(a i ) Function f represents the linear relationship between the air supply speed and air supply temperature of the air conditioner and the power consumption generated; The reward function R of the i-th agent i Expressed as R i =-[αcost+(1-α)power i ×w] α represents cost and power i The weight of the reward, w is the power i Scaling factor, scaling to ensure power i On the same order of magnitude as cost; The actions between agents should avoid conflicts and obtain greater long-term rewards through cooperation. The optimization problem is expressed as Among them, γ t is the discount factor of the reward after t steps, t means t steps have been executed backward from the start of the algorithm, R t +1 Represents the reward obtained by interacting with the environment at time t+1; By solving the model optimization problem, the optimal action is obtained, that is, the optimal set point of the air supply speed and air supply temperature of the air conditioner, with the reduction of the cooling power consumption of the data center room as the optimization goal, and while reducing the cooling power consumption, the equipment can operate normally at a safe temperature; Step 3, solve the model, explore and learn the rack's air inlet temperature and air conditioning energy consumption, dynamically adjust the air supply speed and air supply temperature of the air conditioner, and obtain an optimal strategy that can both reduce air conditioning energy consumption and meet cooling needs, and implement data center cooling based on this strategy.

2. The distributed intelligent control method for data center air conditioning according to claim 1 is characterized in that: In step 2, the joint action value function Q(s,a) is calculated according to the reward function R: where s next Indicates the state at the next moment, a is the action a of each air conditioner i The joint action composed of Q(s,a) is used to guide the update of the policy network.

3. The distributed intelligent control method for data center air conditioning according to claim 2 is characterized in that: In step 3, a multi-agent reinforcement learning algorithm is used to interact with the environment and the obtained sample transition (s, a, r, s next ) storage, use a neural network to approximate Q(s,a), and output the behavior through the policy network.

4. The distributed intelligent control method for data center air conditioning according to claim 3 is characterized in that: The multi-agent reinforcement learning algorithm uses an Actor-Critic framework, in which the Critic network is used to output Q values ​​and the Actor network is used to output behaviors. Each agent uses its own Actor network to take the current state as input and output the corresponding air supply speed and air supply temperature a of the corresponding air conditioner in the current state with a probability of (1-ε) according to the ε-greedy algorithm. i =(f i ,t i ), and generate random actions a for exploration with probability ε i '=(f i ',t i '); The air conditioner executes the action generated by the Actor network and obtains the next state S next , according to a i =(f i ,t i ) and S next Calculate the instantaneous reward, and finally transform its current state s, joint action a={a1,a2,...a i ...,a n }、Next state s next The instantaneous reward r is stored in the experience replay pool, and the current state is transferred to the next state for the next round of sampling.

5. The distributed intelligent control method for data center air conditioning according to claim 4 is characterized in that: The multi-agent reinforcement learning algorithm introduces an online network and a target network. The online network and the target network have the same structure but different parameters. The targetQ value is kept fixed over a period of time through the online network and the target network, thereby reducing the correlation between the onlineQ value and the targetQ value and improving the stability of the algorithm.

6. The distributed intelligent control method for data center air conditioning according to claim 5, characterized in that: In the multi-agent reinforcement learning algorithm, during training, each agent extracts a certain number of sample transitions (s, a, r, s next ), calculate the loss of the Critic network and the Actor network through samples and update the parameters of the online network. The update formula of the Critic network for the loss function is: θ i represents the parameters of the Critic network of the i-th agent, P represents the number of samples, j represents the jth sample sampled from the experience replay pool during training, s j represents the state in the jth sample, represents the cooling strategy adopted by the i-th air conditioner in the j-th sample, n represents the number of air conditioners; μ represents the use of the online network, represents the onlineQ value output by the i-th air conditioner; y j Represents the targetQ value obtained by the decayed value function of the current instantaneous reward and the next moment, y j The calculation formula is: r i j represents the reward obtained by the i-th air conditioner after executing the cooling strategy in the j-th sample, μ' represents the use of the target network, represents the targetQ value output by the i-th air conditioner; s' j Indicates the next state, and the next action is output by the target Actor network of each air conditioner, that is Represents the global state; The Actor network updates its parameters by taking the gradient of the cumulative reward. The cumulative reward is expressed as The cumulative reward is that the state follows a certain distribution ρ π When , the expected value of the reward obtained by executing the action is obtained, and the gradient of the strategy parameters is calculated to obtain the update formula of the Actor network: in represents the parameters of the Actor network, P represents the number of samples, μ i Used according to the state i j The act of producing air conditioning.

7. The distributed intelligent control method for data center air conditioning according to claim 6, characterized in that: The multi-agent reinforcement learning algorithm uses a centralized training method to update the strategy of each air conditioner. During the training, each air conditioner will obtain the cooling strategies of other air conditioners and update their respective strategies through the joint behavior composed of all air conditioner cooling strategies.

Images