A data center resource dormancy and scheduling method and system based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's data center resource dormancy and scheduling method and system, belong to cloud computing system data center energy-saving optimization technical field, the method includes the following steps: S1, establishes simulation model, determines the intelligent agent and environment of research object;S2, based on simulation model and data center server information establishes MDP model;S3, based on MDP model, develops Deep MAHHQN algorithm, trains intelligent agent, updates policy network and value network, establishes and forms preliminary resource dormancy scheduling strategy;S4, preliminary control strategy is deployed to actual data center, guides each server intelligent agent of data center to carry out scheduling decision to data center server switch and power management, and with actual environment is interacted, continuously updates and optimizes to scheduling strategy, until reaching expected simulation result.The application can meet enough according to the actual situation of data center dynamic adjustment resource allocation, maximum degree improves energy utilization efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cloud computing system data center energy saving optimization, and specifically relates to a data center resource dormancy and scheduling method and system based on reinforcement learning. BACKGROUND

[0002] With the rapid development of information technology, data centers as the key infrastructure supporting cloud computing, big data and other applications, their resource management efficiency has become a concern. The energy consumption of data centers is growing, and effective resource scheduling methods are essential to improve energy efficiency. Therefore, a data center resource dormancy scheduling method is needed to optimize the use of data center resources and reduce energy consumption.

[0003] Existing data center resource scheduling methods are often limited to static resource allocation and cannot adapt to changes in dynamic workloads. Traditional methods are usually based on heuristic rules and lack adaptability to system dynamics. Therefore, a method is needed that can adjust resource allocation in real time to better adapt to the dynamic environment of data centers. SUMMARY

[0004] To solve the above problems, the application provides a data center resource dormancy and scheduling method and system based on reinforcement learning, which can dynamically adjust the allocation of resources according to the actual situation of the data center, maximize energy efficiency, and reduce energy consumption.

[0005] A data center resource dormancy and scheduling method based on reinforcement learning includes the following steps:

[0006] S1, collect relevant parameters required by the simulation model, establish a data center room simulation model, and determine the agent and environment of the research object;

[0007] S2, based on the actual data center room situation, establish a Markov decision process model, and determine its parameters through experience;

[0008] S3, based on the Markov decision process model constructed in S2, train the Deep MAHHQN network in combination with the simulation environment, update the parameters in the Deep MAHHQN network, and form a preliminary resource dormancy scheduling strategy;

[0009] S4, deploy the preliminary resource dormancy scheduling strategy obtained by training in S3 to the actual data center, guide the data center server agent to make scheduling decisions on data center server power management, and interact with the actual environment to continuously update and optimize the scheduling strategy until the training effect reaches the expected simulation result.

[0010] Further, step S1 includes the following steps:

[0011] S101, collecting required data according to actual data center machine room conditions;

[0012] S102, using a 6sigma simulator to establish a data center machine room simulation model according to the data collected in S101, and determining agents and environments in the reinforcement learning algorithm.

[0013] Further, step S2 includes the following steps:

[0014] S201, establishing a Markov decision process state space model, and the overall state space being composed of state spaces observed by each agent;

[0015] S202, establishing a Markov decision process action space model, and the overall action space being composed of actions of each agent;

[0016] S203, establishing a Markov decision process reward function model.

[0017] Further, in step S203, the Markov decision process reward function model expression is:

[0018]

[0019] In the formula, r t is the reward value of each agent at time t, refers to the reward obtained by agent i at time t by performing action A t , c t is the real-time electricity price at time t, P i t is the power consumption of server i at time t, λ1 is a temperature penalty coefficient, λ2 is a service quality penalty coefficient, and λ3 is a service load processing penalty coefficient; in addition, v T,i t is the absolute value of server i exceeding its thermal comfort range at time t, v D,i t is the value of the average task response time of server i at time t exceeding the upper limit of the service quality agreement, v W t is the amount of tasks not processed at time t.

[0020] Further, step S3 includes the following steps:

[0021] S301, initialize parameters of all high-level local value function networks in the Deep MAHHQN network and corresponding target network parameters thereof; initialize parameters of the high-level global value function network and corresponding target network parameters thereof; initialize low-level policy function network parameters, low-level value function network parameters and corresponding target network parameters thereof; initialize the experience replay pool;

[0022] S302, initialize the global state;

[0023] S303, update the global state of the environment according to the actions performed by the agents of each server;

[0024] S304, determine whether the experience replay pool array is sufficient: if yes, perform step S305, and if no, perform step S308;

[0025] S305, take out a number of experience arrays from the experience replay pool;

[0026] S306, train and update the high-level network using the experience arrays;

[0027] S307, train and update the low-level network parameters using the experience arrays;

[0028] S308, determine whether the training is finished according to the time value in the current state: if yes, perform step S309, and if no, jump to step S301;

[0029] S309, determine whether the training should be finished according to the benefit output by the current policy: if yes, perform S4, and if no, jump to step S302.

[0030] Further, step S303 includes the following steps:

[0031] S3031, the agent i of each server determines a discrete action k i t selects a discrete action k i t by the ε-greedy policy, and the server performs the discrete action to determine whether to start up; i t is calculated by the following formula:

[0032]

[0033] wherein ε represents a very small positive number, Q i is the high-level local value function, ω i h is the parameter of the high-level local value function Q i (ω i h ) network.

[0034] S3032, Change the currently observed state o i t and discrete action k i t Packaged into The input is then fed into the low-level policy network of each agent i to obtain continuous actions. The workload of the tasks assigned to a server is determined by the series of actions it performs.

[0035] S3033. Calculate the current time-time reward r based on the reward function model in S2. t The global state is updated to S. t+1 ; the quintuple Stored into the experience replay pool, where S t This represents the global state at the current moment. Let χ be the set of all discrete actions of the agents at time t. t Let be the set of all continuous actions of the agents at time t.

[0036] Furthermore, step S306 includes the following steps:

[0037] S3061. Input all continuous actions together with the current state into the super network to generate the parameters of the hybrid network, i.e. the weights of each local function. Input the state and actions observed by each agent into the high-level local value network of the agent to obtain the high-level local value of each agent.

[0038] S3062. Input the local value functions of each high-level layer into the value network of the hybrid network, output the global value function of the high-level layer, and obtain the output value of the high-level layer;

[0039] S3063. Use the target network of the hybrid network to perform time difference prediction to obtain the TD target;

[0040] S3064. Calculate the time-series difference error σ of the high-level network. t ;

[0041] S3065. Update the values ​​of the high-level local value network, the hybrid network, and the parameters of the target network using the TD error, as follows:

[0042]

[0043] Where, ω mix,new For the parameters updated by the hybrid value network, ω mix,now The parameters before the hybrid value network update. To mix the updated parameters of the target network, To mix the parameters of the target network before updating, ω mix For high-level global value function parameters of the network, S t the global state at the current time, the set of all discrete actions of all agents at time t, χ t the set of all continuous actions of all agents at time t; ω h i,new the updated parameters of the high-level local value network of agent i, ω i h the high-level local value function Q i (ω i h parameters of the network, the updated parameters of the high-level goal network of agent i, ω h i,now the parameters of the high-level local value network of agent i before updating, the parameters of the high-level goal network of agent i before updating, ζ is the learning rate of the high-level value network; τ is the update rate of the goal network.

[0044] Further, step S307 includes the following steps:

[0045] S3071, input all continuous actions performed by all agents to the low-level value network of each agent, and use the low-level value network to make q value prediction as follows:

[0046]

[0047] wherein, is the low-level value network, q l,t i is the low-level network predicted q value of agent i at time t, S t the global state at the current time, the set of all discrete actions of all agents at time t, χ t the set of all continuous actions of all agents at time t; the parameters of the low-level value network at the current time;

[0048] S3072, input the action set to the goal network of each agent to make TD prediction as follows:

[0049]

[0050] wherein, y l,t i is the low-level network TD target value of agent i at time t, r t is the reward value of each agent at time t, γ is the discount factor, S t+1 the updated global state, the low-level target network parameter at the current time;

[0051] S3073, calculating q l,t i and y l,t i between the low-level network TD error;

[0052] S3074, updating the value network and the target network parameter of the low-level network using the following formula:

[0053]

[0054] wherein, is the updated low-level value network parameter of the agent i, is the low-level value network parameter of the agent i before updating, ω i l is the low-level value function network parameter ω i l , is the updated low-level target network parameter of the agent i, is the low-level target network parameter of the agent i before updating, α is the learning rate of the low-level value network, σ t is the high-level network time difference error, τ is the learning rate of the target network;

[0055] S3075, based on the updated value network and the target network parameter of the low-level network, the agent i updates the parameter of the policy network μ i (θ i ) of the agent i, and the expression is as follows:

[0056]

[0057] wherein, β is the learning rate of the policy network update, θ i,now is the current parameter of the policy network of the agent i, θ i is the parameter of the policy network of the agent i, is the currently observed state o i t and the discrete action k i t to form a new state, is the continuous action, θ i,new is the parameter of the updated policy network.

[0058] Further, step S4 comprises the following steps:

[0059] S401, deploy the preliminary resource dormancy scheduling strategy learned by the simulation environment training to the actual data center, and use the preliminary resource dormancy scheduling strategy to guide each server to perform actions according to the observed actual environment state;

[0060] S402, for the preliminary resource dormancy scheduling strategy, continuously train each server agent with reference to the training steps of S3 until the output benefit reaches the expected effect;

[0061] S403, execute discrete actions and continuous actions according to the trained policy network and value network to control the server power control module and the workload distribution module to perform dormancy and scheduling operations on the server.

[0062] A data center resource dormancy scheduling system based on multi-agent reinforcement learning, comprising

[0063] An information perception system for collecting relevant parameters required by the simulation model;

[0064] A simulator for establishing a data center computer room simulation model according to the relevant parameters required by the simulation model, determining the agent and the environment of the research object;

[0065] A perception control module for establishing a Markov decision process model based on the actual data center computer room situation and determining its parameters through experience; training the Deep MAHHQN network based on the Markov decision process model constructed in S2, updating the parameters in the Deep MAHHQN network, and forming a preliminary resource dormancy scheduling strategy;

[0066] An actual deployment module for deploying the preliminary resource dormancy scheduling strategy to guide the scheduling decision of the data center server on-off and power management of each server agent in the data center, and interacting with the actual environment to continuously update and optimize the scheduling strategy until the training effect reaches the expected simulation result.

[0067] Compared with the prior art, the present application has at least the following beneficial technical effects:

[0068] The method provided by the present application can fully dynamically perceive the space-time coupling characteristics of the data center. In the data center resource dormancy scheduling, the cold flow, task flow and energy flow are introduced into the Markov decision process (MDP, Markov Decision Process) model of reinforcement learning, so that the algorithm can dynamically perceive the space-time coupling characteristics of environment-energy-business. This means that the agent can respond to changes in the environment in real time, not only considering the current state, but also considering the evolution of the state over time and space. By introducing these factors, the algorithm can better adapt to the complex and dynamic operating conditions in the data center, improving the accuracy and adaptability of decision-making.

[0069] Further, through the Deep MAHHQN, a multi-agent reinforcement learning can be achieved to interact with the computing resources and the running environment. The multi-agent reinforcement learning method can achieve real-time interaction between the server and the environment, avoiding the need for complex modeling and high prior knowledge. Each server acts as an agent, learning the optimal behavior strategy through continuous interaction with the environment. This real-time interaction helps to adapt to changing workloads and environmental conditions in actual operation, improving the flexibility and performance of the system. At the same time, the collaborative learning among multiple agents enables the entire system to learn more global knowledge from distributed information, further improving the intelligence and coordination of decision-making.

[0070] Further, the present application proposes a hierarchical processing collaborative scheduling method, which realizes the collaborative scheduling of server on-off and workload allocation by adopting a hierarchical processing method of discrete-continuous hybrid action space. The high-level network is responsible for processing discrete actions, i.e. the on-off state of the server, while the low-level network processes continuous parameters related to the selected discrete action, such as power level. This hierarchical processing effectively reduces the computational complexity, making the algorithm more efficient in real-time decision-making. In addition, this collaborative scheduling ensures the consistency between the on-off state of the server and the allocation of the workload, optimizing the resource utilization of the entire data center while meeting various constraints.

[0071] Further, the present method can realize the comprehensive benefits of economy, energy saving, green and equipment stability. The algorithm takes into account the device thermal comfort constraints, QoS constraints, business constraints and operating costs, which can help the data center to realize cost reduction and efficiency improvement while ensuring service quality, and improve environmental friendliness.

[0072] In summary, the use of Deep MAHHQN algorithm for data center resource hibernation scheduling brings many advantages. First, by introducing cold flow, task flow and energy flow into the MDP model, dynamic perception of space-time coupling characteristics is realized, making the algorithm more adaptable and accurate. Second, multi-agent reinforcement learning realizes real-time interaction between the server and the environment, avoiding complex modeling of mechanisms and dependence on high prior knowledge, improving the flexibility and performance of the system. In addition, through the hierarchical processing collaborative scheduling, the computational complexity is effectively reduced, making the algorithm more efficient in real-time decision-making. Most importantly, considering factors such as device thermal comfort constraints, QoS constraints, business constraints and operating costs, the optimized resource hibernation scheduling not only improves the environmental friendliness of the data center, but also realizes cost efficiency and sustainable development. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1A system block diagram of a data center resource dormancy scheduling method based on reinforcement learning is provided.

[0074] Figure 2 A flowchart of a data center resource dormancy and scheduling method based on reinforcement learning is provided.

[0075] Figure 3 A Markov decision process model diagram of reinforcement learning is provided.

[0076] Figure 4 A Deep MAHHQN network structure diagram of a reinforcement learning algorithm for a multi-agent discrete-continuous hybrid action space is provided.

[0077] Figure 5 A training flowchart based on the Deep MAHHQN method is provided. DETAILED DESCRIPTION

[0078] In order to make the purpose and technical scheme of the present application clearer and more convenient to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments. The specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0079] The present application proposes a data center resource dormancy scheduling method based on multi-agent reinforcement learning, to achieve the function of reasonably and efficiently allocating resources and improving the energy efficiency of data centers.

[0080] Reinforcement learning is a method of learning through the interaction of agents and the environment to maximize cumulative rewards. In the resource scheduling problem, reinforcement learning can adapt to dynamic changes in workloads, and adjust the allocation of resources through learning, thereby optimizing system performance. Multi-agent reinforcement learning (MARL) is an extension of reinforcement learning in a multi-agent environment. The present application uses the MAHHQN (Multi-Agent Hierarchical Hybrid QNetwork) algorithm in MARL. The MAHHQN algorithm combines the advantages of deep reinforcement learning and hierarchical reinforcement learning, and better handles the scheduling problem of data center resources through a hierarchical structure.

[0081] The present application realizes the collaborative decision-making among multiple agents in the data center by introducing the MAHHQN algorithm. The agent can adjust in real time according to the current environment and the state of other agents, so as to better adapt to the dynamic changes in workloads. In addition, this scheme has real-time adaptability, and can dynamically adjust the allocation of resources according to the actual situation of the data center, maximize the energy utilization efficiency, and reduce energy consumption.

[0082] Example 1

[0083] Referring to Figure 1 The implementation method of the present application includes a resource dormancy scheduling strategy iteration method based on a data center simulation environment and a deployment scheme in an actual data center environment after obtaining a preliminary (optimized) resource dormancy and scheduling strategy.

[0084] The simulation part of the entire data center room includes an information perception system, a 6sigma simulator in a central processor and a data center server perception control module based on multi-agent reinforcement learning, which learns a preliminary server resource dormancy and scheduling strategy through simulation learning.

[0085] The actual deployment part includes a preliminary resource dormancy and scheduling strategy, a strategy performance determinator, a rule boundary determinator, a workload allocation module, a server power control module and an environment state perception module, etc., which realizes the actual deployment of the simulation resource dormancy and scheduling strategy, dynamically adjusts the allocation of resources through the current data center state, maximizes the energy utilization efficiency and reduces energy consumption.

[0086] Among them, the information perception system includes an environment information collection and information data processing module of the data center room, which is used to collect environmental basic information. The environmental basic information includes the basic information of the data center, the time-of-use electricity price information and the data feedback by the dynamic environment monitoring system. The data center basic information is obtained from the company to which the data center belongs, and the time-of-use electricity price information is obtained from the official website of the local power department. The data center basic information includes the geometric shape and size of the data center room, the type, power consumption and quantity of IT equipment, the type and quantity of air conditioners and other data. The dynamic environment monitoring system is a set of software and hardware for monitoring the environment and power environment of the room, monitoring the power of the server and the temperature and humidity data of the environment.

[0087] The perception control module collects the environment state transmitted by the environment state perception module in the 6sigma simulator or in the actual data center, and outputs a real-time control strategy and sends a control action instruction to the power control module.

[0088] In the simulation stage, the information perception system inputs the collected environmental basic information after data cleaning (handling of outliers and missing values) into the data center 6 sigma simulator as input parameters and establishes a data center simulation model. The data center simulation model updates the environmental state by executing the discrete actions output by the Q network algorithm and the continuous actions output by the policy network through the control server, wherein the discrete actions are the on-off of each server, and the continuous actions are the allocation amount of the workload in each server; and the central processor internally generates real-time server sleep and scheduling strategies based on the DQN (Deep Q Learning) and Actor-Critic optimization algorithms, respectively, and updates the network using the state information and reward information output by the data center 6 sigma simulation model as input.

[0089] In the actual deployment stage, the data center central processor outputs the generated server sleep and scheduling strategy to the strategy performance determinator, and after determining that the optimization effect of the current control strategy meets the preset threshold, deploys the optimized strategy to the central processor in each server, and sends the resource sleep and scheduling instructions corresponding to the strategy to the power control module of each server through the workload allocation module for execution. If the optimization effect of the current strategy does not meet the preset threshold, the control strategy needs to be continuously optimized through simulation reinforcement learning. Each server changes the power by executing the real-time strategy, which affects the environmental changes of the entire data center room. The dynamic environment monitoring system in the environmental state perception module dynamically perceives such changes and collects device temperature and power data as feedback information to update and correct the server resource sleep and scheduling strategy online.

[0090] The rule boundary determinator determines whether the current server sleep and scheduling control action meets the rule experience constraint, and corrects the action that does not meet the rule experience constraint based on the rule experience.

[0091] Embodiment 2

[0092] Figure 2 For the flowchart of the method of the present application, referring to Figure 2 The embodiment of the present application provides a data center resource sleep and scheduling method based on a multi-agent reinforcement learning algorithm MAHHQN. The method will first be trained in a simulation environment, initially establish a reasonable strategy, then run in an actual data center environment, and continuously optimize the strategy until optimal, thereby reasonably and efficiently allocating resources and improving the energy efficiency of the data center. Specifically, the method includes the following steps:

[0093] S1, collect relevant parameters required by the simulation model, establish a 6 Sigma simulation model, and determine the agents and environment of the research object.

[0094] S101, collect required data according to actual data center machine room conditions.

[0095] For a specific data center, the geometric shape and size of the data center machine room, IT equipment, air conditioning system and heat load and other data are counted. Specifically, the geometric shape and size of the machine room include the length, width, height of the data center, the material of the wall, floor and ceiling, etc.; the IT equipment information includes the number, type, power, arrangement, heat dissipation mode of the IT equipment, etc., where the IT equipment mainly refers to the server; the air conditioning system includes the number, type, power, arrangement, air speed, temperature, etc. of the air conditioning system; the heat load; including the heat load distribution of the data center, the heat load of the IT equipment, the heat load of the air conditioning system, etc.

[0096] S102, use 6sigma simulator to establish a data center machine room simulation model according to the data collected in S101, and determine the agent and environment in the reinforcement learning algorithm.

[0097] 6SigmaRoom is a branch of 6SigmaDC, which is a special software tool designed for machine room design or improvement and upgrading of existing machine rooms, which can completely control the construction of the machine room and the configuration of the equipment. In order to establish this simulation model, it is necessary to define the physical properties and equipment types and locations of the virtual machine room in the software according to the various actual data collected in S1, combined with the data center specific modules in 6sigmaDC, such as PDU (distributed power distribution unit), UDF (uninterruptible power supply system), precision air conditioning unit, cabinet, etc.

[0098] In reinforcement learning, the agent is an entity or entity set in the system that can perceive the environment, make decisions and perform actions. In this embodiment, the object referred to by the agent is a perception control module that implements the power management and startup and shutdown of each server, which collects the environment state input by the environment state perception module in the 6sigma simulator or the environment state in the actual data center, and outputs a real-time control strategy to send control action instructions to the power control module. In the following, the agent is represented by the server, denoted as E1,...,E i ,...,E N ; where N is the number of servers in the data center. E1,...,E i ,...,E NThe environment is an external system or scenario in which the agent exists. In this embodiment, the environment refers to the operating environment of an Internet data center (IDC). In the simulation link in the present application, the specific environment is simulated by a 6 sigma simulator. In the actual deployment link, the environment refers to the actual real-time environment of the data center. The agent learns how to select appropriate actions in different states through interaction with the environment to obtain the maximum reward.

[0099] S2, based on the actual data center machine room conditions collected in S1, a Markov decision process (MDP) model is established, and the related parameters are determined by experience.

[0100] The decision-making process of reinforcement learning is usually described by constructing a Markov decision process model. The basic principle diagram of the Markov decision process is shown in Figure 3 At each time point, the agent observes the state s of the environment and selects an action a according to the policy π. Then the environment gives the next state s' according to the state transition function P and gives an immediate reward r according to the reward function R. The goal of the agent is to find an optimal policy π* such that the cumulative reward, i.e., the maximum reward, is maximized by following the actions made by the policy from any initial state s0.

[0101] In S2, each agent corresponds to a server perception and control module in the data center machine room simulation model established in S1, which receives the environment state in the simulation model and outputs control instructions to control the server switch and power in the simulation model through a real-time control strategy; and the environment is composed of a data center machine room simulation model, a power grid electricity trading market, and a cloud computing business workload. In actual deployment, each agent corresponds to a server perception and control module in the actual data center mentioned in S101, which receives the actual machine room environment state and outputs control instructions to control the server switch and power in the actual data center machine room through a real-time control strategy; and the environment is composed of the actual situation of the data center machine room, the power grid electricity trading market, and the cloud computing business workload.

[0102] The Markov decision process model mainly includes a Markov decision process state space model, a Markov decision process action space model, and a Markov decision process reward function model. Specifically, S2 includes the following steps:

[0103] S201, a Markov decision process state space model is established. In the data center, since each server is equipped with a separate power control module, that is, each server can independently control its own action, there are multiple agents. The state space observed by each agent is different, and the overall state space is composed of the state space observed by each agent, expressed as follows:

[0104] S t t i t N t} (1)

[0105] In the formula, S t represents the state space of the overall environment at time t, o i t represents the state space observed by agent i at time t, the time interval is set to Δt according to needs and actual conditions, and the training period is T, so the value set of t is Further, the state of a single agent can be expressed as follows:

[0106] i t i t t t} (2)

[0107] In the formula, T i t represents the operating temperature of server i at time t, in ℃, which can be collected by the data center dynamic system; c t represents the real-time electricity price of the electricity trading market at time t, in yuan / kWh; and W t represents the total business load of the data center at time t.

[0108] S202, a Markov decision process action space model is established. Similar to the state space defined in S201, the overall action space is also composed of the actions of each agent, defined as follows:

[0109] t t i t N t} (3)

[0110] In the formula, A t represents the action space of all agents of the overall environment at time t, A​​​​​​​​​​​i t denotes the action performed by agent i at time t, and the action space of a single agent can be defined as follows:

[0111]

[0112] [K] = {0, 1} (5)

[0113]

[0114] wherein, denotes the discrete action performed by agent i at time t; [K] is a set of values including 0 and 1, i.e., two actual states of shutdown and startup, when denotes that server i is started at time t, and when denotes that server i is in sleep at time t; denotes the continuous action performed by agent i at time t, representing the amount of tasks allocated to server i per unit time at time t; is the value range of , which is controlled by the discrete parameter , when i , wherein m i represents the maximum amount of tasks that can be processed by server i per unit time, and when can only take the value of 0.

[0115] S203, a Markov decision process reward function model is established. Since the servers belong to the same data center, the agents have no competitive relationship with each other, and are fully cooperative multi-agent reinforcement learning, so the rewards of all agents are the same. In general, the reward of an agent is composed of four parts, which are the electricity cost brought by the server working consumption of electric energy, the punishment of the server working temperature exceeding the thermal comfort limit range, the punishment of the server processing work load tasks in violation of the quality of service guarantee (QoS, Quality of Service), and the punishment of the entire data center failing to process the total work load. The specific performance is as follows:

[0116]

[0117] wherein, r t is the reward value of each agent at time t, denotes the reward obtained by agent i at time t by performing action t , c t is the real-time electricity price at time t, with a unit of yuan / kWh, P i t ​​is the power consumption of server i at time t, in kW, λ1is the temperature penalty coefficient, λ2is the quality of service penalty coefficient, and λ3is the service load handling penalty coefficient. In addition, v T,i t is the absolute value of the server i's deviation from its thermal comfort range at time t, v D,i t is the value of the server i's average task response time at time t exceeding the upper limit of the quality of service agreement, v W t is the amount of unprocessed tasks at time t. In an ideal case, the values of the three are all zero.

[0118] where P i t is represented by the following formula:

[0119]

[0120] where p peak,i represents the peak power of server i, p peak,i represents the power of server i in an idle state, represents the amount of tasks allocated to server i per unit time at time t, m i represents the maximum amount of tasks that server i can handle per unit time, and the ratio between the two represents the utilization rate of server i. According to this formula, the amount of server workload allocation and the power of the server are in a linear relationship, and the server can adjust its power through workload allocation.

[0121] v T,i t is represented by the following formula:

[0122]

[0123] where T i t represents the operating temperature of server i at time t, and and T i represent the upper and lower limits of the safe and stable operation temperature of server i, respectively.

[0124] v D,i t is represented by the following formula:

[0125]

[0126] where D i represents the maximum tolerable task response time.

[0127] v W t is represented by the following formula:

[0128]

[0129] In the formula, W represents the total task (workload) arrival amount of the data center, and the formula represents the amount of tasks that have not been processed.

[0130] In the foregoing, the reward R i t Refers to the reward obtained by the agent at the current time by performing an action, and the value obtained by weighting and summing all rewards from the current time to the final time in a training is called the harvest, which is expressed as the following formula:

[0131]

[0132] In the formula, Represents the harvest of the agent i at time t; γ is a discount factor, representing the degree of influence of future rewards on current harvest, and the value is determined by experience.

[0133] S3, based on the MDP model constructed in S2, develop Deep MAHHQN algorithm, train Deep MAHHQN network recorded in the agent in combination with the simulation environment, update each parameter in Deep MAHHQN network, and establish a preliminary resource dormancy scheduling strategy.

[0134] Figure 4 The network structure of Deep MAHHQN, Deep MAHHQN is composed of a high-level network for learning coordinated joint discrete action and a low-level network for learning continuous parameter coordination strategy, and the high-level network and the low-level network are trained respectively, and they all follow the paradigm of centralized training but decentralized execution, centralized training refers to the process of centralized model training in a single central server, all training data are concentrated in one place, and the parameters in the network are updated and optimized by the central server. And decentralized execution refers to the process of model deployment and execution in multiple devices (servers) in the inference stage, the model has been optimized in the centralized training stage, and now it is deployed to each server in the data center to execute the agent strategy. When selecting a hybrid action, the Deep MAHHQN agent first selects a discrete action through the high-level network, and then decides the corresponding continuous action on the given discrete action and individual observation through the low-level network.

[0135] More specifically, the high layer network includes N high layer local value networks, each of which is deployed in a server agent, and a hybrid network, which is deployed in the center server to coordinate the discrete actions of the servers. The low layer network includes N policy networks and N low layer value networks, each of which is deployed in a server agent. In order to train, there is a target network with the same structure as the value network but different parameters in each of the high layer local value networks and the low layer value networks.

[0136] wherein o i t is the state observed by the agent server i at time t, i.e. o i t = {T i t , c t , W t}; Q i (ω i h ) is the high layer local value function of the agent server i, which is the output of the high layer local value network, and its corresponding target network is By inputting the current observation o i t = {T i t , c t , W t}, the discrete action k i t at the current time is outputted. i (θ i ) is the low layer policy network of the agent server i, which outputs the continuous action i t at the current time by inputting the current observation o i t = {T t t , c i t and the selected discrete action k t t . Through the two networks, the agent i can select the discrete and continuous actions according to the current observation state.

[0137] In addition, is the low layer value network of the agent server i, the input of which is the hybrid action of all agents, i.e. the discrete action (switching on / off) of all servers and the continuous action (the amount of workload allocation) χ t , and the output is the low layer value network Q value of a single agent i. Through the Q value, the parameters of the low layer policy network can be updated. And is a high-level global value function, and the target network corresponding to it is It is the output of the hybrid network in the high-level network, and the input of the high-level hybrid network is the current global state S t = {o1 t ,...,o i t ,...,o N t}, all high-level local value functions of the agents and the continuous action set A of all agents at the current time t = {A1 t ,...,A i t ,...,A N t , thereby generating the high-level value at time t The values of are backpropagated to update the network parameters.

[0138] Figure 5 is the training process of the Deep MAHHQN algorithm, referring to the figure, S3 is divided into the following steps in total:

[0139] S301, initialize all high-level local value functions Q i (ω i h ) network parameters ω i h and the corresponding target network parameters Initialize the high-level global value function network parameters ω mix and the corresponding target network parameters Initialize the low-level policy function μ i (θ i ) network parameters θ i and low-level value function network parameters ω i l and the corresponding target network parameters Initialize the experience replay pool.

[0140] S302, initialize the global state S 0 .

[0141] S303, update the environment global state according to the actions performed by the agents of each server. Specifically, S303 includes the following steps:

[0142] S3031, each agent i of each server performs an action according to the current observation o i tSelecting discrete action k by ε-greedy policy i t The server performs discrete action decision whether to start up, and the selection of the discrete action is calculated by the following formula:

[0143]

[0144] The above formula indicates that the agent i has an ε probability of randomly selecting an action from the discrete action space [K] at time t, and has a 1-ε probability of selecting the action with the highest value in the current high-level local value network. By selecting a suitable ε value, a good balance between exploration and utilization can be achieved, and the agent i can both utilize the current information and try new actions to discover better strategies.

[0145] S3032, the current observed state o i t and discrete action k i t are packaged into and input into the low-level policy network of each agent i to obtain continuous action The server performs continuous action decision to determine its task workload allocation.

[0146] S3033, the agent calculates the current time reward r t according to the reward function model in S2 t+1 , and the global state is updated to S The five-tuple is stored in the experience replay pool, where t is the set of all agent discrete actions at time t, and χ t is the set of all agent continuous actions at time t.

[0147] S304, determine whether the experience replay pool array is sufficient: if sufficient, execute step S305, if not sufficient, execute step S308.

[0148] S305, take out b experience arrays from the experience replay pool for updating each value network parameter, the value of b is determined by experience.

[0149] S306, update the high-level network using the experience array, the update steps are as follows:

[0150] S3061, combine all continuous actions χ t ={x1 t ,…,x N t} and the current state S tThe parameters, i.e., the weights of each local function, are input into the hypernetwork to generate the hybrid network. The hypernetwork refers to a network designed and determined empirically, independent of experimental settings. The observed states and actions of each agent are input into the agent's high-level local value network to obtain the high-level local value of each agent.

[0151] S3062, The local value functions Q of each high-level unit i (ω i h The input is fed into the value network of the hybrid network, and the output is the high-level global value function, yielding the output value of the high-level value. Its value is calculated by the following formula:

[0152]

[0153] In the formula ω mix,now These are the parameters of the hybrid network before the update.

[0154] S3063. Use the target network of the hybrid network to perform temporal difference (TD) prediction to obtain the TD target. Calculated by the following formula:

[0155]

[0156] Wherein, γ is a discount factor, determined by human experience.

[0157] S3064. The timing difference (TD) error σ of the high-level network is calculated using the following formula. t The error will be used for backpropagation to update the parameters:

[0158]

[0159] S3065. Update the values ​​of the high-level local value network, the hybrid network, and the parameters of the target network using the TD error, as follows:

[0160]

[0161] Where, ω mix,new For the parameters updated by the hybrid value network, ω mix,now The parameters before the hybrid value network update. To mix the updated parameters of the target network, To mix the parameters of the target network before updating, ω h i,new These are the updated parameters of the high-level local value network of agent i. ω represents the updated parameters of the high-level target network of agent i. h i,nowis the parameter of the high-level local value network of the agent i before updating, is the parameter of the high-level target network of the agent i before updating, ζ is the learning rate of the high-level value network (including the mixed network), which is determined by artificial experience; τ is the updating rate of the target network, which is determined by artificial experience.

[0162] S307, use the experience array to train and update the low-level network parameters, and the updating steps are as follows:

[0163] S3071, input the continuous actions performed by all agents together into the low-level value network of each agent The q value prediction using the low-level value network is as follows:

[0164]

[0165] Wherein, q l,t i is the low-level network predicted q value of the agent i at time t, is the low-level value network parameter at the current time.

[0166] S3072, input the action set into the target network of each agent to make TD prediction as follows:

[0167]

[0168] Wherein, y l,t i is the low-level network TD target value of the agent i at time t, is the low-level target network parameter at the current time.

[0169] S3073, calculate the low-level network TD error between the results obtained in S3071 and S3072 As follows:

[0170] σ t i = q l,t i - y l,t i (23)

[0171] S3074, update the value network and target network parameters of the low-level network respectively according to the following formula:

[0172]

[0173] Wherein, is the updated low-level value network parameter of the agent i, is the low-level value network parameter of the agent i before updating, The updated low-level target network parameter of the agent i, The low-level target network parameter of the agent i before updating, a is the learning rate of the low-level value network, and τ is the learning rate of the target network, which is determined by artificial experience.

[0174] S3075, according to the calculation result of S3074, the agent i updates the parameters of the policy network μ i (θ i ) as follows:

[0175]

[0176] Wherein, β is the learning rate of the policy network update, θ i,now is the current parameter of the policy network of the agent i, θ i is the parameter of the policy network of the agent i, is the current observed state o i t and discrete action k i t to form a new state, is the continuous action, θ i,new is the parameter of the updated policy network, θ i,now is the parameter of the policy network before updating.

[0177] S308, according to the time value in the current state, judge whether the training is finished, if yes, execute step S309, if not, jump to step S301.

[0178] S309, according to the current policy output benefit, judge whether the training should be finished, that is, compare whether the harvest of one training is greater than the harvest threshold, which is determined by human in advance, representing the expected effect of energy saving and efficiency improvement: if yes, execute S4, if not, jump to step S302.

[0179] S4, deploy the preliminary resource dormancy scheduling strategy obtained by S3 training to the actual data center, guide the scheduling decision of the data center server agent on the data center server on-off and power management, and interact with the actual environment to continuously update and optimize the scheduling strategy until the training effect reaches the expected simulation result. Specifically, S4 is divided into the following steps:

[0180] S401, deploy the preliminary resource dormancy scheduling strategy learned by S3 simulation environment training to the actual data center, and use the preliminary resource dormancy scheduling strategy to guide each server to execute action according to the observed actual environment state.

[0181] S402, for the preliminary resource dormancy scheduling strategy, referring to the training steps from S301 to S309 in S3, each server agent is continuously trained until the output benefit reaches the expected effect, and the training is ended.

[0182] S403, each server agent executes discrete actions and continuous actions according to the trained policy network and value network to control the server power control module and workload allocation module to perform dormancy and scheduling operations on the server. The specific selection of actions includes the following two steps:

[0183] S4031, selecting a discrete action according to the observed state. At time t, each server agent i observes the current state o i t Through the high-level network Q i (ω i ) to select the discrete action k i t that maximizes the value, which determines whether to turn on or turn off the server i, and its value is determined by the following formula:

[0184] k i t = argmax Q i (o i t ,k i t ,x i t ;ω i ) (27)

[0185] S4032, selecting a continuous action according to the observed state and the discrete action. At time t, each server agent i observes the new observation obtained by the current observation and the discrete action jointly, through the low-level policy function μ i to select the continuous action to allocate the total business of the data center to each server, and its value is determined by the following formula:

[0186]

[0187] Embodiment 3

[0188] A data center resource dormancy scheduling system based on multi-agent reinforcement learning, characterized in that it comprises

[0189] An information perception system for collecting relevant parameters required by the simulation model;

[0190] A simulator for establishing a data center room simulation model according to the relevant parameters required by the simulation model, and determining the agents and the environment of the research object;

[0191] The perception control module is used for establishing a Markov decision process model based on actual data center machine room conditions, and determining parameters of the model through experience; the Markov decision process model constructed based on S2 is used for training the Deep MAHHQN network in combination with a simulation environment, updating parameters in the Deep MAHHQN network, and forming a preliminary resource dormancy scheduling strategy;

[0192] The actual deployment module is used for deploying the preliminary resource dormancy scheduling strategy to guide the scheduling decision of the data center server switch and power management of each server agent of the data center, and continuously updating and optimizing the scheduling strategy in interaction with the actual environment until the training effect reaches the expected simulation result.

[0193] The above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A method for data center resource dormancy and scheduling based on reinforcement learning, characterized in that, The method comprises the following steps: S1, collecting relevant parameters required by the simulation model, establishing a data center room simulation model, and determining the agents and environment of the research object; S2, establishing a Markov decision process model based on the actual data center room situation, and determining the parameters thereof through experience; the Markov decision process model comprises a Markov decision process state space model, a Markov decision process action space model, and a Markov decision process reward function model; The expression of the Markov decision process reward function model is: wherein, is the reward value of each agent at time t, refers to the reward obtained by agent i at time t by performing action , is the real-time electricity price at time t, is the power consumption of server i at time t, is the temperature penalty coefficient, is the quality of service penalty coefficient, is the service load processing penalty coefficient; in addition, is the absolute value of server i exceeding its thermal comfort range at time t, is the value of the average task response time of server i at time t exceeding the upper limit of the quality of service agreement, is the amount of unprocessed tasks at time t; S3, training the Deep MAHHQN network based on the Markov decision process model constructed in S2, updating the parameters in the Deep MAHHQN network, and forming a preliminary resource dormancy scheduling strategy; S3 comprises updating the global state of the environment according to the actions performed by the agents of each server; the updating of the global state of the environment according to the actions performed by the agents of each server comprises: S3031, the agent i of each server calculates the state value function value of the current state according to the current observation By Policy selection discrete action , the server performs the discrete action decision whether to start, the selected discrete action is calculated by the following formula: wherein, represents a very small positive number, is a high-level local value function, is a high-level local value function parameters of the network; S3032、the current observed state and discrete actions packed into and input into the low-level policy network of each agent i to get continuous actions The server executes the continuous action to determine its assigned task workload; S3033, obtain the current time reward according to the reward function model in S2 The global state is updated as The five-tuple is stored in the experience replay pool, wherein, is the global state at the current time, is the set of all discrete actions of the agents at time t, is the set of all continuous actions of the agents at time t; S4, deploying the preliminary resource dormancy scheduling strategy obtained in S3 to an actual data center, guiding the scheduling and decision-making of the server on-off and power management of the data center server by the agents of the data center, and interacting with the actual environment to continuously update and optimize the scheduling strategy until the training effect reaches the expected simulation result. 2.The method of claim 1, wherein, The step S1 comprises the following steps: S101, collecting required data according to the actual data center room situation; S102, establishing a data center room simulation model using a 6sigma simulator according to the data collected in S101, and determining the agents and environment in the reinforcement learning algorithm. 3.The method of claim 1, wherein, The step S2 comprises the following steps: S201, establishing a Markov decision process state space model, wherein the overall state space is composed of the state spaces observed by each agent; S202, establishing a Markov decision process action space model, wherein the overall action space is composed of the actions of each agent; S203, establishing a Markov decision process reward function model. 4.The method of claim 1, wherein, The step S3 comprises the following steps: S301, initializing the parameters of all high-level local value function networks in the Deep MAHHQN network and the corresponding target network parameters thereof; initializing the parameters of the high-level global value function network and the corresponding target network parameters thereof; initializing the low-level policy function network parameters, the low-level value function network parameters, and the corresponding target network parameters thereof; and initializing the experience replay pool; S302, initializing the global state; S303, updating the global state of the environment according to the actions performed by the agents of each server; S304, determining whether the experience replay pool array is sufficient: if yes, performing step S305, and if no, performing step S308; S305, taking out a plurality of experience arrays from the experience replay pool; S306, training and updating the high-level network using the experience arrays; S307, training and updating the low-level network parameters using the experience arrays; S308, determining whether the training is completed according to the time value in the current state: if yes, performing step S309, and if no, jumping to step S301; S309, outputting the benefit according to the current strategy to determine whether the training should be ended: if yes, performing S4, otherwise, jumping to step S302. 5.The method of claim 4, wherein, The step S306 includes the following steps: S3061, inputting all the continuous actions and the current state into the super network to generate the parameters of the mixed network, i.e., the weights of each local function, inputting the state and action observed by each agent into the high-level local value network of the agent to obtain the high-level local value of each agent; S3062, inputting each high-level local value function into the value network of the mixed network to output a high-level global value function, and obtaining an output value high-level value; S3063, using the target network of the mixed network to make a time difference prediction to obtain a TD target; S3064、calculating the timing difference error of the upper layer network ; S3065, updating the parameters of the high-level local value network, the value network of the mixed network and the target network according to the TD error, as follows: wherein, are the updated parameters of the hybrid value network, are the updated parameters of the hybrid value network, are the updated parameters of the hybrid target network, are the updated parameters of the hybrid target network, is the high-level global value function are the parameters of the network, is the current global state, is the set of discrete actions of all agents at time t, is the set of continuous actions of all agents at time t; is the updated parameter of the high-level local value network of agent i, is the high-level local value function are the parameters of the network, is the updated parameter of the high-level target network of agent i, is the updated parameter of the high-level local value network of agent i, is the updated parameter of the high-level target network of agent i, is the learning rate of the high-level value network; is the update rate of the target network. 6.The method of claim 4, wherein, The S307 includes the following steps: S3071, inputting all the continuous actions performed by the agents into the low-level value network of each agent to make a q value prediction as follows: wherein, is a low-level value network, is a low-level network prediction q-value of agent i at time t, is a current global state, is a set of discrete actions of all agents at time t, is a set of continuous actions of all agents at time t; is a low-level value network parameter at the current time; S3072, inputting the action set into the target network of each agent to make a TD prediction as follows: wherein, is the low-level network TD target value for agent i at time t, is the reward value for each agent at time t, is the discount factor, is the updated global state, is the low-level target network parameter at the current time. S3073, compute and low layer network TD error between S3074, updating the value network and the target network parameters of the low-level network using the following formula: wherein, updated low-level value network parameters for agent i, low-level value network parameters for agent i before update, low-level value function network parameters , updated low-level target network parameters for agent i, low-level target network parameters for agent i before update, learning rate for low-level value network, high-level network's temporal difference error, learning rate for target network; S3075, based on the updated low-layer network value network and target network parameters, the agent i updates its policy network Parameter update is performed, and the expression is as follows: wherein, the learning rate for the policy network update, are the current parameters of the policy network for agent i, are the parameters of the policy network for agent i, is the current observed state and a discrete action resulting in a new state, is a continuous action, are the parameters of the updated policy network.

7. The method of claim 1, wherein, The step S4 includes the following steps: S401, deploying the preliminary resource dormancy scheduling strategy learned by the simulation environment training to the actual data center, and guiding each server to perform actions according to the observed actual environment state using the preliminary resource dormancy scheduling strategy; S402, continuously training each server agent according to the training steps of S3 for the preliminary resource dormancy scheduling strategy until the output benefit reaches the expected effect; S403, performing discrete actions and continuous actions according to the trained strategy network and value network to control the power control module and workload distribution module of each server to perform dormancy and scheduling operations on the server.

8. A multi-agent reinforcement learning based data center resource dormancy scheduling system, characterized in that, The information perception system is configured to collect relevant parameters required by the simulation model. The simulator is configured to establish a data center room simulation model according to the relevant parameters required by the simulation model, and determine the agents and the environment of the research object. The perception control module is configured to establish a Markov decision process model based on the actual data center room situation, and determine the parameters of the model through experience; train the Deep MAHHQN network based on the established Markov decision process model and the simulation environment, update the parameters in the Deep MAHHQN network, and form a preliminary resource dormancy scheduling strategy. The Markov decision process model includes a Markov decision process state space model, a Markov decision process action space model and a Markov decision process reward function model. The expression of the Markov decision process reward function model is as follows: ​ wherein, is the reward value of each agent at time t, refers to the reward obtained by agent i at time t by performing action , is the real-time electricity price at time t, is the power consumption of server i at time t, is the temperature penalty coefficient, is the quality of service penalty coefficient, is the service load processing penalty coefficient; in addition, is the absolute value of the server i exceeding its thermal comfort range at time t, is the value of the average task response time of server i at time t exceeding the upper limit of the quality of service agreement, is the amount of unprocessed tasks at time t; The preliminary resource dormancy scheduling strategy is formed by training the Deep MAHHQN network based on the constructed Markov decision process model and combining the simulation environment, including updating the global state of the environment according to the action performed by the agent of each server; the updating of the global state of the environment according to the action performed by the agent of each server includes: S3031, the agent i of each server calculates the state value function value of the current state according to the current observation By Policy selection discrete action , the server executes the discrete action decision whether to start, the selected discrete action is calculated by the following formula: wherein, represents a very small positive number, is a high-level local value function, is a high-level local value function parameters of the network; S3032, the current observed state and discrete actions packed into and input into the low-level policy network of each agent i, resulting in continuous actions , the server performs continuous action determines its allocation of task workload; S3033, obtain the current time reward according to the reward function model in S2 The global state is updated as The quintuple is stored in the experience replay pool, wherein, is the global state at the current time, is the set of all discrete actions of the intelligent agents at time t, is the set of all continuous actions of the intelligent agents at time t; The actual deployment module is configured to deploy the agent of each server of the data center to make scheduling decisions on the power-on and power-off of the servers of the data center and power management based on the preliminary resource dormancy scheduling strategy, and to interact with the actual environment to continuously update and optimize the scheduling strategy until the training effect reaches the expected simulation result.

Images