Data center economic optimization scheduling method based on multi-agent deep reinforcement learning

Abstract

The invention discloses a data center economic optimization scheduling method based on multi-agent deep reinforcement learning. According to the method, a model covering three levels of a CPU, a cabinet and a data center is established, and an electricity-heat-calculation coupling relation among an IT system, a cooling system and an energy supply system is described; an economic optimization problem is converted into a Markov decision process, an operation target is mapped into a cooperative scheduling task among multiple agents, and an observation space, an action space and a reward function of three kinds of agents including load scheduling, temperature adjustment and battery energy management are constructed. And a multi-agent optimization operation framework is constructed, and a multi-agent cooperation mechanism based on an Actor-Critic framework is constructed, so that real-time optimization scheduling of the data center is realized. According to the method, typical day data in July in summer is selected for experimental verification, and the effectiveness and superiority of the method are fully verified through comparative analysis with MAPPO, PPO and MPC algorithms.

Description

Technical Field

[0001] This invention relates to an economic optimization scheduling method for data centers based on multi-agent deep reinforcement learning, belonging to the field of data center operation management and optimization control technology. Background Technology

[0002] With the rapid growth of cloud computing and artificial intelligence, data centers, as critical digital infrastructure, are facing increasingly prominent issues of high operating costs and energy consumption, placing dual pressure on the economy and the environment. Faced with this challenge, the optimization goals of data centers are no longer limited to simply pursuing energy efficiency, but have gradually expanded to the realm of economic optimization. This economic optimization is a complex, multi-objective, constrained operations problem. On the one hand, data centers can achieve direct power cost optimization through workload shifting strategies, but face Quality of Service (QoS) constraints such as task completion time and timeout rate during task scheduling. On the other hand, data center energy equipment can achieve indirect power cost optimization through strategies such as adjusting room air conditioning temperatures and battery energy storage, but faces resource constraints such as adjustable temperature and adjustable capacity. Therefore, data center scheduling is no longer limited to minimizing energy consumption, but rather achieving a balance between operating costs and QoS performance indicators under dynamic electricity prices, load, and resource constraints.

[0003] Traditional data center optimization scheduling methods mainly include linear programming, model predictive control (MMCC), and heuristic algorithms. Linear programming methods typically simplify nonlinear elements in the system to improve solution efficiency. MMCC employs rolling optimization and feedback correction mechanisms to address uncertainties such as load, renewable energy, and temperature fluctuations. Furthermore, heuristic algorithms are suitable for handling multi-objective non-convex problems that are difficult to solve effectively with linear programming and MMCC. In energy consumption modeling, data centers typically use linear models based on server utilization, simplifying the solution process by linearly fitting idle power and peak power consumption at full load. To more accurately reflect the impact of component load changes on total power, some studies have used component power superposition methods for modeling. However, real physical relationships often exhibit strong nonlinearity, making linear simplification difficult to accurately describe their inherent characteristics. Moreover, when the problem scale is large and the number of variables and constraints increases, the solution time of traditional methods increases significantly, making it difficult to meet the response speed requirements of real-time scheduling. Simultaneously, server-level energy consumption modeling methods struggle to accurately characterize the complex internal structure and dynamic characteristics of data centers.

[0004] In recent years, reinforcement learning has been widely applied in power system optimization due to its rapid decision-making and powerful nonlinear processing capabilities. Reinforcement learning methods do not require pre-defined probability distributions or complex scenario models; they are trained directly using historical data from real power systems, learning uncertainties through continuous interaction with the environment. Furthermore, the trained model can be directly used in the inference process, improving the real-time decision-making speed of the system. Reinforcement learning methods can effectively overcome the problems of model dependence and low solution efficiency in traditional optimization methods. However, when applied to large-scale physical systems, single-agent reinforcement learning faces difficulties in parameter tuning due to high-dimensional state and action spaces. Additionally, homogeneous multi-agent systems struggle to effectively adjust to differentiated task objectives when facing heterogeneous tasks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a multi-agent deep reinforcement learning-based economic optimization scheduling method for data centers. This invention transforms the optimization problem into a Markov decision process, maps the economic operation objective into a collaborative scheduling task among multiple agents, designs a collaborative mechanism among multiple agents based on the Actor-Critic framework, and ultimately achieves economic operation of the data center system.

[0006] The objective of this invention is achieved through the following technical solution: 1) Establish a unified model covering three levels: CPU, server rack, and data center, to characterize the electrical-thermal-computational coupling relationship between IT system, cooling system, and power supply system, and construct an economic optimization scheduling problem based on the model; 2) The above-mentioned optimization scheduling problem is transformed into a Markov decision process, and based on the multi-agent cooperative scheduling mechanism, the observation space, action space and reward function of the load scheduling agent, the data center temperature regulation agent and the battery energy management agent are designed respectively. 3) Construct a multi-agent collaborative operation mechanism based on the Actor-Critic framework, including building a centralized Critic network and independent Actor networks for each agent, adopting a sequential update strategy and introducing a correction factor to handle the non-stationarity problem in the multi-agent policy update. 4) Based on the unified model generated in step 1), the interaction environment is used to iteratively train the centralized Critic network and each Actor network constructed in step 3); the observation space data defined in step 2) is input into each Actor network to generate scheduling actions, and the parameters of each network are adjusted according to the feedback of the centralized Critic network using the sequential update strategy until the network converges, thereby obtaining the optimal cooperative scheduling strategy for each agent.

[0007] The beneficial effects of this invention are: This invention constructs an energy consumption model covering the CPU-rack-data center layers, capable of simulating energy consumption variations under different configurations. The model introduces a temperature-based energy consumption impact mechanism at the CPU level, providing a modeling foundation for subsequent design of data center temperature regulation agents. This invention establishes a multi-agent reinforcement learning framework for data centers to achieve rapid and economical decision-making. By constructing three agents—load scheduling, temperature regulation, and battery energy management—the parameter scale and training difficulty of a single network are effectively reduced. This invention designs a reward function that comprehensively considers multiple factors to guide the system towards economical operation. This function integrates the requirements for the healthy and stable operation of each subsystem, as well as diverse objectives such as load shifting, temperature regulation, and battery arbitrage, thereby achieving an effective balance between operating costs and QoS performance indicators while ensuring service quality. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the system model of the present invention; Figure 2 This is a diagram illustrating the interaction between the intelligent agent and the environment in this invention. Figure 3 These are the convergence curves of the policy loss and training convergence curves for each agent. Figure 4 This is the convergence curve of the Critic loss; Figure 5 This is the average step reward convergence curve; Figure 6 This is a typical daily scheduling load transfer diagram; Figure 7 This is a typical daily scheduling task queue dynamic diagram; Figure 8 This is a typical daily scheduling power decomposition diagram; Figure 9 This is a typical daily temperature control chart; Figure 10 This is a typical daily energy management scheduling diagram; Figure 11 This is a typical daily scheduling cost chart. Detailed Implementation

[0010] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Figure 1 This is a schematic diagram of a data center system model; Figure 2Interaction diagram between the agent and the environment. (Refer to...) Figure 1 , Figure 2 This invention proposes a multi-agent deep reinforcement learning-based economic optimization scheduling method for data centers, which specifically includes the following steps: 1) A unified model covering three levels—CPU, server rack, and data center—was established to characterize the electrical-thermal-computational coupling relationship between the IT system, cooling system, and power supply system, and an economic optimization scheduling problem was constructed based on the model.

[0011] In this embodiment, step 1) is specifically implemented in the following subdivided steps: 1.1) Workload Model 1.1.1) Workload partitioning Data center workloads are divided into two categories: schedulable and unschedulable. Unschedulable tasks... For latency-sensitive tasks, computation must be performed immediately in the current step, while schedulable tasks... It can be calculated in the current step or delayed to a future time.

[0012]

[0013]

[0014] In the formula Delayable loads account for a portion of the total load. The proportion.

[0015] 1.1.2) Task Delay Queues Task Delay Queue It is a first-in, first-out (FIFO) queue used to store delayed tasks. Delayable tasks can be added to the queue for delay, and the delay time for a task is specified. The calculation is as follows:

[0016] in and These represent the current and delayed task delays in days, respectively. and These are the delay times for the current and delayed tasks, respectively. The queue and delayed tasks must each meet the maximum queue capacity. and maximum delay time The constraints are as follows:

[0017] 1.2) IT System Model A data center's IT (Information Technology) system consists of multiple server racks, which are made up of multiple servers. Each server consists of a central processing unit (CPU) and IT fans.

[0018] 1.2.1) Server power consumption IT fans dissipate the heat generated during server operation to maintain normal CPU performance. CPU instantaneous operating power... It is determined by the current server ingress temperature. and IT load rate It was decided jointly, as follows:

[0019]

[0020] In the formula, This is the temperature coefficient of the CPU inlet temperature, representing the degree to which temperature changes affect the CPU's base power consumption. This is the load power adjustment factor, representing the sensitivity of the current CPU power to changes in IT load; An empirical adjustment factor is used to fine-tune the instantaneous power consumption of the CPU. This represents the maximum power consumption of the CPU under full load. This represents the CPU's power consumption when idle. Servers have two states: idle and running, therefore, the server's instantaneous power consumption... This is the maximum value between static power consumption and operating power consumption.

[0021] The actual airflow of an IT fan and instantaneous power Depends on its fan speed ratio This coefficient is driven by both CPU temperature and load status, and exhibits the following functional relationship:

[0022]

[0023] In the formula, and These are the fan's response coefficients to temperature and load changes, respectively. For the constant term correction coefficient; Standard reference speed ratio for fans; and These correspond to the rated air volume and rated power consumption under standard operating conditions, respectively.

[0024] 1.2.2) Cabinet Model Total CPU power of the rack Total IT fan power and total air volume It is composed of all the servers in the rack, as shown below:

[0025]

[0026] Because there is no air separation between the cold and hot aisles in each server rack, the cold air exchanges heat with the surrounding environment during air exchange. This causes the supply air temperature in the rack to rise, thus increasing the rack's inlet temperature. have:

[0027] In the formula, The temperature set for CRAC, and These represent the minimum and maximum values ​​of the adjustable temperature range for CRAC. This is the supply air temperature difference, used to correct for deviations in the supply air temperature. The outlet temperature... The total power of each rack is related to the total power of the CPU inside the rack. During operation, some of the energy of the CPU is converted into heat. The IT fan dissipates the heat through the air. The hot air exhausted from the rack becomes a major component of the hot aisle and is sent into the CRAC for cooling as return air.

[0028]

[0029] In the formula, and These are the specific heat capacity and density of air, respectively. Parameters All are empirical fitting coefficients: This is the thermal coupling strength coefficient between the heat source and the heat dissipation medium; and The tables respectively characterize the nonlinear effects of power and air volume on temperature rise; The calibration factor for the denominator term; Used to correct the baseline deviation of the model.

[0030] 1.2.3) Data Center-Level IT System Model A data center IT system consists of multiple server racks, and the total power consumption of a data center IT system is... and CPAC average return air temperature As shown below:

[0031] 1.3) Cooling System 1.3.1) Computer room air conditioning A Computer Room Air Conditioner (CRAC) is an air conditioning unit inside a data center that directly exchanges heat with the hot air from IT equipment. Therefore, the CRAC needs to provide cooling capacity to the data center. Then it is:

[0032] in, This refers to the CRAC airflow per unit IT power. The CPAC's internal fan drives air circulation for cooling; therefore, the fan's instantaneous power consumption is... for:

[0033] In the formula, This is the reference power for the CRAC fan; The airflow rate per unit IT power under rated operating conditions.

[0034] 1.3.2) Chiller Unit Chillers consume a large amount of electricity to produce chilled water, thereby absorbing heat transferred from the CRAC (Crew Refrigerant Regulator). The maximum cooling capacity of the chiller will vary with the outside temperature. Adjustments are made based on changes in capacity. Define a capacity correction factor. To describe the maximum cooling capacity of the unit Temperature change characteristics:

[0035]

[0036] In the formula, and For temperature rise correction parameters; and These are the reference ambient temperature and the rated maximum cooling capacity under standard operating conditions, respectively. , , These are the fitting coefficients. Simultaneously, a power correction coefficient is introduced. This is to reflect the changes in compressor power consumption characteristics under varying operating conditions.

[0037] In the formula, , , This is the power curve coefficient. The energy efficiency degradation of the unit under non-full-load conditions is corrected by a factor. describe:

[0038]

[0039] In the formula, This refers to the partial load rate of the chiller unit. , , This represents the partial load curve coefficient. Based on the above coefficient, the actual power of the chiller unit is... for:

[0040] In the formula, This is the start / stop correction factor; This is the rated energy efficiency ratio of the unit.

[0041] 1.3.3) Cooling towers and water pumps The cooling-side equipment is responsible for dissipating the heat collected by the chiller unit to the outside environment; the cooling tower's fan power consumption... As shown below:

[0042]

[0043]

[0044] In the formula, and This provides the reference power consumption and reference airflow for the CT fan. Standard ambient temperature difference; This represents the maximum operating temperature difference of the CT scanner. This is the required air volume for the CT.

[0045] The system incorporates two types of water pumps: a chilled water pump and a cooling water pump. Its power consumption... As shown below:

[0046] In the formula, For water pressure drop; The water flow rate of the water pump; For water pump efficiency; Represents chilled water pumps. This represents the cooling water pump. In summary, the total power consumption of the HVAC system... and consume electrical energy for:

[0047]

[0048] 1.4) Battery Model The battery has two states: charging and charging. and discharge Status. The amount of charge the battery has during charging. From charging power Charging efficiency The impact.

[0049]

[0050]

[0051] In the formula, It is a response time constant; Rated charging power coefficient; In a charged state, This refers to the battery capacity. This refers to the actual charging power. This refers to the rated charging power. During discharge, this represents the battery's discharge capacity. From output power and discharge efficiency The impact.

[0052]

[0053] In the formula, This is the rated discharge power coefficient; This represents the actual discharge power. This is the rated discharge power. The battery's power rating. and the battery level at the next moment for:

[0054]

[0055] The following constraints must be satisfied:

[0056] 1.5) Optimization Problem Modeling The optimal scheduling of the entire system will use IT load rate, data center temperature regulation, and battery energy storage / discharge as decision variables at different time periods, with the objective of minimizing the system's economic operating cost, to solve the aforementioned nonlinear, nonconvex optimization model. Among these, the system operating cost... The electricity cost of the equipment is shown below:

[0057] in The problem is a mixed-integer nonlinear programming problem, involving time-of-use electricity pricing. Data center systems face time-varying demands on internal and external resources, exhibiting a highly dynamic environment and coupled computing, thermal, and electrical systems. Traditional optimization methods struggle to solve this problem. Therefore, this problem is transformed into a sequential decision-making problem, and a multi-agent reinforcement learning approach is used for optimization.

[0058] 2) By transforming the above-mentioned economic optimization scheduling problem into a Markov decision process, and based on a multi-agent cooperative scheduling mechanism, the observation space, action space and reward function of three types of agents—load scheduling, data center temperature regulation and battery energy management—are designed respectively.

[0059] In a specific embodiment of the present invention, step 3) is implemented as follows: 2.1) Transform the economic optimization problem into a Markov decision process and map the operational objective into a collaborative scheduling task among multiple agents.

[0060] To address the above issues, the collaborative optimization and control problem of data centers is modeled as a decentralized partially observable Markov Decision Process (Dec-POMDP). The entire system consists of three agents. Composition. The Dec-POMDP process can be composed of a global tuple. To describe. It is the global state space of the system, but the agent cannot fully observe it. It is the joint action space of all intelligent agents. It is the state transition probability function, which depends on the current state. and joint actions . It is a global reward function. It is the joint observation space of all intelligent agents, in time Each agent Able to receive its local observations , It's a discount factor. The goal is to find a set of distributed strategies. This enables all agents to make independent decisions based on local observations and allows them to jointly implement a policy. It can work together to achieve the optimal cumulative return for the entire system.

[0061] This system consists of three heterogeneous agents, each making decentralized decisions based on its local observations. strategy Learning from its local observation space To its action space The mapping.

[0062] 2.2) Design the observation space, action space, and reward function of the load scheduling agent; the observation space of the load scheduling agent includes the current total load, schedulable load, unschedulable load, queue state, task waiting time, and predicted load; the action space includes three types of discrete actions: delaying tasks, processing the current load, and processing the current and queue loads; the reward function consists of task discard penalty, queue health reward, and task latency penalty. Specifically: 2.2.1) Observation Space Load scheduling agent observation space It is an 8-dimensional vector space. In any At any moment, intelligent agent Received observation vector Defined as:

[0063] in, and The continuous-time signal is decomposed. and for The maximum and average waiting times for tasks in the time queue. This is a prediction of the total load for the next moment based on historical data, used to assist the agent in predicting load fluctuations.

[0064] 2.2.2) Action Space The load scheduling agent is designed with three discrete actions, one of which can be selected and executed at each time step. Specifically: Action 1: Place all currently schedulable tasks into the delay queue and process the unschedulable load for this time step; Action 2: Process only the current total load without performing any queue operations; Action 3: Process both latency-sensitive loads and backlogged tasks in the delay queue.

[0065] Action 1: Delay the schedulable load The actual number of tasks that can be delayed at the current time step The queue's capacity is limited. If there is no available capacity in the queue, tasks will be dropped.

[0066]

[0067]

[0068] in, To maximize processing capacity; This represents the number of tasks dropped. The queue status and IT load rate at this time are:

[0069] Action 2: Process all load in the current step. The agent selects this action, handling only the total load of the current step. The current queue state and IT load rate are:

[0070] Action 3: Process all load and queued delayed tasks in the current step. The agent will only process tasks in the delay queue after completing the task of the current step. The actual number of tasks processed in the delay queue is... Limited by the current processing capabilities of intelligent agents and the latency load in the queue at the previous time step Then we have:

[0071]

[0072] At this point, the current queue status and IT load rate are:

[0073] 2.2.3) Reward Function Design Reward function for load scheduling agent Penalty for dropping a task Queue health rewards and task delay penalty constitute. Determined by the number of tasks discarded ,and Then based on the queue health status and extreme punishment constitute. To calculate the task ratio, the agent in Specific reward value at any time As shown below:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] 2.3) Design the observation space, action space, and reward function of the data center temperature regulation agent; the observation space of the data center temperature regulation agent includes the current temperature setting, rack inlet / outlet temperature, outdoor temperature, and predicted temperature; the action space includes three types of discrete actions: cooling, maintaining, and heating; the reward function is based on the deviation between the CRAC set temperature and the reference temperature, and is evaluated using a Gaussian function. Specifically: 2.3.1) Observation Space Data Center Temperature Regulation Intelligent Agent observation space It is a 9-dimensional vector space. In any At any moment, intelligent agent Received observation vector Defined as:

[0080] This refers to the current outdoor ambient temperature. The ambient temperature at the next moment, predicted based on short-term weather conditions, is used to help the agent overcome the thermal inertia of the cooling system.

[0081] 2.3.2) Action Space The data center temperature regulation agent is designed with three discrete actions. Action 1 is cooling, then... Action 2 maintains a constant temperature, then Action 3 involves heating up, then... .

[0082]

[0083]

[0084] 2.3.3) Reward Function Design The data center adaptively adjusts its temperature settings based on the ambient temperature to ensure efficient operation of the temperature control system. A recommended temperature is calculated based on the ambient temperature. . This represents the deviation between the CRAC temperature and the reference temperature. An evaluation using a Gaussian function is employed. This refers to the temperature tolerance range. The intelligent agent is in... Specific reward value at any time As shown below:

[0085]

[0086]

[0087] 2.4) Design the observation space, action space, and reward function of the battery energy management temperature regulation agent; the observation space of the battery energy management agent includes battery state of charge, electricity price, charging / discharging power, and total system power; the action space includes three types of discrete actions: charging, discharging, and inaction; the reward function consists of battery arbitrage reward and battery health reward. Specifically: 2.4.1) Observation Space Battery energy management intelligent agent observation space It is a 5-dimensional vector space. In any At any moment, intelligent agent Received observation vector Defined as:

[0088] 2.4.2) Action Space The battery energy management agent is designed with three discrete actions: action 1 is charging; action 2 is discharging; and action 3 is not performing any operation.

[0089] Action 1: Charging

[0090]

[0091]

[0092] in, The charging rate controlled by the intelligent agent. Power at the charging rate controlled by the intelligent agent.

[0093] Action 2: Discharge

[0094]

[0095]

[0096] in, The discharge rate controlled by the intelligent agent. The power output is controlled by the intelligent agent at the specified discharge rate. Furthermore, the battery discharge rate must not exceed the power requirements of the data center.

[0097] Action 3: No operation is performed. The battery status at this time is:

[0098] 2.4.3) Reward Function Design Battery energy management agent reward function Battery arbitrage reward and battery SoC health rewards Composition. Electricity price information. Guide batteries to engage in arbitrage. This represents the average electricity price. Then based on the battery health status and extreme punishment Composition. Intelligent agents in Specific reward value at any time As shown below:

[0099]

[0100]

[0101]

[0102]

[0103] 3) Construct a multi-agent collaborative operation mechanism based on the Actor-Critic framework, including building a centralized Critic network and independent Actor networks for each agent, adopting a sequential update strategy and introducing a correction factor to handle the non-stationarity problem in multi-agent policy updates. In a specific embodiment of the present invention, step 3) is implemented according to the following sub-steps: 3.1) Construct a multi-agent optimized operation framework.

[0104] For the multi-agent environment and agent design constructed above, the Heterogeneous Agent Proximal Policy Optimization (HAPPO) algorithm is used to improve decision-making speed. The interaction process between the agent and the environment, such as... Figure 2 As shown: The data center system consists of three independent intelligent agents. Interact with the environment and observe the local state. Mapped to an action probability distribution, and sampled from it to generate Resource scheduling decisions are made, and then the centralized Critic network evaluates the value of the current state based on the observed complete global state. The system then considers the joint actions of all agents. (Right now The system transitions to the next state and generates an immediate reward. It is trained by continuously repeating this interaction process and optimizes the policy function using the HAPPO algorithm. When the policy function no longer changes significantly, the trained model is considered a deployable model for real-time optimization of the system's scheduling policy.

[0105] 3.2) Constructing the central Critic network for the intelligent agent The central Critic is based on the global state. Starting with the generalized advantage estimation (GAE), we calculate the advantage function. Value function under old parameters Building a critic network goal .

[0106]

[0107]

[0108]

[0109] In the formula, For environmental rewards; This refers to timing difference error; As a reward discount factor; This is the GAE discount factor.

[0110] Using the clipped value function Construction cutting error This restricts updates to the trust region to enhance training stability. The Critic network minimizes the loss function through gradient descent. Used to minimize current value and returns The error between them.

[0111]

[0112]

[0113]

[0114]

[0115] In the formula, These are the trimming parameters.

[0116] 3.3) Construct the Actor network for each agent and introduce correction factors for sequential updates. Each agent in the system has an independent Actor, whose goal is to learn a policy that allows it to choose an action that maximizes the expected reward evaluated by the Critic, given local observations.

[0117] The HAPPO algorithm updates its policy using data collected in the current iteration. To enable multiple gradient updates after a single data collection, weights are assigned based on importance. To correct the differences between the old and new strategies. This is a new strategy that needs to be optimized. It's an old strategy for collecting data.

[0118]

[0119] To prevent excessive size This leads to excessively large policy update steps, which compromises training stability. To address this, a pruning mechanism is introduced to restrict updates to the trusted region. Thus constructing .

[0120]

[0121] HAPPO uses sequential updates, when the agent... When updating, it is necessary to consider the agents that have already been updated. Strategy changes. Introduction of corrective factors. This is to compensate for the non-stationarity caused by policy updates from other agents.

[0122]

[0123] To prevent the strategy from converging prematurely to a suboptimal solution, in the objective function... Policy entropy was added. This is to encourage Actors to explore more diverse actions. This is a hyperparameter used to balance utilization and exploration.

[0124]

[0125]

[0126] The Actor and Critic networks update their parameters through gradient ascent, thereby maximizing the constructed objective function. During training, the Actor and Critic networks are optimized using the Adam method.

[0127] 4) Based on the unified model generated in step 1), the interaction environment is used to iteratively train the centralized Critic network and each Actor network constructed in step 3); the observation space data defined in step 2) is input into each Actor network to generate scheduling actions, and the parameters of each network are adjusted according to the feedback of the centralized Critic network using the sequential update strategy until the network converges, thereby obtaining the optimal cooperative scheduling strategy for each agent.

[0128] To verify the effectiveness of the method of the present invention, the present invention conducted an experimental verification of the effectiveness of multi-agent cooperation by selecting data from a typical day in July, and fully verified the effectiveness and superiority of the proposed method through comparative experiments of MAPPO and PPO methods and tests on typical day scenarios in different seasons.

[0129] a. Simulation examples and training parameter settings This experiment simulates a data center scenario consisting of 4000 servers. Weather data from California, USA, and IT load data from Alibaba's data center were used for model training, with a simulation step size of 15 minutes. The experiment sets the scheduleable load ratio to 30% and the maximum latency queue capacity to 1000. The battery capacity is configured at 0.2MW to simulate the regulation capability under limited energy supply conditions. The cooling system employs a hybrid mode combining mechanical and natural cooling to adapt to the tiered cooling needs of different seasons.

[0130] In this experiment, three independent agent models were constructed using a fully connected neural network. After experimental debugging, both the Actor and Critic networks were configured with three hidden layers, each with 16 neurons, to create a lightweight network structure and prevent overfitting. The tanh function was chosen as the activation function. To further improve training stability, all input features were normalized; the specific hyperparameter configurations are shown in Table 1.

[0131] The experiment built a data center economic scheduling simulation program based on the Gym platform, serving as an environment for multi-agent reinforcement learning training and performance evaluation. The algorithm program was implemented using Python 3.10.18 and PyTorch 2.0.1 frameworks, with a fixed random seed set to ensure the reproducibility of experimental results. After training, independent data from each season that was not used in training was used as the test set to verify the generalization ability and practical performance of the scheduling strategy.

[0132] Table 1 Hyperparameter Settings

[0133] b. Method convergence analysis The training and convergence process of the method is as follows: Figures 3 to 5As shown, in the early stages of training, the Critic network exhibits a large value loss, while the Actor network's policy loss and average step reward are both at low levels. As training progresses, the value loss rapidly decreases and eventually converges to 0, indicating that the Critic network's estimation of state values ​​gradually becomes more accurate. Based on this foundation, the policy losses of each agent also gradually converge to near 0, while the average step reward significantly increases and eventually stabilizes. This convergence process collectively verifies the effectiveness and stability of the proposed algorithm.

[0134] c. Algorithm scheduling result analysis c1) Typical Daily Results Using data from July for testing, the resulting typical daily scheduling scheme is as follows: Figures 6 to 11 As shown: Figure 6 The study demonstrates the load transfer behavior of the load scheduling agent under time-of-use pricing, and compares the distribution of the original load with the load after scheduling. Figure 7 This reflects the change in the length of the delayed queue. Combining the two graphs, the agent's scheduling behavior can be analyzed as follows: During the nighttime low electricity price period from 0:00 to 6:00, the agent processes a large amount of load backlogged in the queue; when the queue length is within a healthy range, it switches to processing only the current task. From 7:00 to 8:00, during the morning electricity price ramp-up phase, the agent prioritizes processing queue load to prepare for the subsequent morning electricity price peak. From 9:00 to 11:00, load execution is delayed, queue tasks backlog, and the delayed queue length increases accordingly. From 12:00 to 16:00, during the afternoon electricity price peak period, although the agent tends to continue delaying tasks, it still chooses to process some tasks to control risk due to queue health and delay time penalties. From 17:00 to 20:00, the highest electricity price of the day arrives; while continuously delaying a large number of tasks, the agent still processes a small number of tasks to maintain queue health and reduce average delay time. From 21:00 to 22:00, the electricity price drops sharply, and after experiencing consecutive peaks, the queue length reaches its peak at 21:00. After 11 PM, nighttime electricity prices gradually decreased, and the intelligent agents began to process the backlog of tasks, gradually clearing the queue.

[0135] Driven by a daily average electricity price of $0.2662 / kWh and a time-of-use price fluctuation range of $0.12–$0.42 / kWh, the load scheduling agent achieved efficient task time-series migration. Scheduling results show an average task queue length of 376.6, a peak queue length of 589, and a 0% task drop rate throughout the day, indicating that the strategy can ensure system reliability while responding to electricity price signals. Further observation reveals that the workload during the original peak hours was significantly shifted to the low-price hours, reflecting the high degree of price signal responsiveness of the scheduling process.

[0136] Figure 8This demonstrates the power consumption and energy usage effectiveness (PUE) of the entire system. Figure 9 The two graphs together illustrate the control results of the data center temperature regulation agent, allowing for joint analysis of the temperature control strategy. The PUE (Power Usage Effectiveness) metric is defined as follows:

[0137] The results are the complex coupling of time-of-use (TOU) pricing, time-of-use (TOU) pricing, and other factors. During the early morning hours when electricity prices are low, despite high IT load, the HVAC system significantly reduced cooling energy consumption by setting the temperature to 23°C and fully utilizing the approximately 12°C outdoor natural cooling source, achieving economical system operation. At 6:00 AM, as outdoor temperatures gradually rose and the load rate remained relatively low, the system continued to maintain the 23°C temperature setting, further reducing the use of mechanical cooling. At this time, the PUE remained at a moderate level of 1.22. At 2:00 PM, although electricity prices and outdoor temperatures were both high, and IT load was moderate, the system still set the temperature to 23°C, sacrificing local thermal environment for a significant reduction in electricity costs. While the PUE increased slightly during this period, overall operating costs were effectively controlled.

[0138] Under conditions where the outdoor temperature ranges from 12.06°C to 31.42°C (average 22.64°C) and the CRAC setpoint ranges from 18°C ​​to 23°C, the temperature regulation agent effectively achieved dynamic temperature control of the data center, driven by outdoor temperature fluctuations, IT load changes, and time-of-use pricing mechanisms. Typical daily scheduling results show that with the CRAC setpoint at 22.21°C ± 1.45°C, the system's average PUE remained stable at 1.255, demonstrating the excellent comprehensive control capability of the hybrid cooling strategy that combines natural cooling and mechanical refrigeration.

[0139] Figure 10 The operational strategy of the battery energy management agent during a typical day was demonstrated. The battery's state of charge was well maintained between 25% and 58%, with 23 charges completed during low-electricity-price periods and a total of 52.06 kWh of energy released during high-electricity-price periods. This agent effectively collaborated with the load scheduling agent, prioritizing data center load demands rather than simply pursuing electricity price arbitrage. Specifically, it performed charging when queue loads were high and utilized energy storage systems to discharge and support task processing, ensuring data center power reliability by sacrificing some high-priced battery energy.

[0140] Figure 11The system operating costs under the overall collaborative strategy are demonstrated. During a typical daily scheduling process, the load scheduling agent performed load transfer in 38.5% of the time steps, the temperature regulation agent participated in temperature regulation in 78.1% of the time steps, and the battery energy management agent performed energy management operations in 30.2% of the time steps. Through the coordinated operation of these three agents, the total daily operating cost of the system was $7409, resulting in a saving of $1223.86 (corresponding to a cost reduction of approximately 14.2%).

[0141] c2) Comparison of Algorithm Scheduling Results The summer season, with its most severe operating conditions, was selected as the test scenario to compare the performance of different algorithms. The results of the comparison are shown in Table 2.

[0142] Table 2 Performance of different scheduling methods on the test set

[0143] In terms of energy efficiency, the HAPPO algorithm achieves a PUE of 1.264, which is approximately 7% lower than the average of the other three algorithms. While the traditional MPC algorithm achieves the lowest theoretical daily operating cost, its computation time of 2645 seconds is insufficient for real-time requirements. In contrast, HAPPO, at a cost only about 1.5% higher, significantly reduces the computation time to 14.08 seconds, achieving the optimal balance between solution quality and response speed. The results demonstrate that HAPPO effectively addresses dynamic environmental changes while improving physical energy efficiency, achieving synergistic optimization of energy efficiency and economy.

[0144] In terms of service quality, all four scheduling algorithms demonstrated good stability, with no task dropouts observed in the test set. Regarding task backlog, the HAPPO algorithm significantly reduced the average number of tasks in the queue compared to PPO and MPC, alleviating task queuing congestion. Although HAPPO's queue length is slightly longer than MAPPO, it demonstrates a more balanced scheduling strategy by trading moderate task buffering for optimized system energy efficiency while ensuring no task dropouts.

[0145] In terms of computational load, the HAPPO algorithm accounts for 79.08% of the data center computing power, significantly higher than other comparative algorithms. This indicates that the algorithm successfully processes more computational tasks per unit time, and its optimization effect is not achieved by sacrificing computing performance to save energy, but rather by synergistically improving system energy efficiency and economy while maintaining higher computing throughput.

Claims

1. A data center economic optimization scheduling method based on multi-agent deep reinforcement learning, characterized in that, The method includes the following steps: 1) Establish a unified model covering three levels: CPU, server rack, and data center, to characterize the electrical-thermal-computational coupling relationship between IT system, cooling system, and power supply system, and construct an economic optimization scheduling problem based on the model; 2) The above-mentioned optimization scheduling problem is transformed into a Markov decision process, and based on the multi-agent cooperative scheduling mechanism, the observation space, action space and reward function of the load scheduling agent, the data center temperature regulation agent and the battery energy management agent are designed respectively. 3) Construct a multi-agent collaborative operation mechanism based on the Actor-Critic framework, including building a centralized Critic network and independent Actor networks for each agent, adopting a sequential update strategy and introducing a correction factor to handle the non-stationarity problem in the multi-agent policy update. 4) Based on the unified model generated in step 1), the interaction environment is used to iteratively train the centralized Critic network and each Actor network constructed in step 3); the observation space data defined in step 2) is input into each Actor network to generate scheduling actions, and the parameters of each network are adjusted according to the feedback of the centralized Critic network using the sequential update strategy until the network converges, thereby obtaining the optimal cooperative scheduling strategy for each agent.

2. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, Step 1) specifically refers to: 11) Establish a workload and task delay queue model Divide data center workloads into schedulable tasks. and unschedulable tasks Establish a task delay queue It is a first-in-first-out queue used to store delayed tasks; 12) Establish an IT system model A data center's IT system consists of multiple server racks, which are made up of multiple servers, and servers are made up of CPUs and IT fans. The IT system model includes a server-level CPU power consumption and IT fan power model, a rack-level inlet and outlet temperature model, and a data center-level total IT power consumption model. 13) Establish a cooling system model The cooling system consists of a computer room air conditioner, a chiller unit, a cooling tower, and cooling and chilled water pumps. Power and cooling capacity models are established for each of them. 14) Establish a battery model This includes charging / discharging states, power and energy dynamic models, and their operational constraints; 15) Optimization Problem Modeling The optimal scheduling of the entire system will use IT load rate, data center temperature regulation, and battery energy storage / discharge as decision variables at different time periods, with the objective of minimizing the system's economic operating cost, to solve the aforementioned nonlinear, nonconvex optimization model; where system operating cost... The electricity cost of the equipment is shown below: ； in, Time-of-use pricing; It is a response time constant; This represents the total number of time steps; This is the current time step; This refers to the battery's power. This represents the total power consumption of the cooling system. This represents the total power consumption of the data center IT system.

3. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that, Step 12) specifically involves: the IT fan dissipating heat generated during server operation to maintain normal CPU operation; and the CPU's instantaneous operating power. It is determined by the current server ingress temperature. and IT load rate The actual airflow of the IT fan is determined jointly. and instantaneous power Depends on its fan speed ratio A rack model is built using the CPU model, and the total CPU power of the rack is calculated. Total IT fan power and total air volume It is composed of all the servers in the rack, as shown below: ； ； Because there is no separation between the cold and hot aisle air in each rack, the inlet temperature of the rack... have: ； In the formula, The temperature set for CRAC; and These represent the minimum and maximum values ​​of the adjustable temperature range for CRAC. This is for supply air temperature difference, used to correct deviations in supply air temperature; rack outlet temperature have: ； In the formula, and These are the specific heat capacity and density of air, respectively; parameters All are empirical fitting coefficients: This is the thermal coupling strength coefficient between the heat source and the heat dissipation medium; and The tables respectively characterize the nonlinear effects of power and air volume on temperature rise; The calibration factor for the denominator term; Used to correct the baseline deviation of the model; A data center IT system consists of multiple server racks, and the total power consumption of a data center IT system is... and CPAC average return air temperature As shown below: 。 4. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that, Step 13) specifically involves: The CRAC (Radio Cooling and Air Conditioning Unit) is the air conditioning unit inside a data center. The CRAC needs to provide a cooling capacity of [missing information - likely a specific value]. ; The CPAC's internal fan drives air circulation for heat dissipation, so the fan's instantaneous power consumption... for: ； In the formula, CRAC airflow per unit IT power; This is the reference power for the CRAC fan; The airflow rate per unit IT power under rated operating conditions; The chiller unit consumes electricity to produce chilled water, thereby absorbing heat transferred from the CRAC. The maximum cooling capacity of the chiller unit will change with the outside temperature. Adjust according to changes; Define capacity correction factor To describe the maximum cooling capacity of the unit Temperature change characteristics: At the same time, a power correction coefficient is introduced. This is to reflect the changes in compressor power consumption characteristics under varying operating conditions. ； In the formula, , , This is the power curve coefficient; the energy efficiency degradation of the unit under non-full load conditions is corrected by the coefficient. describe: ； ； In the formula, This refers to the partial load rate of the chiller unit. , , The partial load curve coefficients are used; based on these coefficients, the actual power of the chiller unit is... for: ； In the formula, This is the start / stop correction factor; The rated energy efficiency ratio of the unit; This refers to the rated maximum cooling capacity under standard operating conditions. Cooling tower fan power consumption As shown below: ； In the formula, and This provides the reference power consumption and reference airflow for the CT fan. The required air volume for the CT; Power consumption of chilled water pumps and cooling water pumps As shown below: ； In the formula, For water pressure drop; The water flow rate of the water pump; For water pump efficiency; Represents chilled water pumps. Represents a cooling water pump; Total power consumption of cooling system HVAC and consume electrical energy for: ； 。 5. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 2, characterized in that, Step 14) specifically involves: The battery has two states: charging and charging. and discharge Status; the amount of charge the battery has during charging. From charging power Charging efficiency The impact; ； ； In the formula, It is a response time constant; Rated charging power coefficient; It is in a charged state; Battery capacity; This refers to the actual charging power. Rated charging power; discharge capacity of the battery during discharge. From output power and discharge efficiency Impact: ； In the formula, This is the rated discharge power coefficient; This represents the actual discharge power. Rated discharge power; battery power. and the battery level at the next moment for: ； ； The following constraints must be satisfied: 。 6. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, Step 2) specifically refers to: 21) The economic optimization scheduling problem is modeled as a distributed partially observable Markov decision process, Dec-POMDP; this Dec-POMDP process can be represented by a global tuple. To describe; It is the system's global state space. It is the joint action space of all intelligent agents; It is the state transition probability function, which depends on the current state. and joint actions ; It is a global reward function; It is the joint observation space of all intelligent agents, in time Each agent Able to receive its local observations , It is a discount factor; all agents make independent decisions based on local observations, finding a set of distributed strategies. This makes its joint strategy It can work together to achieve the optimal cumulative return for the entire system; Each agent makes decentralized decisions based on its local observations; each agent strategy Learning from its local observation space To its action space Mapping; 22) Design the observation space, action space, and reward function of the load scheduling agent; 23) Design the observation space, action space, and reward function of the data center temperature regulation agent; 24) Design the observation space, action space, and reward function of the battery energy management temperature regulation agent.

7. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 6, characterized in that, The observation space of the load scheduling agent includes the current total load, schedulable load, unschedulable load, queue status, task waiting time, and predicted load; the action space includes three types of discrete actions: delaying tasks, processing the current load, and processing the current and queue loads; the reward function consists of task discard penalty, queue health reward, and task delay penalty. The observation space of the data center temperature regulation agent includes the current temperature setting, rack inlet / outlet temperature, outdoor temperature, and predicted temperature; the action space includes three types of discrete actions: cooling, maintaining, and heating; the reward function is based on the deviation between the CRAC set temperature and the reference temperature, and is evaluated using a Gaussian function. The observation space of the battery energy management agent includes battery state of charge, electricity price, charging / discharging power, and total system power; the action space includes three types of discrete actions: charging, discharging, and inaction; and the reward function consists of battery arbitrage reward and battery health reward.

8. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, Step 3) specifically includes: 31) Construct a multi-agent optimization operation framework Based on the joint action of all agents Transition to the next state and generate an immediate reward; train by repeatedly performing the above interaction process and continuously optimize the policy function using the heterogeneous agent proximal policy optimization algorithm HAPPO. 32) Construct the central Critic network for the intelligent agents; 33) Construct an Actor network for each agent and introduce a correction factor for sequential updates.

9. A data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 8, characterized in that, Step 32) specifically involves: The central Critic network starts from the global state. Starting with the generalized advantage estimation (GAE), we calculate the advantage function. Value function under old parameters Building a critic network goal : Using the clipped value function Construction cutting error This restricts updates to a trusted region to enhance training stability; The central Critic network minimizes the loss function through gradient descent. Used to minimize current value and returns The error between them.

10. The data center economic optimization scheduling method based on multi-agent deep reinforcement learning according to claim 8, characterized in that, Step 33) specifically involves: Each agent has an independent Actor, whose goal is to learn a policy that allows it to choose an action that maximizes the expected reward evaluated by the Critic, given local observations. The HAPPO algorithm updates its policy using data collected in the current iteration; to enable multiple gradient updates after a single data collection, weights are assigned based on importance. Correcting the differences between the old and new strategies; A pruning mechanism has been introduced to restrict updates to trusted areas. Thus constructing : ； The HAPPO algorithm uses sequential updates, when the agent... When updating, it is necessary to consider the agents that have already been updated. Strategy changes; introduction of corrective factors This is to compensate for the non-stationarity caused by policy updates from other agents: ； To prevent the strategy from converging prematurely to a suboptimal solution, in the objective function... Policy entropy was added. To encourage Actors to explore more diverse actions; among them, These are hyperparameters used to balance utilization and exploration: ； ； The Actor and Critic networks update parameters through gradient ascent, thereby maximizing the constructed objective function.

Images