Data center cooling system energy consumption optimization method based on deep reinforcement learning

Abstract

The invention provides a data center cooling system energy consumption optimization method based on deep reinforcement learning, and relates to the technical field of data center energy consumption management, and the method comprises the steps: obtaining multi-source data of a data center, and carrying out the preprocessing; training the hybrid model by using the processed data to obtain a target hybrid model, and then performing interactive training on the deep reinforcement learning model by taking the target hybrid model as a simulation environment to obtain a target deep reinforcement learning model; and finally, outputting an optimal cooling system regulation and control signal through interaction between the target deep reinforcement learning model and a real environment, and updating the target hybrid model and the target deep reinforcement learning model. According to the method, the hybrid model simulation environment is constructed, the modeling process of the cooling system is simplified under the condition that the simulation precision is ensured, and the energy consumption optimization level of the data center cooling system is improved through interaction output of the target deep reinforcement learning model and the environment.

Description

Technical Field

[0001] This invention relates to the field of data center energy management technology, specifically to a data center cooling system energy consumption optimization method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of the digital economy, medium and large-sized data centers, as the core of computing power support, are experiencing a continuous expansion in energy consumption. Among these, the cooling system, which uses chilled water, is one of the main energy-consuming units in data centers. The energy management of the cooling system is directly related to the operating costs, energy efficiency, and long-term stable operation of the data center. Therefore, optimizing the energy consumption of the cooling system has become a key requirement for the green development of data centers.

[0003] Traditional methods for managing cooling system energy consumption using deep reinforcement learning are based on model-free deep reinforcement learning algorithms. These algorithms can quickly adapt to new environments through continuous learning when the environment changes, thereby controlling the data center cooling system.

[0004] However, this method requires a large amount of real-world interaction to acquire data, resulting in low sample collection efficiency, long training time, and high trial-and-error costs. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a data center cooling system energy consumption optimization method based on deep reinforcement learning, which solves the problems of high sample dependence and long training time in traditional cooling system energy consumption management.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, this application provides a method for optimizing the energy consumption of a data center cooling system based on deep reinforcement learning, the method comprising: Acquire multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data; The multi-source data is cleaned and normalized to obtain the target dataset; The target hybrid model is obtained by training a pre-built hybrid model using the target dataset; the hybrid model includes a mathematical model and a neural network prediction model. The target hybrid model is used as a simulation environment for interactive training of a pre-built deep reinforcement learning model to obtain a target deep reinforcement learning model; the deep reinforcement learning model is built based on the improved TD3 algorithm; wherein, the improved TD3 algorithm is designed with two independent branches in the action network of the original TD3 algorithm and an attention layer is added; The target deep reinforcement learning model interacts with the real environment to output the optimal cooling system control signal; The data center is regulated according to the cooling system control signal, and the target hybrid model and the target deep reinforcement learning model are updated according to the data center operation data after regulation.

[0007] Preferably, the step of interactively training the target hybrid model as a simulation environment of a pre-built deep reinforcement learning model to obtain the target deep reinforcement learning model includes: evaluating the decision performance of the deep reinforcement learning model using a reward function, and selecting the deep reinforcement learning model that yields the maximum total reward value calculated by the reward function as the target deep reinforcement learning model; wherein, the expression of the reward function is: In the formula, express i Total reward value at any given moment; Energy consumption bonus for the cooling system; Incentives for compliance with chilled water temperature difference regulations; This indicates a reward for compliance with cooling water temperature difference regulations. This represents the reward for changes in chilled water and cooling water temperatures; the smaller the changes in chilled water and cooling water temperatures, the more stable the system operation, hence the reward value. The coefficient is negative; , , and These are the weighting coefficients for each reward, which are used to balance the reward values ​​of each part.

[0008] Preferably, the deep reinforcement learning model includes state variables, which represent the state information of the data center and are used to provide the basic information required for decision-making by the deep reinforcement learning model. The expression for the state variables is: In the formula, express Time-state variables; The ambient temperature; To meet cooling load requirements; This refers to the temperature difference of the cooling water. Total power of the cooling system; This is the control signal for the chilled water pump; This is the control signal for the cooling water pump; This refers to the outlet temperature of the chilled water. This represents the rate of change of chilled water temperature. This represents the rate of change of cooling water temperature difference. This represents the chilled water outlet temperature at the previous moment.

[0009] Preferably, the deep reinforcement learning model includes action variables, whereby the action vector represents the control signal output by the deep reinforcement learning model, and the expression for the action variables is: In the formula, express Action variables at time; This indicates the control signal for the chilled water pump; This indicates the control signal for the cooling water pump.

[0010] Preferably, the step of regulating the data center according to the cooling system regulation signal and updating the target hybrid model and the target deep reinforcement learning model according to the regulated data center operating data includes: Collect real-time operational data after data center adjustments; If the difference between the real-time running data and the simulated data output by the hybrid model is greater than a preset threshold, the parameters of the target hybrid model are updated using the real-time running data. The updated target fusion model is used as a simulation environment to incrementally train the TD3 algorithm of the target deep reinforcement learning model, resulting in the updated target deep reinforcement learning model.

[0011] Preferably, the cooling system operating data includes chilled water supply and return water temperatures, cooling water supply and return water temperatures, chilled water flow rate, cooling water flow rate, cooling tower air flow rate, chilled water pump power, cooling water pump power, chiller unit power, and cooling tower power; the environmental data includes ambient temperature, data center interior ambient temperature, and humidity.

[0012] Preferably, the mathematical model includes calculations of cooling system energy consumption, chilled water pump power, cooling water pump power, chilled water supply and return water temperature difference, cooling water supply and return water temperature difference, actual chilled water mass flow rate, and actual cooling water mass flow rate; the neural network prediction model includes an input layer, a shared feature extraction layer, a prediction layer, and an output layer, wherein the shared feature extraction layer includes two hidden layers, and the prediction layer includes a temperature branch and a power branch.

[0013] Secondly, this application also provides a data center cooling system energy consumption optimization system based on deep reinforcement learning, the system comprising: The data acquisition module acquires multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data. The data processing module cleans and normalizes the multi-source data to obtain the target dataset; The first model training module trains the pre-built hybrid model using the target dataset to obtain the target hybrid model; the hybrid model includes a mathematical model and a neural network prediction model. The second model training model uses the target hybrid model as a simulation environment for interactive training of a pre-built deep reinforcement learning model to obtain the target deep reinforcement learning model; the deep reinforcement learning model is built based on the improved TD3 algorithm; wherein, the improved TD3 algorithm is designed with two independent branches in the action network of the original TD3 algorithm and an attention layer is added; The strategy output module interacts with the real environment through the target deep reinforcement learning model to output the optimal cooling system control signal; The strategy execution feedback module regulates the data center according to the cooling system control signal, and updates the target hybrid model and the target deep reinforcement learning model according to the data center operation data after regulation.

[0014] Thirdly, this application also provides a computer-readable storage medium storing a computer program for energy consumption optimization of a data center cooling system based on deep reinforcement learning, wherein the computer program causes a computer to execute the energy consumption optimization method for a data center cooling system based on deep reinforcement learning as described above.

[0015] Fourthly, this application also provides an electronic device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing energy consumption optimization of data center cooling systems based on deep reinforcement learning as described above.

[0016] (III) Beneficial Effects This invention provides a method and system for optimizing energy consumption in data center cooling systems based on deep reinforcement learning. Compared with existing technologies, it has the following advantages: 1. This application considers the collaborative management of multiple objectives by acquiring multi-source data information such as cooling system operation data, environmental data, and data center cooling load data; it calculates quantifiable parameters through mathematical models and solves the accurate prediction of complex parameters through neural network prediction models, providing a high-precision training environment for deep reinforcement learning agents; by constructing a simulated operating environment to replace real environment interaction, it significantly reduces the dependence on real data, improves sample collection efficiency, and shortens the training cycle; in addition, the hybrid model combines the determinism of mathematical models with the prediction accuracy of neural networks, ensuring that system parameters are within a safe range during training, reducing trial and error risks, and solving the problems of difficult modeling and low accuracy.

[0017] 2. This application improves the traditional TD3 algorithm structure by designing a dual independent branch in the action network based on the original TD3 algorithm structure. This decouples the control logic of the chilled water pump and the cooling water pump, avoids interference between actions, and generates a more independent control strategy. At the same time, an attention layer is added to the action network to enhance the algorithm's attention to key state features, accelerate the convergence speed, and improve the performance of the deep reinforcement learning algorithm.

[0018] 3. This application establishes a control feedback mechanism, which updates the hybrid model and deep reinforcement learning model by detecting the data after control, thus completing a dynamic optimization loop and enabling timely and reasonable control in response to fluctuations in the system. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the energy consumption optimization method for a data center cooling system provided in this application embodiment; Figure 2 A schematic diagram of the data center cooling system energy consumption optimization system provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This application provides a data center cooling system energy consumption optimization method based on deep reinforcement learning, which solves the problem of the large limitations in traditional cooling system energy consumption management, realizes precise control of the cooling system, and improves the energy consumption management effect of the data center cooling system.

[0023] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: Traditional PID (Proportional-Integral-Derivative Control) methods focus solely on correcting parameter deviations during regulation. This can lead to overshooting during data center cooling load fluctuations, increasing energy consumption and potentially causing localized overheating or overcooling of servers. MPC (Model Predictive Control) methods rely on precise mathematical models of the cooling system. However, data center cooling systems are affected by external nonlinear factors such as wind speed and humidity, making real-time dynamic correction difficult. This results in accumulated model errors and a significantly increased probability of control strategy failure. Traditional reinforcement learning focuses solely on reducing energy consumption, neglecting the impact of policy fluctuations on equipment. Frequent and significant adjustments to control signals can shorten the lifespan of core equipment such as pumps and chillers. Model-free deep reinforcement learning methods require extensive real-world environmental interaction data, resulting in low sample efficiency, long training times, and high trial-and-error costs due to potential deviations in system parameters from safe ranges. Model-based deep reinforcement learning methods, due to the coupling of multiple components in the cooling system, struggle to accurately simulate dynamic characteristics, making the construction of accurate environmental models extremely difficult.

[0024] To address the shortcomings of existing technologies, this application provides a data center cooling system energy consumption optimization method based on deep reinforcement learning. To enhance the multi-objective optimization capability of the cooling system, relevant variables such as energy consumption, chilled water temperature difference, and cooling water temperature difference are incorporated into the reward function. This enables multi-objective collaborative management, improving the algorithm's multi-objective optimization capability for the cooling system's energy consumption and temperature, and guiding the strategy to output more stable control signals, further reducing strategy fluctuations and improving the cooling system's operational performance. To resolve the contradiction between the difficulty and low accuracy of cooling system modeling, a hybrid model is constructed as the training environment for the deep reinforcement learning agent, replacing real-world interaction. This significantly reduces reliance on real data, improves sample efficiency, and shortens the training cycle. Furthermore, the hybrid model combines the determinism of mathematical models with the predictive accuracy of neural networks, ensuring that system parameters remain within a safe range during training, reducing trial-and-error risks, and ultimately resolving the contradiction between modeling difficulty and low accuracy. To improve the energy management of the cooling system, the traditional TD3 algorithm structure is improved by adding an attention layer and independent action branches to the action network. This enhances the algorithm's attention to key state features and decouples the control logic of the chilled water pump and cooling water pump, avoiding mutual interference between actions. This improves the fine-grained control capability and convergence speed of the strategy, thereby enhancing the performance of the deep reinforcement learning algorithm and ultimately achieving the goal of improving the energy management of the cooling system.

[0025] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0026] like Figure 1As shown in the figure, this application provides a method for optimizing the energy consumption of a data center cooling system based on deep reinforcement learning, which specifically includes the following steps: Step S110: Obtain multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data; Step S120: Clean and normalize the multi-source data to obtain the target dataset; Step S130: Train the pre-built hybrid model using the target dataset to obtain the target hybrid model; the hybrid model includes a mathematical model and a neural network prediction model. Step S140: The target fusion model is used as a simulation environment for the pre-built deep reinforcement learning model for interactive training to obtain the target deep reinforcement learning model; the deep reinforcement learning model is built based on the improved TD3 algorithm; wherein, the improved TD3 algorithm is to design a dual independent branch in the action network of the original TD3 algorithm and add an attention layer; Step S150: The target deep reinforcement learning model interacts with the real environment to output the optimal cooling system control signal. Step S160: Adjust the data center according to the cooling system control signal, and update the target hybrid model and the target deep reinforcement learning model according to the adjusted data center operation data.

[0027] The data center cooling system energy consumption optimization method based on deep reinforcement learning provided in this application constructs a hybrid model of the cooling system by integrating a mathematical model and a dual-branch neural network prediction model. This model can calculate quantifiable parameters such as pump power and temperature difference through mathematical formulas, and can also make accurate predictions of complex parameters through neural networks. The system simulation operating environment is constructed using quantifiable parameters and prediction data to provide a high-precision training environment for the deep reinforcement learning agent.

[0028] Step S110: Obtain multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data.

[0029] Specifically, the collection of multi-source data from the data center provides input support for the hybrid model and deep reinforcement learning algorithm. (1) Environmental data includes external temperature, internal temperature and humidity of the data center; (2) Cooling system operation data includes chilled water supply and return water temperature, cooling water supply and return water temperature, chilled water flow rate, cooling water flow rate, cooling tower air flow rate, chilled water pump power, cooling water pump power, chiller unit power and cooling tower power; (3) Data center cooling load data includes IT equipment cooling load data, building envelope cooling load data and lighting cooling load data. It should be noted that although personnel and infiltration cooling load data are part of the cooling load data, their proportion is extremely small and can be ignored in this embodiment. The calculation method of data center cooling load data is as follows: (1) in, Indicates the cooling load of IT equipment. Indicates the cooling load of the building envelope. Indicates the cooling load of the lighting. For the first Each time unit This is the total time unit.

[0030] (1-1) in, Indicates server cold load. Indicates the cooling load of the power distribution unit. This indicates the cold load of the uninterruptible power supply.

[0031] (1-2) Where K is the heat transfer coefficient of the building envelope, A is the heat transfer area, and T is the outdoor ambient temperature. This refers to the internal ambient temperature of the data center.

[0032] (1-3) in, Illuminance per unit area inside the data center. This refers to the area of ​​the data center.

[0033] Step S120: Clean and normalize the multi-source data to obtain the target dataset.

[0034] Specifically, data cleaning is divided into outlier detection and missing value imputation. Outlier detection can be based on the 3σ principle to remove data points exceeding the mean ± 3 standard deviations. Missing value imputation can be performed using linear interpolation, with the specific imputation formula as follows: (2) in, Indicates the location of missing values The value of is determined by the value of the preceding valid data point. The value of the next valid data point is .

[0035] Preferably, this embodiment uses the Min-Max normalization method, and the specific formula is as follows: (3) in, Represents the original data. This represents the normalized data. This represents the minimum value of the data. This indicates the maximum value of the data.

[0036] In this embodiment of the application, multi-source data is collected and preprocessed to lay the foundation for subsequent steps.

[0037] Step S130: Train the pre-built hybrid model using the target dataset to obtain the target hybrid model; the hybrid model includes a mathematical model and a neural network prediction model.

[0038] Specifically, the mathematical model includes calculations of cooling system energy consumption, chilled water pump power, cooling water pump power, chilled water supply and return water temperature difference, cooling water supply and return water temperature difference, actual chilled water mass flow rate, and actual cooling water mass flow rate; the neural network prediction model includes an input layer, a shared feature extraction layer, a prediction layer, and an output layer. The shared feature extraction layer includes two hidden layers, and the prediction layer includes a temperature branch and a power branch.

[0039] The target dataset is input into the hybrid model, and the power of the chilled water pump, the power of the cooling water pump, the power of the cooling tower, and the temperature deviation are calculated through the mathematical model. The neural network prediction model outputs the power of the chiller unit and the cooling water supply temperature. The two are integrated through parameter transfer to form a complete cooling system operation simulation environment, which supports the state interaction and action feedback of the deep reinforcement learning agent.

[0040] The specific expression of the model is shown below: (1) Mathematical model (4) in, The total power of the cooling system, This refers to the actual power of the chilled water pump. This represents the actual power of the chiller unit. This represents the actual power of the cooling water pump. This represents the actual power of the cooling tower. For the first Each time unit This is the total time unit.

[0041] (4-1) in, This refers to the rated power of the chilled water pump. This is the control signal for the chilled water pump; This refers to the rated power of the cooling water pump. This is the control signal for the cooling water pump; These are the polynomial coefficients of the chilled water pump, obtained by fitting cooling system operating data. This represents the highest degree of the polynomial for the chilled water pump.

[0042] (4-2) in, These are the polynomial coefficients of the cooling water pump, obtained by fitting cooling system operating data. This represents the highest degree of the polynomial for the cooling water pump.

[0043] (4-3) in, For the temperature difference of chilled water, This refers to the outlet temperature of the chilled water. For chilled water supply temperature, To meet the cooling load demand of air conditioning terminals, This refers to the specific heat capacity of chilled water. This represents the actual mass flow rate of the chilled water.

[0044] (4-4) in, For the cooling water temperature difference, This refers to the outlet temperature of the cooling water. For cooling water supply temperature, To meet the cooling load demand of air conditioning terminals, This represents the actual power of the chiller unit. The specific heat capacity of cooling water, This represents the actual mass flow rate of the cooling water.

[0045] (4-5) (4-6) in, This is the rated mass flow rate of the chilled water; This refers to the rated mass flow rate of the cooling water. and These are control signals for the chilled water pump and the cooling water pump, respectively, with values ​​ranging from 0 to 1.

[0046] (2) Neural network prediction model In this embodiment of the application, the power of the chiller unit in the chilled water system is affected by the cooling load demand, the characteristics of the equipment itself, and the operating environment. The cooling water supply temperature is the result of heat exchange in multiple stages and is affected by external environmental factors, cooling tower operation factors, cooling water pump operation factors, and heat release factors of the chiller unit. This results in the chiller unit power and cooling water supply temperature not being independent parameters, but forming a complex relationship of coupled loops and nonlinear responses, making the numerical calculation difficult. Therefore, this embodiment of the invention uses a two-branch fully connected neural network structure to predict the cooling water supply temperature and chiller unit operating power during the operation of the cooling system, so as to simplify the modeling process of the cooling system.

[0047] Specifically, the dual-branch fully connected neural network structure used in this application includes an input layer, a shared feature extraction layer, a prediction layer, and an output layer. The shared feature extraction layer is represented by two hidden layers to learn a general representation of the input features. The prediction layer is divided into a temperature branch and a power branch to improve the independence of temperature and power predictions.

[0048] The dual-branch fully connected neural network of this application embodiment receives an 11-dimensional feature vector in its input layer, including time series, ambient temperature, chilled water supply temperature, actual chilled water mass flow rate, actual cooling water mass flow rate, cooling load demand, chiller unit rated capacity, cooling tower power, cooling tower air velocity, chilled water outlet temperature, and chilled water temperature difference. The output layer outputs the cooling water supply temperature and the chiller unit operating power. The activation functions are all modified linear unit functions to enhance nonlinear expression capabilities. Training employs an adaptive moment estimation optimizer to minimize the mean square error loss function. The mean square error of the cooling water supply temperature is calculated as shown in formula (5-1), and the mean square error of the chiller unit operating power is calculated as shown in formula (5-2). (5-1) (5-2) In the formula, The mean square error for predicting the cooling water supply temperature. The number of samples in the training dataset. For the first The actual value of cooling water supply temperature in each sample For the first Predicted cooling water supply temperature for each sample This represents the predicted mean square error of the chiller unit's operating power. For the first The true value of the operating power of the chiller units in each sample. For the first Predicted operating power of chiller units for each sample.

[0049] The data center cooling system energy consumption optimization method of this application uses a mathematical model to calculate the power of chilled water pump, cooling water pump, cooling tower, and temperature deviation. It uses a neural network prediction model to output the power of chiller unit and the cooling water supply temperature. The two are integrated through parameter transfer to form a complete cooling system operation simulation environment, which supports the state interaction and action feedback of deep reinforcement learning agents.

[0050] Step S140: The target fusion model is used as a simulation environment for the pre-built deep reinforcement learning model for interactive training to obtain the target deep reinforcement learning model; the deep reinforcement learning model is built based on the improved TD3 algorithm; wherein, the improved TD3 algorithm is to design a dual independent branch in the action network of the original TD3 algorithm and add an attention layer.

[0051] In this embodiment of the application, in order to resolve the contradiction between the difficulty and low accuracy of modeling the cooling system, a hybrid model is constructed as the training environment for the deep reinforcement learning agent. This replaces the interaction with the real environment, greatly reduces the dependence on real data, improves the efficiency of sample acquisition, and shortens the training cycle. In addition, the hybrid model combines the determinism of the mathematical model with the prediction accuracy of the neural network, ensuring that the system parameters are within a safe range during the training process, reducing the risk of trial and error, and resolving the contradiction between the difficulty and low accuracy of modeling.

[0052] The deep reinforcement learning model based on the improved TD3 algorithm uses the improved TD3 algorithm as its core to construct a Markov decision process of "state-action-reward" to achieve optimal energy management strategy learning. It mainly includes state variables, action variables and reward function.

[0053] (1) State variables: (6) in, express Time-state variables; The ambient temperature; To meet cooling load requirements; This refers to the temperature difference of the cooling water. Total power of the cooling system; This is the control signal for the chilled water pump; This is the control signal for the cooling water pump; This refers to the outlet temperature of the chilled water. This represents the rate of change of chilled water temperature. This represents the rate of change of cooling water temperature difference. This represents the chilled water outlet temperature at the previous moment.

[0054] In this embodiment, the state variables can characterize the specific state of the data center at a certain moment, providing the basic information required for decision-making in deep reinforcement learning algorithms, and will gradually transition to the next state as the agent takes actions on the environment. To more comprehensively represent the state information of the data center, the state variables in the improved TD3 algorithm of this invention include ambient temperature, cooling load demand, cooling water temperature difference, total power of the cooling system, chilled water pump control signal, cooling water pump control signal, chilled water outlet temperature, chilled water temperature change rate, cooling water temperature difference change rate, and the chilled water outlet temperature at the previous moment.

[0055] (2) Action variables: (7) in, express Action variables at time; This indicates the control signal for the chilled water pump; This indicates the control signal for the cooling water pump.

[0056] In this embodiment, action variables characterize the decisions made by the agent based on the current state. The algorithm incorporates two dimensions for these action variables: the chilled water pump control signal and the cooling water pump control signal. By adding two independent action branches to the action network, the control logic of the chilled water pump and cooling water pump is decoupled, preventing interference between actions, generating more independent control strategies, and improving the fine-grained control capability of the strategies.

[0057] (3) Reward function: (8) In the formula, express i Time-based reward function; Energy consumption bonus for the cooling system; Incentives for compliance with chilled water temperature difference regulations; This indicates a reward for compliance with cooling water temperature difference regulations. This represents the reward for changes in chilled water and cooling water temperatures. The smaller the changes in chilled water and cooling water temperatures, the more stable the system operation, hence the reward value. The coefficient is negative; , , and These are the weighting coefficients for each reward, which are used to balance the reward values ​​of each part.

[0058] In this embodiment, the reward function is used to evaluate the agent's performance in each decision, guiding the model to learn better strategies. During agent training, the agent's actions in each decision cycle are given a reward value, representing the environment's feedback to the agent's actions. The goal of agent training in deep reinforcement learning is to obtain the maximum cumulative reward. The reward function in this embodiment consists of four parts: cooling system energy consumption reward, chilled water temperature difference compliance reward, cooling water temperature difference compliance reward, and chilled water and cooling water temperature change reward. Furthermore, this embodiment uses weight coefficients to balance the reward values ​​of each part. If the difference in reward values ​​is too large, the algorithm may over-satisfy the requirements of a certain dimension, causing the model to get stuck in a local optimum. If the difference is too small, the algorithm will not be able to learn effectively. Therefore, the weight coefficients must be determined experimentally.

[0059] In this embodiment, a deep reinforcement learning model is trained in a cooling system operation simulation environment. By continuously training, a strategy that maximizes the reward value is obtained, and the cooling system is regulated to ultimately improve the energy management effect of the data center cooling system.

[0060] Step S150: The target deep reinforcement learning model interacts with the real environment to output the optimal cooling system control signal.

[0061] Specifically, the trained deep reinforcement learning module based on the improved TD3, which is deployed in the actual data center cooling system, outputs control signals for the chilled water pump and the cooling water pump to adjust the flow rate of chilled water and cooling water.

[0062] Step S160 involves regulating the data center according to the cooling system control signal, and updating the target mixture model and the target deep reinforcement learning model based on the regulated data center operating data. The specific implementation of this step includes the following steps: Step S161: Collect real-time operational data after data center adjustments.

[0063] Step S162: In response to the difference between the real-time running data and the simulated data output by the hybrid model being greater than a preset threshold, the parameters of the target hybrid model are updated using the real-time running data.

[0064] Step S163: Use the updated target fusion model as a simulation environment to incrementally train the TD3 algorithm of the target deep reinforcement learning model to obtain the updated target deep reinforcement learning model.

[0065] Specifically, actual operating data is collected every hour, including equipment energy consumption, temperature difference between chilled water and cooling water, flow rate of chilled water and cooling water, and operating status data of each piece of equipment. This data is compared with the predicted values ​​of the hybrid model. If the deviation exceeds the set threshold, the parameters of the target hybrid model are updated using real-time operating data. The updated target hybrid model is then used as a simulation environment to incrementally train the TD3 algorithm of the target deep reinforcement learning model, resulting in an updated target deep reinforcement learning model. This ensures that the policy adapts to the parameter updates during the long-term operation of the system.

[0066] like Figure 2 As shown, this embodiment of the invention also provides an energy consumption optimization system for a data center cooling system based on deep reinforcement learning, comprising: Data acquisition module 210 acquires multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data. Data processing module 220 cleans and normalizes multi-source data to obtain the target dataset; The first model training module 230 trains the pre-built hybrid model using the target dataset to obtain the target hybrid model; the hybrid model includes a mathematical model and a neural network prediction model. The second model training model 240 uses the target fusion model as a simulation environment for interactive training of a pre-built deep reinforcement learning model to obtain the target deep reinforcement learning model. The deep reinforcement learning model is built based on the improved TD3 algorithm. The improved TD3 algorithm is designed with two independent branches in the action network of the original TD3 algorithm and an attention layer is added. The strategy output module 250 interacts with the real environment through the target deep reinforcement learning model to output the optimal cooling system control signal; The strategy execution feedback module 260 regulates the data center according to the cooling system control signal, and updates the target hybrid model and the target deep reinforcement learning model based on the data center operation data after regulation.

[0067] It is understood that the data center cooling system based on deep reinforcement learning provided in this embodiment of the invention corresponds to the data center cooling method based on deep reinforcement learning described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the data center cooling method based on deep reinforcement learning, and will not be repeated here.

[0068] This invention also provides a computer-readable storage medium storing a computer program for a deep reinforcement learning-based data center cooling method, wherein the computer program causes a computer to execute the deep reinforcement learning-based data center cooling method as described above.

[0069] This application also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing a data center cooling method based on deep reinforcement learning as described above.

[0070] In summary, compared with existing technologies, it has the following beneficial effects: 1. The hybrid model constructed in this application embodiment integrates a mathematical model and a dual-branch neural network prediction model. It calculates quantifiable parameters, such as pump power and temperature difference, through mathematical formulas, and solves the accurate prediction of complex parameters through neural networks. This constructs a simulated operating environment to simplify the modeling process, reduce environmental errors, reduce sample dependence, and provide a high-precision training environment for deep reinforcement learning agents.

[0071] 2. The improved TD3 algorithm structure provided in this application adds an attention layer to the action network to dynamically increase the weight of key state features during algorithm training, enabling the network to focus on key states more quickly and accelerate convergence, resulting in better model training performance. At the same time, it adopts a dual independent action branch design to decouple the control logic of chilled water pump and cooling water pump, avoid interference between actions, and generate a better control strategy.

[0072] 3. This application embodiment sets up a multi-objective reward function. By adding relevant variables such as energy consumption, chilled water temperature difference, and cooling water temperature difference to the reward function, "energy consumption reduction - temperature difference compliance - strategy stability" are incorporated into a unified reward framework. By balancing the objectives through weight coefficients, temperature runaway caused by single energy consumption optimization is avoided, and the impact of strategy fluctuations on equipment lifespan is prevented, thus achieving the triple objectives of "energy saving - stability - long lifespan".

[0073] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing energy consumption in a data center cooling system based on deep reinforcement learning, characterized in that, The method includes: Acquire multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data; The multi-source data is cleaned and normalized to obtain the target dataset; The target hybrid model is obtained by training a pre-built hybrid model using the target dataset; the hybrid model includes a mathematical model and a neural network prediction model. The target hybrid model is used as a simulation environment for interactive training of a pre-built deep reinforcement learning model to obtain a target deep reinforcement learning model; the deep reinforcement learning model is built based on the improved TD3 algorithm; wherein, the improved TD3 algorithm is designed with two independent branches in the action network of the original TD3 algorithm and an attention layer is added; The target deep reinforcement learning model interacts with the real environment to output the optimal cooling system control signal; The data center is regulated according to the cooling system control signal, and the target hybrid model and the target deep reinforcement learning model are updated according to the data center operation data after regulation.

2. The data center cooling system energy consumption optimization method according to claim 1, characterized in that, The step of interactively training the target hybrid model as a simulation environment for a pre-built deep reinforcement learning model to obtain the target deep reinforcement learning model includes: evaluating the decision performance of the deep reinforcement learning model using a reward function, and selecting the deep reinforcement learning model that yields the maximum total reward value calculated by the reward function as the target deep reinforcement learning model; wherein, the expression for the reward function is: In the formula, express i Total reward value at any given moment; Energy consumption bonus for the cooling system; Incentives for compliance with chilled water temperature difference regulations; This indicates a reward for compliance with cooling water temperature difference regulations. This represents the reward for changes in chilled water and cooling water temperatures; the smaller the changes in chilled water and cooling water temperatures, the more stable the system operation, hence the reward value. The coefficient is negative; , , and These are the weighting coefficients for each reward, which are used to balance the reward values ​​of each part.

3. The data center cooling system energy consumption optimization method according to claim 2, characterized in that, The deep reinforcement learning model includes state variables, which represent the state information of the data center and are used to provide the basic information required for decision-making in the deep reinforcement learning model. The expression for the state variables is: In the formula, express Time-state variables; The ambient temperature; To meet cooling load requirements; This refers to the temperature difference of the cooling water. Total power of the cooling system; This is the control signal for the chilled water pump; This is the control signal for the cooling water pump; This refers to the outlet temperature of the chilled water. This represents the rate of change of chilled water temperature. This represents the rate of change of cooling water temperature difference. This represents the chilled water outlet temperature at the previous moment.

4. The data center cooling system energy consumption optimization method according to claim 3, characterized in that, The deep reinforcement learning model includes action variables, where the action vector represents the control signal output by the deep reinforcement learning model, and the expression for the action variables is: In the formula, express Action variables at time; This indicates the control signal for the chilled water pump; This indicates the control signal for the cooling water pump.

5. The data center cooling system energy consumption optimization method according to claim 1, characterized in that, The step of regulating the data center according to the cooling system regulation signal and updating the target hybrid model and the target deep reinforcement learning model according to the regulated data center operating data includes: Collect real-time operational data after data center adjustments; If the difference between the real-time running data and the simulated data output by the hybrid model is greater than a preset threshold, the parameters of the target hybrid model are updated using the real-time running data. The updated target fusion model is used as a simulation environment to incrementally train the TD3 algorithm of the target deep reinforcement learning model, resulting in the updated target deep reinforcement learning model.

6. The data center cooling system energy consumption optimization method according to claim 1, characterized in that, The cooling system operating data includes chilled water supply and return water temperatures, cooling water supply and return water temperatures, chilled water flow rate, cooling water flow rate, cooling tower air flow rate, chilled water pump power, cooling water pump power, chiller unit power, and cooling tower power; the environmental data includes ambient temperature, data center interior temperature, and humidity.

7. The data center cooling system energy consumption optimization method according to claim 1, characterized in that, The mathematical model includes calculations of cooling system energy consumption, chilled water pump power, cooling water pump power, chilled water supply and return water temperature difference, cooling water supply and return water temperature difference, actual chilled water mass flow rate, and actual cooling water mass flow rate. The neural network prediction model includes an input layer, a shared feature extraction layer, a prediction layer, and an output layer. The shared feature extraction layer includes two hidden layers, and the prediction layer includes a temperature branch and a power branch.

8. A data center cooling system energy consumption optimization system based on deep reinforcement learning, characterized in that, The system includes: The data acquisition module acquires multi-source data from the data center, including cooling system operation data, environmental data, and data center cooling load data. The data processing module cleans and normalizes the multi-source data to obtain the target dataset; The first model training module trains the pre-built hybrid model using the target dataset to obtain the target hybrid model; the hybrid model includes a mathematical model and a neural network prediction model. The second model training model uses the target hybrid model as a simulation environment for interactive training of a pre-built deep reinforcement learning model to obtain the target deep reinforcement learning model; the deep reinforcement learning model is built based on the improved TD3 algorithm; wherein, the improved TD3 algorithm is designed with two independent branches in the action network of the original TD3 algorithm and an attention layer is added; The strategy output module interacts with the real environment through the target deep reinforcement learning model to output the optimal cooling system control signal; The strategy execution feedback module regulates the data center according to the cooling system control signal, and updates the target hybrid model and the target deep reinforcement learning model according to the data center operation data after regulation.

9. A computer-readable storage medium, characterized in that, It stores a computer program for energy consumption optimization of a data center cooling system based on deep reinforcement learning, wherein the computer program causes a computer to execute the energy consumption optimization method for a data center cooling system based on deep reinforcement learning as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing energy consumption optimization of data center cooling systems based on deep reinforcement learning as described in any one of claims 1 to 7.

Images