Machine room air conditioner intelligent scheduling method and system based on deep reinforcement learning

Abstract

This invention provides a method and system for intelligent scheduling of air conditioning in a data center based on deep reinforcement learning. The method includes: acquiring high-dimensional state feature vectors based on multimodal perception data of the data center; inputting the high-dimensional state feature vectors into an initial reinforcement learning model including an initial policy network and a value network to obtain predicted joint actions and predicted value scores; inputting the predicted joint actions into an interactive simulation environment to obtain a set of predicted data and completing a single round of iteration; minimizing the pruning and substitution objective function through the predicted data set under several single rounds of iteration to obtain the final reinforcement learning model, and then obtaining the air conditioning control actions. Through this approach, the scheduling decisions of multiple air conditioners can be globally and collaboratively optimized, reducing energy consumption; transforming post-event responses into pre-event predictions; and ensuring the stability and convergence of the reinforcement learning process.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for intelligent scheduling of computer room air conditioning based on deep reinforcement learning. Background Technology

[0002] With the rapid development of cloud computing, big data, and artificial intelligence technologies, data centers, as the core carriers of information infrastructure, are experiencing a continuous increase in scale and power density. In the total energy consumption structure of data centers, air conditioning and cooling systems account for as much as 30% to 40% of the energy consumption.

[0003] Air conditioning scheduling is a crucial aspect of ensuring the safe and stable operation of data centers. Its main function is to maintain the temperature in various areas of the computer room within the permissible operating range of IT equipment by adjusting operating parameters such as supply air temperature, fan speed, and compressor start / stop, preventing equipment downtime or shortened lifespan due to overheating. At the same time, reasonable air conditioning scheduling can effectively reduce the total power consumption of the cooling system, achieving energy conservation and cost reduction goals, thereby significantly lowering the operating costs of the data center.

[0004] Existing air conditioning scheduling methods generally have the following limitations: methods based on fixed thresholds or PID control are usually passive responses to excessive temperatures, unable to predict heat load changes in advance, resulting in drastic temperature fluctuations and uneven cooling effects; maintenance personnel often set lower target temperatures to ensure safety, causing air conditioners to operate at full load for extended periods, resulting in significant energy waste; currently, most methods adjust individual air conditioners independently, lacking coordination and linkage between multiple air conditioners, which can easily lead to problems such as airflow short-circuiting or mixing of hot and cold air, reducing overall cooling efficiency. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for intelligent scheduling of computer room air conditioning based on deep reinforcement learning, which aims to solve the technical problems of slow response, high energy consumption, and inability to achieve global optimization in existing technologies.

[0006] To achieve the above objectives, in a first aspect, embodiments of this application provide a method for intelligent scheduling of computer room air conditioning based on deep reinforcement learning, comprising the following steps: Acquire multimodal sensing data of the computer room, including initial temperature field data, initial airflow organization data and initial IT equipment load data, and preprocess the multimodal sensing data to obtain high-dimensional state feature vectors; An initial reinforcement learning model is constructed, which includes several agents corresponding to several air conditioners. The agents include an initial policy network and a value network. An interactive simulation environment is constructed, which includes a thermodynamic simulation engine, a load simulator and an air conditioner energy consumption model. The high-dimensional state feature vector is input into the initial reinforcement learning model to obtain the predicted joint action and predicted value score through the initial policy network and value network, respectively. The predicted joint action is input into the interactive simulation environment to obtain the predicted temperature field data, predicted airflow organization data, predicted IT equipment load data and predicted energy consumption data, and complete a single round of iteration. By minimizing the pruning and substitution objective function through several single-round iterations of predicted joint actions, predicted value scores, predicted temperature field data, predicted airflow organization data, predicted IT equipment load data, and predicted energy consumption data, the initial policy network is transformed into the final policy network, and the final reinforcement learning model is obtained. The air conditioning control actions are obtained through high-dimensional state feature vectors and the final reinforcement learning model.

[0007] Furthermore, the step of preprocessing the multimodal sensing data to obtain a high-dimensional state feature vector includes: Outlier removal and normalization are performed on the initial temperature field data, initial airflow organization data, and initial IT equipment load data to obtain the temperature field data, airflow organization data, and IT equipment load data to be used. Spatial feature extraction of the temperature field data to be used is performed through a convolutional neural network to obtain a temperature spatial feature map. Temporal features of the airflow organization data to be used are extracted using a gated cyclic network to obtain the airflow temporal feature vector, and temporal features of the load data of the IT equipment to be used are extracted using a gated cyclic network to obtain the load temporal feature vector. The temperature spatial feature map, airflow time series feature vector, and load time series feature vector are concatenated into a high-dimensional state feature vector.

[0008] Furthermore, the temperature field data to be used includes several temperature time series unit groups, each corresponding to several temperature sensors. The step of extracting spatial features from the temperature field data to be used through a convolutional neural network to obtain a temperature spatial feature map includes: Obtain the sensor coordinates of the temperature sensor in the computer room, construct a grid space through the physical boundary of the computer room, and construct several grid nodes within the grid space; The temperature time series unit group is mapped to the grid space based on the sensor coordinates to obtain the sensor nodes. The Euclidean distance between the grid node and the sensor node is obtained. The Euclidean distance is compared with the distance threshold to select a number of first candidate nodes corresponding to the grid node from a number of sensor nodes. The two-dimensional plan of the computer room is used to determine whether there is a physical barrier between the grid node and the first candidate node. Several second candidate nodes are selected from several first candidate nodes. Based on the temperature time series unit group corresponding to the second candidate node, the node temperature unit group corresponding to the grid node is obtained to form a temperature field matrix. The temperature field matrix is ​​input into a convolutional neural network to output a temperature space feature map after several convolutions and pooling.

[0009] Furthermore, the gated recurrent network includes an update gate and a reset gate. The step of extracting temporal features from the airflow organization data to be used through the gated recurrent network to obtain an airflow temporal feature vector includes: Input the airflow organization data to be used into the update gate and the reset gate respectively to obtain the retained features and forgotten features; Candidate hidden states are obtained based on the forgotten features, and airflow time-series feature vectors are obtained through the candidate hidden states and retained features.

[0010] Furthermore, the step of minimizing the trimmed alternative objective function through several single-round iterations of prediction joint action, prediction value score, prediction temperature field data, prediction airflow organization data, prediction IT equipment load data, and prediction energy consumption data includes: Based on the predicted temperature field data, predicted airflow organization data, and predicted IT equipment load data of each single iteration, the predicted state feature vector of the current single iteration is obtained, and the predicted state feature vector of the next single iteration is obtained through the predicted state feature vector of the current single iteration. Construct an empirical tuple based on the predicted state feature vector, predicted joint action, predicted value score, predicted energy consumption data, and the predicted state feature vector for the next single iteration under the current single iteration. Construct a pruning substitution objective function and perform at least three mini-batch gradient updates on the pruning substitution objective function using several empirical tuples to minimize the pruning substitution objective function.

[0011] Furthermore, the step of constructing the pruning alternative objective function includes: The instant reward for the current iteration is constructed using the predicted temperature field data and predicted energy consumption data for the next iteration. Several time-series difference errors are obtained by using the instant reward for the current iteration and the predicted value score for each iteration. The generalized dominance estimate for the current single iteration is constructed using several time-series difference errors; Based on the predicted state feature vector and the initial policy network under the current single iteration, obtain the probability ratio of the old and new policies under the current single iteration; The pruning substitution sub-function for the current single iteration is constructed by the probability ratio of the new and old strategies and the generalized advantage estimate for the current single iteration. The pruning substitution sub-function is then averaged to obtain the pruning substitution objective function.

[0012] Furthermore, the formula for obtaining the instant reward is as follows: , in, Indicates the first Instant rewards for each iteration Indicates the first Predicted energy consumption data for each single iteration Indicates the first In the predicted temperature field data of the first single iteration One temperature sub-data point, Indicates the temperature threshold. Indicates the first The total number of temperature sub-data in the predicted temperature field data of each single iteration. , All represent weighting coefficients; The formula for obtaining the timing difference error is: , in, Indicates the first Temporal difference error in a single iteration Indicates the first discount factor. Value network right The prediction value score output by the predicted state feature vector in each single iteration. Value network right The predictive value score is the output of the predicted state feature vector in a single iteration.

[0013] Furthermore, the formula for obtaining the generalized advantage estimate is as follows: , in, Indicates the first The generalized advantage estimate for each single iteration. Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration This represents the second discount factor. This represents the hyperparameters of the generalized dominance estimation. This represents the total number of iterations in a single round.

[0014] Furthermore, the formula for obtaining the trimming substitution sub-function is as follows: , in, The network parameters of the initial policy network are: The first time The trimming and substitution sub-function in a single iteration. Indicates the first Expected results from a single iteration Indicates the first The network parameters of the initial policy network in the second iteration are: The probability ratio of the old and new strategies at that time Indicates the first The generalized advantage estimate for each single iteration. This represents the clipping hyperparameter. This indicates taking the minimum value. This represents the clipping function.

[0015] Secondly, embodiments of this application provide a data center air conditioning intelligent scheduling system based on deep reinforcement learning, applied to the data center air conditioning intelligent scheduling method based on deep reinforcement learning as described in the first aspect above. The system includes: The acquisition module is used to acquire multimodal sensing data of the computer room, including initial temperature field data, initial airflow organization data and initial IT equipment load data. The multimodal sensing data is preprocessed to obtain a high-dimensional state feature vector. The construction module is used to build an initial reinforcement learning model, which includes several agents corresponding to several air conditioners. The agents include an initial policy network and a value network. An interactive simulation environment including a thermodynamic simulation engine, a load simulator and an air conditioner energy consumption model is constructed. The prediction module is used to input high-dimensional state feature vectors into the initial reinforcement learning model to obtain the predicted joint action and predicted value score through the initial policy network and value network respectively. The predicted joint action is input into the interactive simulation environment to obtain predicted temperature field data, predicted airflow organization data, predicted IT equipment load data and predicted energy consumption data, and complete a single round of iteration. The optimization module is used to minimize the pruning and substitution objective function through several single-round iterations of predicted joint actions, predicted value scores, predicted temperature field data, predicted airflow organization data, predicted IT equipment load data, and predicted energy consumption data, so as to transform the initial policy network into the final policy network and obtain the final reinforcement learning model. The air conditioning control actions are obtained through high-dimensional state feature vectors and the final reinforcement learning model.

[0016] Thirdly, embodiments of this application provide a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent scheduling method for computer room air conditioning based on deep reinforcement learning as described in the first aspect above.

[0017] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the intelligent scheduling method for computer room air conditioning based on deep reinforcement learning as described in the first aspect above.

[0018] Compared with existing technologies, the beneficial effects of this invention are as follows: By acquiring multimodal sensing data and preprocessing it to generate high-dimensional state feature vectors, the agent can simultaneously perceive the thermal environment state, airflow organization state, and IT equipment load state of the computer room, thereby providing a comprehensive and multi-dimensional environmental representation for subsequent decision-making and avoiding the information loss problem caused by single-dimensional perception; by constructing an initial reinforcement learning model and an interactive simulation environment, the organic combination of multi-agent collaborative decision-making and high-fidelity environmental simulation is achieved, enabling the scheduling decisions of multiple air conditioners to be globally and collaboratively optimized, ensuring cooling efficiency while globally... The control system avoids prolonged full-load operation of all air conditioners, reducing energy consumption. Through several single-round iterations, a closed-loop iteration of perception-decision-inference-feedback is achieved. The acquisition of predictive data enables the agent to proactively predict the trend of heat load changes, transforming post-event response into pre-event prediction. By constructing a pruning and substitution objective function and minimizing this function to transform the initial policy network into the final policy network, stable optimization of the policy network is achieved. The pruning and substitution objective function, by limiting the magnitude of policy updates, prevents training collapse caused by excessively large single update steps, ensuring the stability and convergence of the reinforcement learning process. Attached Figure Description

[0019] Figure 1 This is a flowchart of the intelligent scheduling method for computer room air conditioning based on deep reinforcement learning in the first embodiment of the present invention; Figure 2 This is a structural block diagram of the intelligent scheduling system for computer room air conditioning based on deep reinforcement learning in the second embodiment of the present invention; The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0020] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0021] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Please see Figure 1 The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning provided in the first embodiment of the present invention includes the following steps: S10: Acquire multimodal sensing data of the computer room, including initial temperature field data, initial airflow organization data and initial IT equipment load data, and preprocess the multimodal sensing data to obtain a high-dimensional state feature vector; Initial temperature field data was collected by several temperature sensors distributed at the top and bottom of the cabinet and in the corridor. Initial airflow organization data was collected by wind speed and direction sensors installed under the raised floor or on the ceiling. Initial IT equipment load data was obtained by accessing the server.

[0024] Step S10 includes: S110: Perform outlier removal and normalization on the initial temperature field data, initial airflow organization data, and initial IT equipment load data to obtain the waiting temperature field data, waiting airflow organization data, and waiting IT equipment load data. It should be noted that different acquisition processes can be controlled to sample data at the same sampling frequency and start time to achieve time alignment between different data. If the sampling frequencies are inconsistent, the data with the highest sampling frequency can be used as a benchmark, and interpolation can be performed on the data with lower sampling frequencies (e.g., by averaging the data adjacent to the missing sampling point and then filling in the missing sampling point) to achieve time alignment. In this embodiment, outlier removal is achieved through the 3σ criterion, which is widely used and will not be elaborated here. Furthermore, taking the initial temperature field data as an example, the maximum and minimum data are taken respectively, and then all data are normalized based on the maximum and minimum data. The initial airflow organization data and initial IT equipment load data are processed similarly and will not be elaborated here.

[0025] S120: Spatial feature extraction of the temperature field data to be used is performed through a convolutional neural network to obtain a temperature spatial feature map; Specifically, the temperature field data to be used includes several temperature time series unit groups, which can be understood to correspond to several temperature sensors respectively.

[0026] Obtain the sensor coordinates of the temperature sensor in the computer room, construct a grid space through the physical boundary of the computer room, and construct several grid nodes within the grid space; Understandably, a grid is formed by taking a corner of the computer room as the origin coordinate (0,0) and combining it with the physical boundary. This is the grid space. By adding horizontal and vertical lines at equal intervals, the intersection points are the grid nodes.

[0027] The temperature time series unit group is mapped to the grid space based on the sensor coordinates to obtain the sensor nodes. The Euclidean distance between the grid node and the sensor node is obtained. The Euclidean distance is compared with the distance threshold to select a number of first candidate nodes corresponding to the grid node from a number of sensor nodes. The distance threshold can be preset according to the thermal diffusion characteristics of the computer room, and sensor nodes with a Euclidean distance less than the distance threshold are selected as the first standby nodes.

[0028] The two-dimensional plan of the computer room is used to determine whether there is a physical barrier between the grid node and the first candidate node. Several second candidate nodes are selected from several first candidate nodes. Based on the temperature time series unit group corresponding to the second candidate node, the node temperature unit group corresponding to the grid node is obtained to form a temperature field matrix. Understandably, since a two-dimensional planar graph contains obstacles, by mapping the obstacles to the grid space using the same mapping rules, it can be determined whether there is a physical barrier between the grid node and the first candidate node. The first candidate node, which does not have a physical barrier, is selected as the second candidate node.

[0029] After obtaining the second node to be used, the Euclidean distance between the second node to be used and the mesh node is extracted. The initial fusion weight is determined based on the Euclidean distance. The formula for obtaining the initial fusion weight is as follows: , in, Indicates the first The initial fusion weights of the second node to be used. Indicates the first The Euclidean distance between the second node to be used and the grid node.

[0030] All initial fusion weights are normalized to obtain final fusion weights. The node temperature unit group is obtained by weighted fusion with the temperature time series unit group corresponding to the second candidate node and the final fusion weight.

[0031] The temperature field matrix is ​​input into a convolutional neural network to output a temperature space feature map after several convolutions and pooling. In this embodiment, after 5 convolutions, in the first 2 convolutions, through 3 Three convolutional kernels (16-32 channels) extract basic local features (such as edges and small-scale hotspots). In the last three convolutions, three... Three convolutional kernels (64-128 channels) extract abstract global features (such as large-scale temperature distribution and complex heat diffusion patterns). After several convolutions, a stage feature map is formed. After pooling, the stage feature map is downsampled to retain the "representative values" (such as maximum and average values) of local regions. This reduces the spatial dimension while preserving key features (such as hotspot locations and gradient directions) and enhances the model's translation invariance to form a temperature spatial feature map. By introducing physical obstacle judgment, unreasonable interpolation through obstacles (such as server racks and walls) is avoided, making the reconstructed temperature field matrix more consistent with the actual physical topology of the data center. Through weighted fusion based on Euclidean distance, temperature sensors closer to grid nodes contribute more weight, improving interpolation accuracy and providing more accurate spatial feature input.

[0032] S130: The time-series features of the airflow organization data to be used are extracted through a gated cyclic network to obtain the airflow time-series feature vector, and the time-series features of the load data of the IT equipment to be used are extracted through a gated cyclic network to obtain the load time-series feature vector. Specifically, the airflow organization data to be used is input into the update gate and the reset gate respectively to obtain the retained features and forgotten features; The formula for obtaining the preserved features is: , in, express Preservation characteristics of time, This represents the Sigmoid activation function. This indicates that the weight matrix should be retained. Indicates in The hidden state at time step 1 should be noted; initially, its value is 0. Indicating the airflow organization data to be used Sub-data of time, This indicates that the bias term is retained. The update gate controls how much information from the previous time step is carried over to the current time step. The method for obtaining the forgotten feature is the same as that for obtaining the retained feature, only the weight matrix and the bias term are different. The reset gate determines how much information from the previous time step is ignored.

[0033] Based on the forgotten features, candidate hidden states are obtained, and airflow time-series feature vectors are obtained through the candidate hidden states and retained features. The formula for obtaining the candidate hidden state is: , in, express The candidate hidden state at time step. Represents the hyperbolic tangent function. Represents the candidate weight matrix. express The characteristics of forgetting over time, This indicates element-wise multiplication. Indicates candidate bias terms; The formula for obtaining the airflow time-series feature vector is: , in, express The airflow time-series feature vector at any given time. The method for obtaining the load time-series feature vector is the same and will not be repeated here. It should be noted that in actual processing, the waiting airflow organization data and the waiting IT equipment load data can be directly concatenated into combined data in the time dimension and processed synchronously.

[0034] S140: Concatenate the temperature space feature map, airflow time series feature vector and load time series feature vector into a high-dimensional state feature vector; Specifically, a 2D CNN network is used to expand the temperature space feature map in the time dimension to obtain several time-series feature maps. The time-series feature maps are then averaged to obtain time-series feature vectors. Several time-series feature vectors are combined to form a temperature space feature vector. The temperature space feature vector, airflow time-series feature vector, and load time-series feature vector are then concatenated to form a high-dimensional state feature vector.

[0035] The standby temperature field data possesses spatial continuous field properties, with a definite temperature value existing at any spatial location within the computer room. Furthermore, the temperature values ​​at adjacent locations exhibit a smooth physical transition, and hot spots and cold / hot aisles form semantically informative topological structures in space. Therefore, by acquiring the temperature field matrix, a convolutional neural network is used to extract its spatial local correlation, gradient distribution, and hot spot morphology features, generating a temperature spatial feature map that retains spatial structural information. While the standby airflow organization data is physically a spatial vector field, actual acquisition is limited by the number of measurement points, exhibiting sparse sampling characteristics. Moreover, the key information in the standby airflow organization data lies in the temporal evolution patterns of supply and return air volumes and their dynamic matching relationship with heat load, rather than fine spatial texture features. The standby IT equipment load data reflects the computing power load at the equipment level; the key information lies in the collaborative change patterns of multiple equipment loads and their temporal dependencies. It does not itself constitute a spatial continuous field. Therefore, both are input as temporal sequences into a gated recurrent network to extract their temporal evolution patterns. Through differentiated processing, the physical attributes of multimodal perception data were adapted, providing a well-structured and information-complete high-dimensional state representation for subsequent feature fusion and deep reinforcement learning decision-making.

[0036] S20: Construct an initial reinforcement learning model, which includes several agents corresponding to several air conditioners. The agents include an initial policy network and a value network. Construct an interactive simulation environment including a thermodynamic simulation engine, a load simulator, and an air conditioner energy consumption model. A three-dimensional geometric model including server racks, air conditioners, and hot and cold aisles is constructed using CAD drawings of the computer room. Thermodynamic property parameters are assigned to the server racks, air conditioners, and hot and cold aisles to obtain an initial three-dimensional simulation model. Performance curve libraries of air conditioning compressors, fans, and chillers are then embedded into the initial three-dimensional simulation model to form the final three-dimensional simulation model. A thermodynamic simulation engine, a load simulator, and an air conditioning energy consumption model are then deployed in the final three-dimensional simulation model to form an interactive simulation environment.

[0037] The intelligent agent is a deep neural network implemented using PyTorch / TensorFlow. The thermodynamic simulation engine is a CFD solver (such as OpenFOAM, Fluent, 6SigmaDC) that solves the Navier-Stokes equations and energy equations. The load simulator is a time-series prediction model (such as LSTM, Transformer, Prophet). The air conditioning energy consumption model is a hybrid model driven by both mechanism and data, built based on equipment performance curves and historical energy consumption data. By constructing a three-dimensional geometric model based on CAD drawings and assigning thermodynamic property parameters, the simulation environment is highly consistent with the geometric structure and physical characteristics of the real computer room. By incorporating a performance curve library, the energy consumption model can accurately reflect the actual energy consumption characteristics of the equipment under different operating conditions. By deploying the thermodynamic simulation engine, load simulator, and air conditioning energy consumption model, a comprehensive simulation platform integrating physical deduction, load prediction, and energy consumption calculation is constructed, providing a high-fidelity interactive environment for reinforcement learning training.

[0038] S30: Input the high-dimensional state feature vector into the initial reinforcement learning model to obtain the predicted joint action and predicted value score through the initial policy network and value network respectively. Input the predicted joint action into the interactive simulation environment to obtain the predicted temperature field data, predicted airflow organization data, predicted IT equipment load data and predicted energy consumption data, and complete a single round of iteration. Upon receiving the high-dimensional state feature vector, the agent outputs a predicted joint action based on the initial policy network. This predicted joint action comprises several sub-actions, which are air conditioning temperature adjustment commands. The value network evaluates the corresponding predicted value score based on the input high-dimensional state feature vector. When the predicted joint action is input into the interactive simulation environment, the environment adjusts the temperature of each air conditioner accordingly. Then, it acquires predicted temperature field data and predicted airflow organization data through a thermodynamic simulation engine, predicted IT equipment load data through a load simulator, and predicted energy consumption data through an air conditioning energy consumption model. Understandably, after acquiring the predicted temperature field data, predicted airflow organization data, and predicted IT equipment load data, a predicted state feature vector can be formed using the same method described above. This predicted state feature vector is essentially the same as the high-dimensional state feature vector. The predicted state feature vector is then input back into the initial reinforcement learning model for another single-round iteration. It should be noted that after the predicted joint action is input into the interactive simulation environment, the thermodynamic simulation engine solves the Navier-Stokes equation and the energy equation based on the input temperature field data, airflow organization data, and IT equipment load data to complete the data extrapolation. The airflow organization data determines the spatial diffusion pattern of the temperature field, and the IT equipment load data determines the heat source intensity. Together with the predicted joint action, these two factors determine the temperature of each monitoring point in the next single iteration, and thus indirectly but decisively dominate the generation of the reward signal through the instant reward formula.

[0039] S40: Minimize the pruning and substitution objective function by predicting joint actions, predicting value scores, predicting temperature field data, predicting airflow organization data, predicting IT equipment load data and predicting energy consumption data under several single-round iterations, so as to transform the initial policy network into the final policy network and obtain the final reinforcement learning model. The air conditioning control action is obtained through the high-dimensional state feature vector and the final reinforcement learning model. Step S40 includes: S410: Based on the predicted temperature field data, predicted airflow organization data, and predicted IT equipment load data of each single iteration, obtain the predicted state feature vector of the current single iteration, and obtain the predicted state feature vector of the next single iteration through the predicted state feature vector of the current single iteration. Understandably, the predicted state feature vector under the current single iteration is input again into the initial reinforcement learning model, and the output of the initial reinforcement learning model is input again into the interactive simulation environment to obtain the predicted state feature vector under the next single iteration.

[0040] S420: Construct an empirical tuple based on the predicted state feature vector, predicted joint action, predicted value score, predicted energy consumption data, and the predicted state feature vector for the next single iteration under the current single iteration. Understandably, each iteration corresponds to a set of predicted joint actions, predicted value scores, predicted temperature field data, predicted airflow organization data, predicted IT equipment load data, and predicted energy consumption data. The data corresponding to each iteration is extracted into empirical tuples, forming several empirical tuples.

[0041] S430: Construct a pruning substitution objective function and perform at least three mini-batch gradient updates on the pruning substitution objective function using several empirical tuples to minimize the pruning substitution objective function; The instant reward for the current iteration is constructed using the predicted temperature field data and predicted energy consumption data for the next iteration. Several time-series difference errors are obtained by using the instant reward for the current iteration and the predicted value score for each iteration. The formula for obtaining the instant reward is: , in, Indicates the first Instant rewards for each iteration Indicates the first Predicted energy consumption data for each single iteration Indicates the first In the predicted temperature field data of the first single iteration One temperature sub-data point, This represents the temperature threshold, which is set according to the operating temperature requirements of the computer room. In this embodiment, its value is 27°C. Indicates the first The total number of temperature sub-data in the predicted temperature field data of each single iteration. , All represent weighting coefficients; The formula for obtaining the timing difference error is: , in, Indicates the first Temporal difference error in a single iteration Indicates the first discount factor. Value network right The prediction value score is the output of the predicted state feature vector in a single iteration. Value network right The predictive value score is the output of the predicted state feature vector in a single iteration.

[0042] The generalized dominance estimate for the current single iteration is constructed using several time-series difference errors; The formula for obtaining the generalized dominance estimate is as follows: , in, Indicates the first The generalized advantage estimate for each single iteration. Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration This represents the second discount factor. This represents the hyperparameters of the generalized dominance estimation. This represents the total number of iterations in a single round; Based on the predicted state feature vector and the initial policy network under the current single iteration, obtain the probability ratio of the old and new policies under the current single iteration; Understandably, during the first mini-batch gradient update, the initial policy network parameters are... Furthermore, the network parameters of the initial policy network were not pruned to replace the objective function update; therefore, the network parameters of the initial policy network during the first mini-batch gradient update... By inputting the predicted state feature vectors from several single-round iterations into the network parameters, and The initial policy network is used to obtain two output probabilities of the predicted joint action after several single-round iterations of the initial policy network output, and then combined with... The corresponding predicted output probability of the joint action divided by The corresponding predicted output probability of the joint action is used to obtain the probability ratio between the old and new strategies.

[0043] The pruning and substitution sub-functions for the current single iteration are constructed by the probability ratio of the new and old strategies and the generalized advantage estimate for the current single iteration. The pruning and substitution sub-functions are then averaged to obtain the pruning and substitution objective function. The formula for obtaining the trimming substitution sub-function is as follows: , in, The network parameters of the initial policy network are: No. The trimming and substitution sub-function in a single iteration. Indicates the first Expected results from a single iteration Indicates the first The network parameters of the initial policy network in the second iteration are: The probability ratio of the old and new strategies at that time Indicates the first The generalized advantage estimate for each single iteration. This represents the clipping hyperparameter. This indicates taking the minimum value. Let represent the pruning function. Understandably, assuming 5 single-round iterations, there are 5 pruning substitution sub-functions. These 5 sub-functions are averaged to obtain the pruning substitution objective function. Backpropagation is then performed using this objective function to optimize the network parameters of the initial policy network. Update the algorithm by performing one mini-batch gradient update, and then repeat the above steps to perform at least three mini-batch gradient updates. It should be noted that the target value of the value network is constructed based on the temporal difference error, and the network parameters of the value network are updated synchronously by minimizing the mean squared error loss between the value network output and the target value.

[0044] By combining temperature exceedance penalties with energy consumption penalties to construct an immediate reward, a balance between safety and economy is achieved. A generalized dominance estimation method is used to comprehensively consider multi-step temporal difference errors, improving the accuracy and stability of dominance estimation. The probability ratio between the old and new policies is used to measure the policy update magnitude, and a pruning mechanism is employed to limit the update step size, ensuring the stability and convergence of the policy optimization. Understandably, air conditioning control actions include several sub-control actions, which control the corresponding air conditioners to regulate temperature.

[0045] By acquiring multimodal perception data and preprocessing it to generate high-dimensional state feature vectors, the agent can simultaneously perceive the thermal environment, airflow organization, and IT equipment load status of the computer room. This provides a comprehensive and multi-dimensional environmental representation for subsequent decision-making, avoiding the information loss problem caused by single-dimensional perception. By constructing an initial reinforcement learning model and an interactive simulation environment, the organic combination of multi-agent collaborative decision-making and high-fidelity environmental simulation is achieved. This enables global collaborative optimization of the scheduling decisions of multiple air conditioners, ensuring cooling efficiency while avoiding prolonged full-load operation of all air conditioners and reducing energy consumption. Through several single-round iterations, a closed-loop iteration of perception-decision-inference-feedback is achieved. The acquisition of predictive data enables the agent to proactively predict the trend of heat load changes, transforming post-event response into pre-event prediction. By constructing a pruning and substitution objective function and minimizing this function to transform the initial policy network into the final policy network, stable optimization of the policy network is achieved. The pruning and substitution objective function, by limiting the magnitude of policy updates, prevents training collapse caused by excessively large single update steps, ensuring the stability and convergence of the reinforcement learning process.

[0046] Please see Figure 2 The second embodiment of the present invention provides a data center air conditioning intelligent scheduling system based on deep reinforcement learning. This system is applied to the data center air conditioning intelligent scheduling method based on deep reinforcement learning described in the above embodiments, and will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0047] The system includes: The acquisition module 10 is used to acquire multimodal sensing data of the computer room, including initial temperature field data, initial airflow organization data and initial IT equipment load data. The multimodal sensing data is preprocessed to obtain a high-dimensional state feature vector. The acquisition module 10 includes: The first unit is used to perform outlier removal and normalization processing on the initial temperature field data, initial airflow organization data and initial IT equipment load data to obtain the temperature field data, airflow organization data and IT equipment load data to be used. The second unit is used to extract spatial features from the temperature field data to be used through a convolutional neural network to obtain a temperature spatial feature map. The second unit is specifically used to obtain the sensor coordinates of the temperature sensor in the computer room, construct a grid space through the physical boundary of the computer room, and construct several grid nodes within the grid space; The temperature time series unit group is mapped to the grid space based on the sensor coordinates to obtain the sensor nodes. The Euclidean distance between the grid node and the sensor node is obtained. The Euclidean distance is compared with the distance threshold to select a number of first candidate nodes corresponding to the grid node from a number of sensor nodes. The two-dimensional plan of the computer room is used to determine whether there is a physical barrier between the grid node and the first candidate node. Several second candidate nodes are selected from several first candidate nodes. Based on the temperature time series unit group corresponding to the second candidate node, the node temperature unit group corresponding to the grid node is obtained to form a temperature field matrix. The temperature field matrix is ​​input into a convolutional neural network to output a temperature space feature map after several convolutions and pooling. The third unit is used to extract time-series features from the airflow organization data to be used through a gated cyclic network to obtain the airflow time-series feature vector, and to extract time-series features from the IT equipment load data to be used through a gated cyclic network to obtain the load time-series feature vector. The third unit is specifically used to input the airflow organization data to be used into the update gate and the reset gate respectively, so as to obtain the retained features and forgotten features. Based on the forgotten features, candidate hidden states are obtained, and airflow time-series feature vectors are obtained through the candidate hidden states and retained features. The fourth unit is used to concatenate the temperature spatial feature map, the airflow time series feature vector, and the load time series feature vector into a high-dimensional state feature vector; The construction module 20 is used to construct an initial reinforcement learning model, which includes several agents corresponding to several air conditioners. The agents include an initial policy network and a value network. An interactive simulation environment including a thermodynamic simulation engine, a load simulator and an air conditioner energy consumption model is constructed. The prediction module 30 is used to input the high-dimensional state feature vector into the initial reinforcement learning model, so as to obtain the prediction joint action and prediction value score through the initial policy network and value network respectively. The prediction joint action is input into the interactive simulation environment to obtain the prediction temperature field data, the prediction airflow organization data, the prediction IT equipment load data and the prediction energy consumption data, and complete a single round of iteration. The optimization module 40 is used to minimize the pruning and substitution objective function through several single-round iterations of predicted joint actions, predicted value scores, predicted temperature field data, predicted airflow organization data, predicted IT equipment load data and predicted energy consumption data, so as to transform the initial policy network into the final policy network and obtain the final reinforcement learning model. The air conditioning control actions are obtained through the high-dimensional state feature vector and the final reinforcement learning model. The optimization module 40 includes: The fifth unit is used to obtain the predicted state feature vector of the current single iteration based on the predicted temperature field data, predicted airflow organization data, and predicted IT equipment load data of each single iteration, and to obtain the predicted state feature vector of the next single iteration through the predicted state feature vector of the current single iteration. The sixth unit is used to construct empirical tuples based on the predicted state feature vector, predicted joint action, predicted value score, predicted energy consumption data, and predicted state feature vector under the current single iteration; The seventh unit is used to construct the pruning substitution objective function. The pruning substitution objective function is updated with at least three mini-batch gradients using several empirical tuples to minimize the pruning substitution objective function. The seventh unit is specifically used to construct the generalized advantage estimate under the current single-round iteration through several time-series difference errors; Based on the predicted state feature vector and the initial policy network under the current single iteration, obtain the probability ratio of the old and new policies under the current single iteration; The pruning substitution sub-function for the current single iteration is constructed by the probability ratio of the new and old strategies and the generalized advantage estimate for the current single iteration. The pruning substitution sub-function is then averaged to obtain the pruning substitution objective function.

[0048] The present invention also provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent scheduling method for computer room air conditioning based on deep reinforcement learning as described in the above technical solutions.

[0049] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the intelligent scheduling method for computer room air conditioning based on deep reinforcement learning as described in the above technical solution.

[0050] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0051] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for intelligent scheduling of computer room air conditioning based on deep reinforcement learning, characterized in that, Includes the following steps: Acquire multimodal sensing data of the computer room, including initial temperature field data, initial airflow organization data and initial IT equipment load data, and preprocess the multimodal sensing data to obtain high-dimensional state feature vectors; An initial reinforcement learning model is constructed, which includes several agents corresponding to several air conditioners. The agents include an initial policy network and a value network. An interactive simulation environment is constructed, which includes a thermodynamic simulation engine, a load simulator and an air conditioner energy consumption model. The high-dimensional state feature vector is input into the initial reinforcement learning model to obtain the predicted joint action and predicted value score through the initial policy network and value network, respectively. The predicted joint action is input into the interactive simulation environment to obtain the predicted temperature field data, predicted airflow organization data, predicted IT equipment load data and predicted energy consumption data, and complete a single round of iteration. By minimizing the pruning and substitution objective function through several single-round iterations of predicted joint actions, predicted value scores, predicted temperature field data, predicted airflow organization data, predicted IT equipment load data, and predicted energy consumption data, the initial policy network is transformed into the final policy network, and the final reinforcement learning model is obtained. The air conditioning control actions are obtained through high-dimensional state feature vectors and the final reinforcement learning model.

2. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 1, characterized in that, The step of preprocessing the multimodal sensing data to obtain a high-dimensional state feature vector includes: Outlier removal and normalization are performed on the initial temperature field data, initial airflow organization data, and initial IT equipment load data to obtain the temperature field data, airflow organization data, and IT equipment load data to be used. Spatial feature extraction of the temperature field data to be used is performed through a convolutional neural network to obtain a temperature spatial feature map. Temporal features of the airflow organization data to be used are extracted using a gated cyclic network to obtain the airflow temporal feature vector, and temporal features of the load data of the IT equipment to be used are extracted using a gated cyclic network to obtain the load temporal feature vector. The temperature spatial feature map, airflow time series feature vector, and load time series feature vector are concatenated into a high-dimensional state feature vector.

3. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 2, characterized in that, The temperature field data to be used includes several temperature time series unit groups, each corresponding to several temperature sensors. The step of extracting spatial features from the temperature field data to be used using a convolutional neural network to obtain a temperature spatial feature map includes: Obtain the sensor coordinates of the temperature sensor in the computer room, construct a grid space through the physical boundary of the computer room, and construct several grid nodes within the grid space; The temperature time series unit group is mapped to the grid space based on the sensor coordinates to obtain the sensor nodes. The Euclidean distance between the grid node and the sensor node is obtained. The Euclidean distance is compared with the distance threshold to select a number of first candidate nodes corresponding to the grid node from a number of sensor nodes. The two-dimensional plan of the computer room is used to determine whether there is a physical barrier between the grid node and the first candidate node. Several second candidate nodes are selected from several first candidate nodes. Based on the temperature time series unit group corresponding to the second candidate node, the node temperature unit group corresponding to the grid node is obtained to form a temperature field matrix. The temperature field matrix is ​​input into a convolutional neural network to output a temperature space feature map after several convolutions and pooling.

4. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 2, characterized in that, The gated recurrent network includes update gates and reset gates. The step of extracting time-series features from the airflow organization data to be used through the gated recurrent network to obtain airflow time-series feature vectors includes: Input the airflow organization data to be used into the update gate and the reset gate respectively to obtain the retained features and forgotten features; Candidate hidden states are obtained based on the forgotten features, and airflow time-series feature vectors are obtained through the candidate hidden states and retained features.

5. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 1, characterized in that, The steps of minimizing the tailored alternative objective function through several single-round iterations of prediction joint action, prediction value score, prediction temperature field data, prediction airflow organization data, prediction IT equipment load data, and prediction energy consumption data include: Based on the predicted temperature field data, predicted airflow organization data, and predicted IT equipment load data of each single iteration, the predicted state feature vector of the current single iteration is obtained, and the predicted state feature vector of the next single iteration is obtained through the predicted state feature vector of the current single iteration. Construct an empirical tuple based on the predicted state feature vector, predicted joint action, predicted value score, predicted energy consumption data, and the predicted state feature vector for the next single iteration under the current single iteration. Construct a pruning substitution objective function and perform at least three mini-batch gradient updates on the pruning substitution objective function using several empirical tuples to minimize the pruning substitution objective function.

6. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 5, characterized in that, The steps for constructing the trimming alternative objective function include: The instant reward for the current iteration is constructed using the predicted temperature field data and predicted energy consumption data for the next iteration. Several time-series difference errors are obtained by using the instant reward for the current iteration and the predicted value score for each iteration. The generalized dominance estimate for the current single iteration is constructed using several time-series difference errors; Based on the predicted state feature vector and the initial policy network under the current single iteration, obtain the probability ratio of the old and new policies under the current single iteration; The pruning substitution sub-function for the current single iteration is constructed by the probability ratio of the new and old strategies and the generalized advantage estimate for the current single iteration. The pruning substitution sub-function is then averaged to obtain the pruning substitution objective function.

7. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 6, characterized in that, The formula for obtaining the instant reward is: ， in, Indicates the first Instant rewards for each iteration Indicates the first Predicted energy consumption data for each single iteration Indicates the first In the predicted temperature field data of the first single iteration One temperature sub-data point, Indicates the temperature threshold. Indicates the first The total number of temperature sub-data in the predicted temperature field data of each single iteration. , All represent weighting coefficients; The formula for obtaining the timing difference error is: ， in, Indicates the first Temporal difference error in a single iteration Indicates the first discount factor. Value network right The prediction value score is the output of the predicted state feature vector in a single iteration. Value network right The predictive value score is the output of the predicted state feature vector in a single iteration.

8. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 6, characterized in that, The formula for obtaining the generalized dominance estimate is as follows: ， in, Indicates the first The generalized advantage estimate for each single iteration. Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration Indicates the first Temporal difference error in a single iteration This represents the second discount factor. This represents the hyperparameters of the generalized dominance estimation. This represents the total number of iterations in a single round.

9. The intelligent scheduling method for computer room air conditioning based on deep reinforcement learning according to claim 6, characterized in that, The formula for obtaining the trimming substitution sub-function is: ， in, The network parameters of the initial policy network are: The first time The trimming and substitution sub-function in a single iteration. Indicates the first Expected results from a single iteration Indicates the first The network parameters of the initial policy network in the second iteration are: The probability ratio of the old and new strategies at that time Indicates the first The generalized advantage estimate for each single iteration. This represents the clipping hyperparameter. This indicates taking the minimum value. This represents the clipping function.

10. A data center air conditioning intelligent scheduling system based on deep reinforcement learning, applied to the data center air conditioning intelligent scheduling method based on deep reinforcement learning as described in any one of claims 1 to 9, characterized in that, The system includes: The acquisition module is used to acquire multimodal sensing data of the computer room, including initial temperature field data, initial airflow organization data and initial IT equipment load data. The multimodal sensing data is preprocessed to obtain a high-dimensional state feature vector. The construction module is used to build an initial reinforcement learning model, which includes several agents corresponding to several air conditioners. The agents include an initial policy network and a value network. An interactive simulation environment including a thermodynamic simulation engine, a load simulator and an air conditioner energy consumption model is constructed. The prediction module is used to input high-dimensional state feature vectors into the initial reinforcement learning model to obtain the predicted joint action and predicted value score through the initial policy network and value network respectively. The predicted joint action is input into the interactive simulation environment to obtain predicted temperature field data, predicted airflow organization data, predicted IT equipment load data and predicted energy consumption data, and complete a single round of iteration. The optimization module is used to minimize the pruning and substitution objective function through several single-round iterations of predicted joint actions, predicted value scores, predicted temperature field data, predicted airflow organization data, predicted IT equipment load data, and predicted energy consumption data, so as to transform the initial policy network into the final policy network and obtain the final reinforcement learning model. The air conditioning control actions are obtained through high-dimensional state feature vectors and the final reinforcement learning model.

Images