A data center refrigeration system cooperative energy-saving control method

Abstract

The application discloses a kind of data center refrigeration system synergic energy-saving control methods.The application is introduced into the multi-path feature extraction network of fusion convolution local feature extraction, long short-term memory time series dependence capture and time attention mechanism screening, effectively overcome the control lag problem caused by refrigeration system thermal inertia, so that control strategy has the foresight ability of predication to the trend of heat load change.Parameterized reinforcement learning architecture is used to model the discrete adjustment of chilled water unit outlet water temperature and the continuous adjustment of terminal air supply and water pump frequency, avoiding the loss of control accuracy or dimension explosion problem caused by the forced conversion of mixed action space in traditional deep reinforcement learning algorithm.The joint decision of cold source equipment and air conditioning terminal equipment under the unified target is realized, which significantly reduces the energy consumption of refrigeration system and improves the overall energy efficiency of data center under the premise of ensuring the thermal safety of computer room.

Description

Technical Field

[0001] This invention relates to the technical field of energy-saving optimization control, and specifically to a method for coordinated energy-saving control of a data center cooling system. Background Technology

[0002] As data centers continue to expand, the energy consumption of cooling systems is becoming increasingly prominent. As a critical infrastructure ensuring the safe operation of IT equipment, cooling systems are responsible for continuously removing waste heat generated by server operation. Their energy consumption typically accounts for about one-third of the total energy consumption of a data center, making it the largest energy consumer besides IT loads. Related research indicates that for every 1 degree Celsius increase in the supply air temperature of a data center, cooling energy consumption can be reduced by approximately 2% to 4%, demonstrating that reducing cooling redundancy through refined control methods offers considerable energy-saving potential.

[0003] From the perspective of the controlled objects, data center cooling systems can be divided into two levels: the cold source end and the terminal end. The cold source end mainly includes chillers, chilled water pumps, and cooling towers, responsible for the preparation and distribution of cooling capacity; the terminal end mainly consists of server room air conditioners (CRAH), responsible for distributing cooling capacity to the server area. Traditional control strategies often focus on local optimization of a single link, such as adjusting only the terminal airflow or only adjusting the chilled water outlet temperature. However, due to the close physical coupling between the various devices in the entire cooling chain, changes in terminal airflow will affect the chilled water return temperature, thereby changing the cooling coefficient of performance (COP) of the chiller; the operating status of the cold source end will also constrain the adjustment capability of the terminal end. Therefore, isolated control is difficult to achieve optimal energy efficiency at the system level, and a joint optimization method that can coordinate the cold source end and the terminal end is urgently needed.

[0004] Furthermore, intelligent control of refrigeration systems faces two key constraints. First, control lag due to thermal inertia: There is typically a 10-20 minute physical delay between the generation of cooling capacity from the cold source and its transfer to the server area. If the control strategy relies solely on real-time room temperature feedback, it is difficult to predict changes in heat load, easily leading to adjustment lag, temperature fluctuations, and even localized overheating. Second, difficulty in handling mixed action spaces: The chilled water outlet temperature of the chiller unit is set as a discrete variable, while the pump frequency and CRAH air volume are continuous variables. Existing deep reinforcement learning algorithms are usually designed for single-type action spaces, making it difficult to directly handle scenarios where discrete and continuous actions coexist. Forcibly transforming the action space will result in a loss of control accuracy or the curse of dimensionality, failing to meet engineering application requirements.

[0005] In summary, among the relevant technologies, in the scenario of collaborative energy-saving control of data center cooling systems, there is a technical problem that cannot effectively balance thermal inertia hysteresis compensation and spatial collaborative optimization of discrete-continuous hybrid motion. Summary of the Invention

[0006] The technical problem this invention aims to solve is that, in related technologies, under the scenario of collaborative energy-saving control of data center cooling systems, there is a technical problem that cannot effectively balance thermal inertia lag compensation and spatial collaborative optimization of discrete-continuous hybrid motions. The purpose is to provide a collaborative energy-saving control method for data center cooling systems that solves the technical problem of not being able to effectively balance thermal inertia lag compensation and spatial collaborative optimization of discrete-continuous hybrid motions.

[0007] This invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides a method for coordinated energy-saving control of a data center cooling system, comprising:

[0009] Obtain a multivariate time series state vector including the current time and N past times; wherein the multivariate time series state vector includes an IT load sequence and a thermodynamic state sequence; wherein the thermodynamic state sequence includes a computer room temperature sequence, a chiller unit outlet water temperature sequence, and an outdoor ambient temperature sequence;

[0010] A high-order state feature vector is obtained by processing the multivariate time series state vector through a feature extraction network. The feature extraction network includes a convolutional path, a long short-term memory path with a time attention mechanism, and an autoregressive linear path. The convolutional path is used to extract local coupling features between variables in the state vector. The long short-term memory path and the time attention mechanism are used to capture long-term dependencies in state evolution and generate a context vector. The autoregressive linear path is used to perform a linear transformation on the state vector to extract linear trends. The high-order state feature vector is obtained by fusing the context vector and the output of the autoregressive linear path.

[0011] For each discrete action in the preset discrete action set, the continuous control parameters corresponding to the discrete action are generated by the parameter network based on the high-order state feature vector; wherein, the discrete action set includes three discrete actions corresponding to the decrease, maintenance and increase of the chiller unit outlet water temperature respectively; wherein, the continuous control parameters include the terminal air volume and the water pump frequency;

[0012] The expected return value of each discrete action and the corresponding continuous control parameter is determined by using a value network based on high-order state feature vectors, discrete actions, and the continuous control parameters corresponding to the discrete actions.

[0013] Select the discrete action corresponding to the maximum expected return value, and output the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume and water pump frequency.

[0014] In a second aspect, the present invention provides a data center cooling system collaborative energy-saving control device, comprising:

[0015] The acquisition module is used to acquire a multivariate time series state vector including the current time and N past times; wherein the multivariate time series state vector includes an IT load sequence and a thermodynamic state sequence; wherein the thermodynamic state sequence includes a computer room temperature sequence, a chiller unit outlet water temperature sequence, and an outdoor ambient temperature sequence.

[0016] An extraction module is used to process multivariate time series state vectors through a feature extraction network to obtain high-order state feature vectors. The feature extraction network includes a convolutional path, a long short-term memory path with a time attention mechanism, and an autoregressive linear path. The convolutional path extracts local coupling features between variables in the state vector. The long short-term memory path and the time attention mechanism capture long-term dependencies in state evolution and generate a context vector. The autoregressive linear path performs a linear transformation on the state vector to extract linear trends. The high-order state feature vector is obtained by fusing the context vector and the output of the autoregressive linear path.

[0017] The generation module is used to generate continuous control parameters corresponding to each discrete action in the preset discrete action set by using a parameter network based on a high-order state feature vector. The discrete action set includes three discrete actions corresponding to the decrease, maintenance, and increase of the chiller unit outlet water temperature, respectively. The continuous control parameters include the terminal air volume and the water pump frequency.

[0018] The determination module is used to determine the expected return value of each discrete action and the corresponding continuous control parameter of the hybrid action based on the high-order state feature vector, discrete action and the continuous control parameter corresponding to the discrete action through the value network.

[0019] The output module is used to select the discrete action corresponding to the maximum expected return value, and output the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume and water pump frequency.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to implement the method described above.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] This invention effectively overcomes the control lag problem caused by the thermal inertia of the cooling system by introducing a multi-path feature extraction network that integrates convolutional local feature extraction, long short-term memory temporal dependency capture, and temporal attention mechanism filtering. This enables the control strategy to have the ability to predict the trend of heat load changes. Simultaneously, a parametric reinforcement learning architecture is used to collaboratively model the discrete adjustment of the chiller outlet water temperature with the continuous adjustment of the terminal air volume and water pump frequency, avoiding the control accuracy loss or dimensionality explosion problems caused by the forced transformation of the mixed action space in traditional deep reinforcement learning algorithms. Furthermore, through the alternating optimization of the value network and the parameter network, joint decision-making between the cold source equipment and the air conditioning terminal equipment under a unified goal is achieved, significantly reducing the energy consumption of the cooling system and improving the overall energy utilization efficiency of the data center while ensuring the thermal safety of the data center. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0025] Figure 1 A flowchart illustrating a collaborative energy-saving control method for a data center cooling system provided in the embodiments of this specification;

[0026] Figure 2 This is a diagram of a collaborative energy-saving control architecture for a data center cooling system based on parametric reinforcement learning, provided in the embodiments of this specification.

[0027] Figure 3 The diagram shows the neural network architecture based on LSTNet-Attn and PDQN provided in the embodiments of this specification.

[0028] Figure 4 A block diagram of a data center cooling system collaborative energy-saving control device provided in the embodiments of this specification;

[0029] Figure 5 This is a block diagram of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0031] like Figure 1 As shown in the figure, this embodiment provides a collaborative energy-saving control method for a data center cooling system, including:

[0032] Step S11: Obtain a multivariate time series state vector including the current time and the past N times; wherein, the multivariate time series state vector includes an IT load sequence and a thermodynamic state sequence; wherein, the thermodynamic state sequence includes a computer room temperature sequence, a chiller unit outlet water temperature sequence, and an outdoor ambient temperature sequence.

[0033] like Figure 2 As shown, a ParaCool system architecture is provided, which consists of three core parts: a data center cooling system, an IT system, and a ParaCool reinforcement learning agent. The data center cooling system, as the controlled object of the algorithm, encompasses the entire process of cooling production, transportation, and distribution, and mainly consists of cold source equipment (chillers, chilled water pumps, and cooling towers) and air conditioning terminal equipment (CRAH). The IT system, mainly composed of server clusters, storage devices, and network equipment, is the main source of heat load for the system. The ParaCool reinforcement learning agent constructs a closed-loop interaction loop with the physical system. The controller first obtains the load power of the IT system and the thermodynamic state of the cooling system in real time through state perception; secondly, it uses LSTNet-Attn to extract temporal features and perform inference, collaboratively outputting joint control commands containing discrete and continuous actions for the current mixed action space; finally, it sends the commands to the cooling system for execution and continuously updates the strategy based on the energy consumption and temperature penalty signals fed back by the system.

[0034] In this embodiment, the multivariate time series state vector can be represented as a set of state data collected and arranged according to a preset historical time window at the same control decision moment.

[0035] Specifically, the executing entity can acquire the IT load status and cooling system thermal status of the data center at each control decision moment, and assemble the data corresponding to the current moment and the past N moments into a multivariate time series state vector in chronological order. The control decision moment corresponds to the control cycle of the cooling system. In a specific implementation, the control cycle can be a fixed time interval, and the executing entity can update the multivariate time series state vector once according to this fixed time interval.

[0036] In this embodiment, the IT load sequence can be a time series composed of the IT load power at the current moment and the past N moments. The IT load power is used to characterize the business load and thermal load generated by the data center server equipment at the corresponding moment. Specifically, the CPU utilization of each server at the corresponding moment can be obtained, and the operating power of each server can be determined by combining the idle power consumption and the full-load power consumption of the server. Then, the operating power of each server is summarized to obtain the data center IT load power at that moment.

[0037] In this embodiment, the thermodynamic state sequence can be used to characterize the time series of thermal state changes in the data center server room, cold source, and external environment.

[0038] In this embodiment, the data center temperature sequence may include data center temperature data from the current moment and the past N moments. Specifically, temperature data collected within the data center can be acquired, and the average temperature of the data center can be determined based on the collected data. The average temperature of the data center at each moment can then be arranged in chronological order to form a data center temperature sequence.

[0039] In this embodiment, the chiller unit outlet water temperature sequence may include chilled water outlet water temperature data of the chiller unit at the current time and at the past N times.

[0040] Specifically, chilled water outlet temperature data can be obtained from the chiller unit or refrigeration system controller, and the chilled water outlet temperature data of the current moment and the past N moments can be arranged in chronological order to form a chiller unit outlet temperature sequence.

[0041] In this embodiment, the outdoor ambient temperature sequence may include outdoor ambient temperature data for the current moment and the past N moments.

[0042] Specifically, when forming a multivariate time series state vector, the IT load sequence, the computer room temperature sequence, the chiller outlet water temperature sequence, and the outdoor ambient temperature sequence can be combined according to a preset variable order so that the IT load power, computer room temperature, chiller outlet water temperature, and outdoor ambient temperature at the same time have a corresponding relationship in the state vector.

[0043] In one possible and specific implementation scheme, to facilitate the explanation of the source of cooling energy consumption data in the multivariate time series state vector and subsequent reward value, the physical model of the data center cooling system is described below.

[0044] An IT system consists of a large number of servers, and its energy consumption is linearly related to CPU utilization. Assuming a data center contains M servers, what is the total IT load power at time t? Described as:

[0045] ;

[0046] In the formula, This represents the baseline power consumption of the server in idle state. This represents the peak power consumption of the server when it is fully loaded. Let be the CPU utilization of the i-th server at time t.

[0047] The cooling system is responsible for removing the heat generated by IT equipment, and its total energy consumption... It consists of the real-time power consumption of the electric chiller, water pump, cooling tower, and CRAH:

[0048] ;

[0049] The energy consumption of an electric chiller (EC) depends on the real-time cooling capacity. and cooling performance coefficient The model is described as follows:

[0050] ;

[0051] ;

[0052] In the formula, The EC real-time cooling capacity at time t; The coefficient of performance (COP) of EC is a dynamic function that is nonlinearly affected by various operating conditions such as ambient temperature, cooling water return temperature, and load rate. The specific heat capacity at constant pressure for chilled water; Let be the chilled water mass flow rate at time t; and These are the return water temperature and the outlet water temperature of the chilled water at time t, respectively.

[0053] In addition, to ensure the safety and stability of the chiller unit's operation, the chilled water outlet temperature is subject to strict physical constraints:

[0054] ;

[0055] in, This is the current actual outlet water temperature. The lower limit of the outlet water temperature, This is the upper limit of the outlet water temperature.

[0056] Cooling towers accelerate heat exchange between cooling water and air using fans. The power consumption of these fans follows the similarity law of fluid mechanics, meaning that power is proportional to the cube of the rotational speed. Its energy consumption model is described as follows:

[0057] ;

[0058] In the formula, This refers to the rated power of the cooling tower fan. This refers to the rated speed of the fan. Let t be the actual operating speed at time t. Let t be the actual power of the wind turbine at time t.

[0059] The energy consumption of a CRAH primarily stems from the power consumption of the fan driving the air circulation, a value directly dependent on the fan's real-time operating status. According to the similarity law of fluid machinery, the fan's shaft power is proportional to the cube of the air volume. Therefore, the energy consumption model of a CRAH fan and its physical constraints can be described as follows:

[0060] ;

[0061] ;

[0062] In the formula, Let be the energy consumption of CRAH at time t; This refers to the rated power of the CRAH fan. Rated air volume; This is the actual air supply volume at time t. This variable must be within the minimum allowable air volume of the equipment. and maximum air volume Adjustments are made between these parameters to meet the heat dissipation requirements of the computer room.

[0063] The energy consumption of water pumps in a refrigeration system mainly consists of chilled water pumps and cooling water pumps, and its value can be directly estimated based on the real-time operating frequency of the pumps. Based on the similarity law between pumps and fans in fluid machinery, the shaft power of a centrifugal pump is proportional to the cube of its frequency. Therefore, the real-time energy consumption model and operating frequency constraints of the water pumps can be described as follows:

[0064] ;

[0065] ;

[0066] In the formula, Let t be the total energy consumption of the water pump at time t; The number of water pumps that are turned on and running; Let j be the rated power of the j-th water pump; Let j be the actual operating frequency of the j-th water pump at time t; This refers to the rated frequency of the water pump. During actual operation, the pump frequency must be controlled within the minimum permissible range. and maximum frequency Between these points, the pipeline flow requirements are met while ensuring equipment safety.

[0067] Average room temperature is a core indicator for measuring the thermal safety of a data center, and it is affected by both cooling capacity supply and IT load fluctuations. Due to the significant specific heat capacity of the air and equipment inside the data center, temperature evolution exhibits obvious thermal inertia, resulting in a physical lag of several minutes in the impact of control actions on the environment.

[0068] To accurately describe this dynamic physical process and enable the agent to learn forward-looking control strategies, this implementation plan constructs a state evolution model of the computer room temperature based on the thermodynamic law of conservation of energy. (Computer room temperature) The discretized time-series update formula is described as follows:

[0069] ;

[0070] In the formula, and The average temperatures of the data center server room at time t+1 and time t are respectively. air density; The specific heat capacity of air at constant pressure; The effective air volume of the computer room control area; Let be the IT load energy consumption at time t; The effective cooling capacity output by the refrigeration system is determined by the CRAH air volume and the chilled water supply temperature. Let t be the environmental thermal disturbance term at time t.

[0071] In one possible and specific implementation scheme, based on the above-mentioned system energy consumption model and thermal dynamics model, this implementation scheme further determines the energy efficiency indicators, safety penalty items and optimization objectives used to evaluate the results of coordinated control.

[0072] Power Usage Effectiveness (PUE) is an internationally recognized standard for measuring the energy efficiency of data centers. It is defined as the ratio of total data center energy consumption to the energy consumption of IT equipment. The closer this value is to 1.0, the lower the energy consumption for auxiliary facilities such as cooling, and the more energy-efficient the system. Based on the aforementioned component energy consumption model, the average PUE of the system over its operating cycle T is defined as follows:

[0073] ;

[0074] in, This refers to the efficiency of electricity use. The total energy consumption of the cooling system is the sum of electrical energy consumed by the data center cooling system within the operating cycle T. The total energy consumption of IT equipment is the sum of electrical energy consumed by IT equipment within an operating cycle T.

[0075] To improve the energy efficiency and reduce the energy consumption of data center cooling systems, this implementation plan proposes the ParaCool method for joint optimization of the cold source equipment and the air conditioning terminal system, specifically by optimizing the chilled water outlet temperature setpoint. CRAH air volume and chilled water pump operating frequency Coordinated regulation is implemented. The stability of the data center temperature is crucial for ensuring reliable server operation. To transform the rigid temperature constraint into a continuous feedback signal perceptible to the reinforcement learning agent, this implementation introduces a temperature violation penalty mechanism based on the Softplus function. This mechanism is a smooth soft constraint; the penalty value approaches zero when the temperature is below a safe threshold, and increases sharply as the temperature approaches or exceeds the threshold, thereby guiding the agent to actively avoid high-temperature risks during training. The data center temperature penalty model is described as follows:

[0076]

[0077] In the formula, The temperature violation penalty value at time t; Let t be the temperature of the computer room at time t. This is a preset temperature safety threshold. This is the Softplus activation function.

[0078] Based on the aforementioned temperature penalty term, the ParaCool algorithm aims to minimize the weighted sum of the cumulative PUE index and the temperature penalty within the decision period T. The objective function is formalized as follows:

[0079] ;

[0080] In the formula, To optimize the objective and decision variables; Set the water supply temperature for the cooling tower; This refers to the air volume supplied by the air conditioning terminal. This refers to the operating frequency of the chilled water pump. and The weighting coefficients are used to balance the energy-saving benefits and the temperature control risks.

[0081] In one possible and specific implementation scheme, based on the above optimization objectives, this implementation scheme transforms the collaborative control process of the data center cooling system into a reinforcement learning decision-making process, and defines the state space, action space and reward function.

[0082] Traditional rule-based or model-based control methods struggle to balance energy efficiency and safety in dynamic environments due to the highly nonlinear, significant thermal inertia, and end-to-end physical coupling characteristics of data center cooling systems. To address this, this implementation proposes ParaCool, an intelligent energy-saving optimization algorithm based on parametric reinforcement learning. This algorithm dynamically generates the optimal cooling strategy while ensuring the thermal safety of the data center through end-to-end autonomous learning. To enable interaction between the agent and the physical environment, the optimization problem is first transformed into a standard Markov decision process.

[0083] During the training and inference process of the ParaCool algorithm, the agent interacts periodically with the data center environment. This interaction process is modeled as a standard quintuple. ,in Represents the state space. Represents the action space, Let be the state transition probability. For the reward function, This is the discount factor.

[0084] state space The environmental characteristics that the agent can observe are defined. Considering the significant thermal inertia and control lag in data center cooling systems, it is difficult to accurately infer the system's evolution trend based solely on current observations. Therefore, ParaCool models the state space as a state vector containing the current moment and historical observation information from the past N moments. The system's state vector at time t is defined as:

[0085] ;

[0086] In the formula, This is the time-series state vector of IT load. Taking IT load as an example, to capture the fluctuation trend of the load, this vector contains historical data from the current time t to the past N steps, and its mathematical form is described as follows:

[0087] ;

[0088] In the formula, This represents the system's thermal state vector, used to describe the system's thermal response and its state information under the influence of external disturbances. It consists of time series data of the average room temperature, chiller outlet water temperature, and outdoor ambient temperature. These variables also take N historical values ​​to reflect the system's thermal response delay and environmental change patterns. Defined as:

[0089] ;

[0090] in, , , These represent time series vectors for the average temperature of the computer room, the chilled water outlet temperature, and the outdoor ambient temperature, respectively.

[0091] ParaCool's control range covers both the cold source and terminal sides of the refrigeration system, enabling end-to-end collaborative optimization of the chiller, water pump, and CRAH. Considering the differences in control characteristics among various devices in the refrigeration chain, the chiller outlet water temperature is typically adjusted discretely, while the water pump operating frequency and CRAH air volume are typically adjusted continuously. Therefore, this implementation scheme uses a parameterized motion space for modeling.

[0092] ParaCool defines action vectors as... .in, For discrete actions, For discrete actions Associated continuous action parameters. By introducing a parameterized structure, the discrete-continuous hybrid action space of the data center cooling system is defined. It can be defined as:

[0093] ;

[0094] In the formula, the discrete action set Defined as These correspond to three adjustment modes: decreasing, keeping constant, and increasing the set temperature of the chilled water outlet of the chiller unit. For discrete actions The relevant continuous motion vector is determined by the CRAH air volume. Operating frequency of chilled water pump Together they form a continuous action space Defined as:

[0095] ;

[0096] reward function This is the core signal guiding the agent's learning strategy, reflecting the trade-off between energy efficiency optimization and safety assurance in the system. To transform the constrained optimization problem into a solvable reward maximization problem within a reinforcement learning framework, this implementation scheme designs a composite reward function that balances energy efficiency optimization and safety constraints. The reward function consists of a basic reward term, an energy efficiency index term, and a temperature-related penalty term, with the following mathematical definition:

[0097] ;

[0098] In the formula, It is a large positive constant to ensure that the reward value remains positive in most cases; and These are the weighting coefficients for energy efficiency targets and safety targets, respectively. The set temperature safety threshold; This represents the penalty value when the system violates physical constraints.

[0099] State transition probability Describes the current state of the data center Next action Then transition to the next state The probability distribution is determined by the physical thermodynamic properties of the data center cooling system. In this implementation, the state transition process is essentially determined by the physical thermodynamic properties of the data center cooling system. Because ParaCool uses a model-free DRL framework, the agent does not need to obtain the precise expression of the state transition probability in advance, but learns the dynamic law of system state evolution step by step through continuous interaction and sampling with the simulation environment.

[0100] Discount factor satisfy This is used to weigh the relative importance of immediate rewards and long-term returns. Considering the significant thermal inertia of data center cooling systems, this algorithm sets a relatively high [value / value]. This enhances the agent's focus on future cumulative gains, enabling it to learn forward-looking control strategies such as pre-cooling and effectively alleviate the hysteresis problem caused by thermal inertia.

[0101] Step S12: Process the multivariate time series state vector through a feature extraction network to obtain a high-order state feature vector; wherein, the feature extraction network includes a convolutional path, a long short-term memory path with a time attention mechanism, and an autoregressive linear path; wherein, the convolutional path is used to extract local coupling features between variables in the state vector; wherein, the long short-term memory path and the time attention mechanism are used to capture the long-term dependencies of state evolution and generate a context vector; wherein, the autoregressive linear path is used to perform a linear transformation on the state vector to extract the linear change trend; wherein, the high-order state feature vector is obtained by fusing the context vector and the output of the autoregressive linear path.

[0102] Step S13: For each discrete action in the preset discrete action set, the continuous control parameters corresponding to the discrete action are generated by the parameter network based on the high-order state feature vector; wherein, the discrete action set includes three discrete actions corresponding to the decrease, maintenance and increase of the chiller unit outlet water temperature respectively; wherein, the continuous control parameters include the terminal air supply volume and the water pump frequency.

[0103] Step S14: Based on the high-order state feature vector, discrete action and the corresponding continuous control parameter of the discrete action, determine the expected return value of each discrete action and the corresponding continuous control parameter of the hybrid action through the value network.

[0104] Step S15: Select the discrete action corresponding to the maximum expected return value, and output the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume and water pump frequency.

[0105] In some implementations, the step of processing the multivariate time series state vector through a feature extraction network to obtain a higher-order state feature vector includes:

[0106] Step S121: Input the multivariate time series state vector into the convolution path, and extract the local coupling features between variables by using a convolution kernel that slides along the time axis to obtain the convolution feature matrix.

[0107] In this embodiment, the multivariate time series state vector can be organized in matrix form. The rows of the matrix correspond to the types of state variables, namely, IT load power, average room temperature, chiller outlet water temperature, and outdoor ambient temperature. The columns of the matrix correspond to the time slots on the time axis, tracing back from the current time t to the Nth time in the past, including a total of N+1 time slots.

[0108] Specifically, the convolutional path receives the matrix at the network input and processes it using a sliding convolution along the time axis. Sliding along the time axis means the convolution kernel moves window by window along the column direction of the matrix, i.e., the time sequence direction, rather than moving along the variable direction. This ensures that each convolution operation covers a local time window of all four state variables across multiple consecutive time slots. This convolutional method can capture the coupling patterns of different variables within the same time period, such as the local correlation between a continuous increase in IT load and a lagged increase in data center temperature over several consecutive time slots.

[0109] In one possible and specific implementation, the convolutional path can be configured with a preset number of convolutional kernels, each kernel's size being the number of state variable types multiplied by the convolutional window size. The convolutional window size represents the number of consecutive time slots covered by each convolutional operation. For a single convolutional kernel, the specific computation process can include: at a certain time step, multiplying the kernel's weight matrix element-wise with the corresponding slice submatrix within the time window of the input matrix, summing the results, adding a bias term, and then processing the result using a ReLU activation function to output a scalar value. The ReLU activation function can be defined as: outputting the input value directly when the input is greater than zero, and outputting zero when the input is less than or equal to zero. The convolutional kernel starts from the beginning of the time series and slides along the time axis with a stride of 1, outputting one scalar value for each step, until all time windows have been traversed. The outputs of a single convolutional kernel at all time steps are concatenated to form a 1-dimensional feature vector. The outputs of all convolutional kernels are stacked along the variable dimension to form a convolutional feature matrix.

[0110] In one possible and specific implementation, the convolutional window size is set to 3, meaning each convolutional kernel covers a local time window of 3 consecutive time slots. Since the decision time slot length is 10 minutes, the 3 time slots correspond to a 30-minute time span, which can capture relatively close short-term dependencies between adjacent time slots. The activation function used is the ReLU function, which can alleviate the gradient vanishing problem during deep network training while maintaining positive activation.

[0111] Step S122: Input the convolutional feature matrix into the long short-term memory pathway with a time attention mechanism, process it through the long short-term memory network to obtain the hidden state at each historical moment, calculate the correlation score between the hidden state at each historical moment and the hidden state at the last moment through the time attention mechanism, and perform a weighted summation of the hidden states at each historical moment based on the correlation score to generate a context vector.

[0112] In this embodiment, the Long Short-Term Memory (LSTM) pathway receives a convolutional feature matrix as the input sequence. Each column of this matrix corresponds to a feature vector at one time step. The LSM unit processes these feature vectors sequentially along the time direction, outputting a hidden state vector at each time step. Internally, the LSM unit selectively retains or forgets information from historical moments through the coordinated operation of forget gates, input gates, and output gates, thereby alleviating the gradient vanishing problem that occurs in traditional recurrent neural networks when processing long sequences. Specifically, the number of hidden units in the LSM network can be a preset value. At each time step, the LSM unit calculates the current hidden state based on the current input and the hidden state of the previous time step. After processing the sequence of all time steps, a hidden state sequence is obtained. This sequence can be composed of the hidden state vectors of each time step arranged chronologically, with each hidden state vector encoding the cumulative temporal information from the beginning of the sequence to the current time step. More specifically, the number of hidden units in the LSM network is set to 12, maintaining the same scale as the number of historical time slots, ensuring that each hidden state vector has sufficient expressive power to encode the evolutionary information in the time series.

[0113] In this embodiment, based on the hidden state sequence output by the Long Short-Term Memory Network, the temporal attention mechanism can perform correlation analysis between the hidden state at the last moment and the hidden states at all historical moments, and calculate the importance of each historical moment to the current decision.

[0114] Specifically, the computational process of the attention mechanism may include:

[0115] First, the correlation score between the hidden state at the last moment and the hidden states at each historical moment is calculated. More specifically, the correlation score can be calculated using a scaled dot product attention function, which involves taking the dot product of the hidden state vector at the last moment and the hidden state vector at a certain historical moment, and then dividing by a scaling factor. This scaling factor is the square root of the number of hidden units, used to prevent the dot product from being too large, causing the gradient of the subsequent softmax function to approach zero.

[0116] Then, the relevance scores corresponding to all historical moments are input into the softmax function for normalization to obtain the attention weights corresponding to each historical moment. After softmax normalization, the sum of all weights equals 1, and each weight value reflects the relative importance of the corresponding historical moment to the final decision.

[0117] Finally, using attention weights as weighting coefficients, the hidden state vectors of all historical moments are summed to obtain the context vector. This context vector has the same dimension as the number of hidden units in the Long Short-Term Memory network, condensing key information highly relevant to the current decision throughout the time series. It highlights moments that have a decisive impact on control strategy selection, such as sudden changes in IT load or inflection points in temperature changes, while suppressing interference from noisy or stable periods.

[0118] In this embodiment, the temporal attention mechanism can adopt a multi-head attention approach, that is, multiple independent attention calculations are performed in parallel, each with its own learnable projection parameters, focusing on various key patterns in the sequence from different perspectives. The context vectors calculated by each group are concatenated or averaged at the output to fuse multi-faceted temporal features.

[0119] Step S123: Input the multivariate time series state vector into the autoregressive linear path, extract the linear trend through linear transformation, and obtain the autoregressive output vector.

[0120] In this embodiment, the autoregressive linear pathway can perform a linear weighted summation of the data from the most recent preset number of time slots in the input state sequence to extract the overall linear trend of the data.

[0121] Specifically, the linear weighted summation calculation process may include: for each time slot position within the autoregressive window, multiplying the state input vector of that time slot by the corresponding autoregressive weight coefficient matrix, summing the product results at all window positions, and then adding a bias term to obtain the autoregressive output vector. In this embodiment, the autoregressive window size is 12, consistent with the number of historical time slots, ensuring that the autoregressive path can cover the same historical range as the main path.

[0122] Understandably, while the convolutional-long short-term memory-attention pathway excels at capturing non-linear temporal dependencies, the non-linear nature of deep neural networks may make them less sensitive to simple, stable linear trends in input data. For example, when IT load continuously increases at an approximately constant rate over a period of time, such linear trends are more suitable for direct capture by a linear model. The autoregressive linear pathway, without requiring a non-linear activation function, can directly transmit linear components to the output with a shorter signal path, thus compensating for the shortcomings of deep neural networks in this regard and improving the overall model's accuracy in representing linear patterns in time-series data.

[0123] Step S124: Fuse the context vector and the autoregressive output vector to obtain the higher-order state feature vector.

[0124] In this embodiment, the fusion operation specifically employs vector concatenation, whereby the context vector and the autoregressive output vector are joined end-to-end along their respective vector dimensions, merging them into a single vector with a larger dimension. This merged vector is the higher-order state feature vector. The context vector originates from the output of the Long Short-Term Memory-Attention pathway, encoding the nonlinear long-term dependency features in the multivariate time series after attention-weighted filtering. The autoregressive output vector originates from the output of the autoregressive linear pathway, encoding the linear trend features in the original state sequence. Through this concatenation operation, the two heterogeneous time-series features are merged into a unified higher-order state feature vector. This vector simultaneously contains both nonlinear evolution patterns and linear trend information, providing a more complete input representation for the parameter network and value network in subsequent hybrid decision-making steps.

[0125] In one possible and specific implementation scheme, to address the issues of high-dimensional state space and time-delayed thermal response in data center cooling systems, and to improve policy learning efficiency in a hybrid action space, this implementation scheme constructs an end-to-end deep neural network architecture. For example... Figure 3 As shown, the architecture consists of cascaded PDQN and LSTNet-Attn, realizing a nonlinear mapping from multi-order states to mixed control commands.

[0126] Regarding the state space For the multivariate time series data contained therein, this implementation scheme introduces an improved LSTNet-Attn network. This module integrates convolutional neural networks, long short-term memory networks, attention mechanisms, and linear autoregressive pathways to capture local short-term dependencies, long-term evolution trends, key moment features, and linear change trends of the data, respectively.

[0127] Convolutional layers are located at the network input and are used to extract local coupling features between time-series variables. Let the input state sequence matrix be... ,in For the state variable dimension, This is the historical time step. The network uses... Each convolutional kernel performs sliding convolution along the time axis. For the k-th convolutional kernel... ,in The size of the convolution window is its output at time step t. Defined as:

[0128] ;

[0129] In the formula, * represents dot product and convolution operations; This represents a slice matrix of the input matrix within the time window [t, t+w-1]. This is the bias term. To enhance nonlinear expressiveness and alleviate the gradient vanishing problem, this layer uses ReLU as the activation function. .

[0130] Furthermore, by stacking the outputs of each convolutional kernel at each time step, the convolutional feature matrix can be obtained. And use it as the input for subsequent LSTM layers.

[0131] Considering the significant time delay in the thermal response of the refrigeration system, this implementation scheme introduces an LSTM layer to capture the long-term dependence of the time series. Let the hidden state of the LSTM at time t be... To enhance the model's ability to filter key historical information, a temporal attention mechanism is introduced at the LSTM output. This mechanism dynamically generates attention weights by calculating the correlation score between the current hidden state and historical hidden states. :

[0132] ;

[0133] In the formula, This represents the attention score function; This indicates the hidden state at the last moment.

[0134] By summing the weighted hidden states at each time step, we can obtain the context vector. This vector can highlight historical moments that have a significant impact on current decisions.

[0135] To address the issue of deep neural networks' inability to capture the linear trends of input data, this implementation retains the linear autoregressive path of LSTNet in the network structure. This path directly processes the input state sequence. A linear transformation is performed to extract the overall linear trend of the data. The output vector of the autoregressive pathway. Defined as:

[0136] ;

[0137] In the formula, q represents the autoregressive weight coefficients; q represents the autoregressive window size. For bias terms; Let Ni represent the state input vector at time Ni.

[0138] Finally, the context vector output by the attention mechanism is... Output of the autoregressive pathway The features are concatenated to form the final high-order state feature vector. This information is then used as input for subsequent decision-making modules.

[0139] The decision-making module of this implementation adopts the PDQN architecture to handle the hybrid action space. This module consists of two cooperating sub-networks: the continuous action parameter network (xNet) and the hybrid action value network (QNet).

[0140] xNet is responsible for generating corresponding continuous control parameters for each candidate discrete action. This network uses the high-order state feature vector output by the temporal feature extraction module. For each discrete action, the input is... Output the corresponding continuous parameter vector :

[0141] ;

[0142] in, Corresponding CRAH air volume With chilled water pump frequency The combined continuous control quantity.

[0143] QNet is responsible for evaluating the value of combinations of discrete actions and continuous control parameters. Specifically, it evaluates the high-order state feature vectors. With xNet As a combined input, the expected reward Q value corresponding to the mixed action is output:

[0144] ;

[0145] During the decision-making phase, the agent selects the discrete action with the highest Q value based on the evaluation results of QNet. and execute the corresponding continuous actions. This enables the coordinated optimization of discrete action selection and continuous parameter adjustment.

[0146] In some implementations, the step of generating continuous control parameters corresponding to each discrete action in a preset set of discrete actions using a parameter network based on a high-order state feature vector includes:

[0147] Step S131: Concatenate the high-order state feature vector with the discrete action identifier to obtain the joint input vector of the parameter network.

[0148] In this embodiment, the higher-order state feature vector is a fixed-dimensional vector that encodes the nonlinear long-term dependence and linear trend characteristics of the multivariate time series under the current decision time slot. This higher-order state feature vector itself does not include information about discrete actions; therefore, discrete action identifiers need to be introduced as additional input information so that the parameter network can generate differentiated continuous control parameters for different discrete actions.

[0149] Specifically, the discrete action identifier is used to distinguish three discrete actions in a preset set of discrete actions, namely, a decrease in chilled water outlet temperature, a constant chilled water outlet temperature, and an increase in chilled water outlet temperature. The specific encoding method for the discrete action identifier can be one-hot encoding, which uses a single three-dimensional vector to represent the three discrete actions. The decrease corresponds to a 1 in the first bit and 0 in the remaining bits, the constant temperature corresponds to a 1 in the second bit and 0 in the remaining bits, and the increase corresponds to a 1 in the third bit and 0 in the remaining bits. In other implementations, the discrete action identifier can also use integer index encoding, that is, using the three integers 0, 1, and 2 to identify the three discrete actions respectively. In this case, the parameter network internally maps the integer indices to dense vectors through an embedding layer before concatenating them with higher-order state feature vectors.

[0150] In one possible and specific implementation, the concatenation operation may include: concatenating the higher-order state feature vector and the discrete action identifier vector end-to-end along the vector dimension to form a joint input vector with a larger dimension. This joint input vector carries both a higher-order abstract representation of the environmental state and information about the currently considered discrete action type, providing complete input conditions for the subsequent nonlinear mapping of the fully connected network.

[0151] In this embodiment, the parameter network performs forward inference once for each of the three discrete actions, that is, it concatenates the decrementing label, the unchanged label, and the enlarging label with the same higher-order state feature vector to obtain three different joint input vectors of the parameter network, which are then sent to step S132 for processing in sequence. The three forward inferences share the same set of network parameters, and only the different discrete action labels produce different output results.

[0152] Step S132: Input the parameter network and the input vector into the fully connected network for nonlinear mapping, and output the continuous control parameters corresponding to the discrete action.

[0153] In this embodiment, the parameter network adopts a fully connected network structure, consisting of an input layer, several hidden layers, and an output layer stacked together. Each layer of the fully connected network includes a predetermined number of neurons, and neurons in adjacent layers are connected pairwise, with each connection corresponding to one learnable weight parameter.

[0154] Specifically, for a particular neuron in the hidden layer, the computation process may include: multiplying the output values ​​of all neurons in the previous layer by their corresponding connection weights and summing the results; adding the bias term of the neuron; and then processing the sum using a nonlinear activation function to obtain the output value of that neuron. The nonlinear activation function endows the network with the ability to express nonlinear mapping relationships, enabling the parametric network to learn complex correspondences between high-order state feature vectors and continuous control parameters.

[0155] In one possible and specific implementation, the parameter network employs a two-layer hidden layer structure. The first hidden layer contains 128 neurons, and the second hidden layer contains 64 neurons. All hidden layers are fully connected, as are the hidden layers with the input and output layers. The activation function is a hyperbolic tangent function, which maps the input to an output range between -1 and -1. Its output is symmetrically distributed around the value of 0, which helps maintain the numerical stability of the outputs of each layer during deep network training. The output layer contains two neurons, corresponding to the terminal airflow rate and pump frequency in the continuous control parameters, respectively. No activation function is applied to the output layer; it directly outputs two scalar values ​​linearly to ensure that the values ​​of the continuous control parameters are not limited by the saturation range of the activation function.

[0156] In one possible and specific implementation, the parameter network outputs a set of continuous control parameters for each of the three discrete actions, for a total of three sets. Each set of continuous control parameters includes two scalar values:

[0157] The first scalar value corresponds to the terminal air volume, and the second scalar value corresponds to the pump frequency. Specifically, the unit of measurement for the terminal air volume is consistent with the engineering measurement unit of the refrigeration system, and the unit of measurement for the pump frequency is Hertz. It should be noted that the terminal air volume and pump frequency output by the parameter network are raw values ​​without physical constraints. Subsequently, during the execution phase, these raw values ​​will be subject to boundary constraint processing based on the preset allowable ranges for air volume and frequency to ensure that the final control commands issued do not exceed the physical operating boundaries of the refrigeration equipment.

[0158] In some implementations, the step of determining the expected return value of each discrete action and its corresponding continuous control parameter-based hybrid action using a value network based on higher-order state feature vectors, discrete actions, and corresponding continuous control parameters includes:

[0159] Step S141: Concatenate the high-order state feature vector, the discrete action identifier, and the continuous control parameters corresponding to the discrete action to obtain the joint input vector of the value network.

[0160] In this embodiment, unlike the parameter network in step S131 which only requires high-order state feature vectors and discrete action identifiers as input, the value network requires additional continuous control parameters as input. It is understood that the task of the value network is to evaluate the expected return value of a complete hybrid action, which is defined by both discrete actions and continuous control parameters. Therefore, the value network must simultaneously perceive both the discrete action type and the specific values ​​of the continuous parameters to accurately judge the quality of the hybrid action. If the value network only knows the discrete action type but not the continuous parameter values, it cannot distinguish the value differences of different combinations of continuous parameters under the same discrete action. If it only knows the continuous parameters but not the discrete action type, it cannot distinguish the influence of different discrete action strategies.

[0161] Specifically, the splicing operation may include concatenating the high-order state feature vector, the discrete action identifier vector, and the continuous control parameter vector output in step S13 along their vector dimensions to form a larger-dimensional joint input vector for the value network. The high-order state feature vector encodes a high-order abstract representation of the environmental state. The discrete action identifier vector distinguishes between three discrete actions: reduction, maintaining the status quo, or increase. The continuous control parameter vector consists of two scalar values: the terminal air volume and the pump frequency, representing the specific continuous control parameters generated in step S13 for that discrete action.

[0162] In terms of encoding, the discrete action identifiers use the same one-hot encoding scheme as in step S131, that is, using a three-dimensional vector to represent the three discrete actions. The continuous control parameter vectors directly use the two original scalar values ​​output in step S13 without additional encoding processing to maintain numerical accuracy. The three input parts are concatenated end-to-end according to a predetermined concatenation order, which remains fixed and consistent during training and inference.

[0163] In this embodiment, step S14 requires evaluating the three candidate hybrid actions in the discrete action set separately. For each candidate discrete action, the value network performs a complete forward inference with the same high-order state feature vector, the one-hot encoded identifier of the discrete action, and the continuous control parameters generated by the parameter network for the discrete action in step S13 as input. The three forward inferences share the same set of value network parameters, and only the discrete action identifier and continuous control parameters in the input produce different expected reward values.

[0164] Step S142: Input the joint input vector of the value network into the fully connected network for nonlinear mapping, and output the scalar expected return value corresponding to the hybrid action.

[0165] In this embodiment, both the value network and the parameter network adopt a fully connected network structure, specifically including a stack of an input layer, several hidden layers, and an output layer. Specifically, the output layer of the parameter network may include two neurons, corresponding to the two continuous control parameters of the terminal airflow and the pump frequency, respectively. The output layer of the value network includes only one neuron, outputting a scalar value, which is the expected reward of the current hybrid action in the current state. The expected reward value can be an estimate of the expected cumulative reward that can be obtained by continuously interacting with the environment according to subsequent strategies after performing the hybrid action, starting from the current state. A higher expected reward value indicates a greater expected long-term benefit from performing the hybrid action.

[0166] In one possible and specific implementation, the value network employs the same two-layer hidden layer structure as the parameter network. The first hidden layer comprises 128 neurons, and the second hidden layer comprises 64 neurons. All hidden layers are fully connected, as are the hidden layers with the input and output layers. The activation function of the hidden layers is a hyperbolic tangent function, which maps the input to an output range between -1 and 1. Its output is symmetrically distributed around the value of 0, which helps maintain the output of each layer within a stable range during deep network training. The output layer does not apply an activation function and directly outputs a scalar value linearly, ensuring that the range of the expected reward value is not limited by the bounded interval of the activation function.

[0167] In some implementations, after outputting the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume, and pump frequency, the method further includes:

[0168] Step S161: Obtain the energy consumption data of the refrigeration system and the temperature data of the computer room after joint adjustment.

[0169] In this embodiment, after the control command is issued and executed in step S15, each device in the refrigeration system operates for a complete decision time slot according to the command parameters. At the end of this time slot, the control device collects system operation feedback data under this round of control action through sensors and metering devices deployed in various parts of the refrigeration system.

[0170] In this embodiment, the energy consumption data of the refrigeration system may include the total electrical energy consumption of the electric chiller, cooling tower, computer room air conditioning terminal, chilled water pump, and cooling water pump within the time slot. Specifically, the acquisition method may include: reading the real-time power or cumulative power consumption of each device by using a power meter or energy meter installed in the power supply circuit of each device, and summing the power values ​​of the chiller unit, cooling tower fan, computer room air conditioning terminal fan, chilled water pump, and cooling water pump to obtain the total energy consumption of the refrigeration system in the time slot. The computer room temperature data is the average temperature of the computer room at the end of the time slot. Its acquisition method is the same as the acquisition method of the computer room temperature sequence in step S11, which is obtained by taking the arithmetic mean after collecting data from multiple temperature sensors distributed in the cold aisle of the computer room.

[0171] Step S162: Calculate the energy efficiency value based on the energy consumption data of the refrigeration system and the IT load data in the multivariate time series state vector.

[0172] In this embodiment, the Power Usage Effectiveness (PUE) value is an indicator that measures the energy utilization efficiency of a data center. It is defined as the ratio of the total energy consumption of the data center to the energy consumption of IT equipment. The total energy consumption is the sum of the energy consumption of the cooling system and the energy consumption of IT equipment. Specifically, the PUE value is calculated as follows: the total energy consumption of the cooling system in the current time slot is added to the IT load power in the current time slot to obtain the total energy consumption. Then, the total energy consumption is divided by the IT load power in the current time slot, and the quotient is the PUE value for that time slot. The IT load power in the current time slot is directly read from the multivariate time series state vector obtained in step S11, specifically the current time component of the IT load sequence in the state vector. The closer the PUE value is to 1, the lower the auxiliary energy consumption of the cooling system and the higher the energy utilization efficiency.

[0173] In one possible and specific implementation, the PUE value is used in the reward calculation as a transient value for a single time slot. In other implementations, the average PUE value of multiple consecutive time slots within a sliding window can also be used in the reward calculation to smooth out the interference of transient fluctuations on the reward signal.

[0174] Step S163: Calculate the temperature violation penalty value based on the difference between the computer room temperature data and the preset temperature safety threshold.

[0175] In this embodiment, the temperature violation penalty value is used to quantify the thermal safety risks of the data center. Its function is to guide the agent to actively avoid temperature states that may threaten the safe operation of the server during training. Specifically, the temperature violation penalty value can be calculated using the Softplus function. More specifically, the Softplus function is a smooth soft constraint function. Its input is the difference between the data center temperature and a preset temperature safety threshold, and its output is the penalty value after nonlinear mapping of this difference. The mathematical characteristics of the Softplus function are as follows: when the input difference is much less than 0, that is, when the data center temperature is much lower than the safety threshold, the function output approaches zero, indicating that there is no safety risk in the current temperature state. When the input difference is close to 0, that is, when the data center temperature is approaching the safety threshold, the function output begins to gradually increase, generating a mild penalty signal to prompt the agent to control the temperature from continuing to rise. When the input difference is greater than 0, that is, when the data center temperature exceeds the safety threshold, the function output increases approximately linearly with the increase of the difference, generating a strong penalty signal to force the agent to take cooling measures. This smooth, gradual penalty mechanism, compared to a hard-truncation indicator function, provides the agent with continuous gradient signals, which helps to achieve stable convergence during the training process.

[0176] Step S164: Calculate the physical constraint violation penalty value based on whether the continuous control parameters exceed the preset operating boundary.

[0177] In this embodiment, it is understood that the continuous control parameters output by the parameter network in step S13 are raw values ​​without physical constraint pruning. In the execution phase of step S15, these values ​​need to undergo boundary constraint processing to ensure they fall within the device's allowed operating range. However, during training, when the raw values ​​output by the parameter network exceed the preset operating boundary, it indicates that the network's current strategy tends to select unexecutable actions. An additional penalty signal needs to be applied to guide the network to output continuous control parameters within the feasible region. The physical constraint violation penalty value is the specific carrier of this penalty signal.

[0178] In this embodiment, the calculation method for the physical constraint violation penalty value may specifically include:

[0179] The system checks whether the initial value of the terminal airflow exceeds the preset allowable airflow range and whether the initial value of the pump frequency exceeds the preset allowable frequency range. If the initial value of the terminal airflow is within the allowable range, this dimension does not contribute a penalty. If the initial value of the terminal airflow exceeds the upper limit of the allowable range or falls below the lower limit of the allowable range, a corresponding penalty component is generated based on the magnitude of the exceedance. The same applies to the pump frequency dimension. The sum of the penalty components of the two dimensions yields the physical constraint violation penalty value.

[0180] Step S165: Based on the preset positive reward constant, subtract the product of the energy use efficiency value and the first weighting coefficient, the product of the temperature violation penalty value and the second weighting coefficient, and the physical constraint violation penalty value to obtain the reward value.

[0181] In this embodiment, the reward value may include four linear terms.

[0182] Specifically, the first item is the preset positive reward constant, which is a fixed positive value. Its function is to provide a positive baseline value for the reward function, so that the reward value remains positive under most normal working conditions, which is beneficial for the agent to establish positive behavioral guidance in the early stage of training.

[0183] The second item is the energy efficiency penalty item, which can include the product of the power usage efficiency value and the first weighting coefficient. This item generates a larger penalty when the PUE is high, guiding the agent to reduce cooling energy consumption to improve energy efficiency.

[0184] The third item is the temperature safety penalty item, which can include the product of the temperature violation penalty value and the second weighting coefficient. This item generates a penalty when the computer room temperature approaches or exceeds the safety threshold, guiding the intelligent agent to ensure thermal safety.

[0185] The fourth item is the physical constraint penalty item, which is the physical constraint violation penalty value output in step S164 itself. This item generates a penalty when the continuous control parameters exceed the operating boundary of the device, guiding the agent to output physically executable actions.

[0186] In this embodiment, the first weighting coefficient and the second weighting coefficient are used to adjust the relative importance of energy efficiency targets and safety targets in the reward function. When the first weighting coefficient is large and the second weighting coefficient is small, energy efficiency optimization takes precedence over temperature safety, and the agent tends to reduce cooling capacity to lower energy consumption, but the data center temperature may approach or even exceed the safety threshold. When the second weighting coefficient is large and the first weighting coefficient is small, temperature safety takes precedence over energy efficiency optimization, and the agent tends to maintain a large cooling capacity to ensure temperature safety, but this may lead to over-cooling and increased energy consumption.

[0187] In one possible and specific implementation, the preset positive reward constant is set to 1.5, the first weighting coefficient is set to 0.4, and the second weighting coefficient is set to 0.6. This configuration, determined through sensitivity analysis experiments, achieves a good balance between operational safety and energy saving. In other implementations, the first and second weighting coefficients can be dynamically adjusted according to the actual data center's emphasis on energy efficiency and safety. For example, the second weighting coefficient can be appropriately increased during the high-temperature period in summer to strengthen the focus on temperature safety.

[0188] Step S166: Based on the reward value, iteratively update the parameters of the feature extraction network, parameter network, and value network.

[0189] In this embodiment, the network parameters are updated using an experience replay mechanism and the Bellman error minimization method. Specifically, after each decision-making time slot interacts with the environment, the control device stores an experience sample in the experience replay pool. Each experience sample includes the multivariate time series state vector of the current time slot, the high-order state feature vector extracted in step S12, the discrete action selected in steps S13 to S15, the continuous control parameters corresponding to the discrete action, the reward value calculated in step S165, and the multivariate time series state vector of the new time slot after the state transition. The experience replay pool has an upper limit on its capacity. When the number of samples in the pool exceeds the capacity limit, the earliest stored sample is removed to ensure that the experience replay pool stores the most recent interaction data.

[0190] In this embodiment, when the number of samples in the experience replay pool reaches a preset threshold, the control device randomly samples a batch of experience samples from the experience replay pool and uses these samples to calculate the target value and loss function of the value network.

[0191] In this embodiment, the target value of the value network can be calculated based on the Bellman equation. Specifically, for each empirical sample, the higher-order state feature vector of the new state after the transition is input into the parameter network to generate continuous control parameters corresponding to each discrete action in the new state. Then, the higher-order feature vector of the new state, each discrete action, and its continuous parameters are input into the target value network. The maximum value among the expected reward values ​​corresponding to each discrete action is taken, multiplied by a preset discount factor, and then added to the current reward value to obtain the target value corresponding to that sample. The loss function of the value network is the mean of the squares of the differences between the target value of each sample and the expected reward value output by the current value network. The gradient is calculated based on this loss function, and the parameters of the value network are updated through backpropagation.

[0192] In this embodiment, the update objective of the parameter network is to maximize the expected reward value output by the value network. For each empirical sample, the high-order feature vector of the current state is input into the parameter network to obtain the continuous control parameters corresponding to the current discrete action. These continuous control parameters, along with the high-order feature vector and the discrete action identifier, are then input into the value network to obtain the expected reward value. The loss function of the parameter network is the mean of the negative expected reward value; minimizing this loss function is equivalent to maximizing the expected reward value. The gradient is calculated based on this loss function, and backpropagation updates only the parameters of the parameter network, while the parameters of the value network remain unchanged during this step.

[0193] In this embodiment, the parameters of the feature extraction network are backpropagated to the feature extraction network through cascaded connections in the value network update step, along with the gradient of the value network loss function, and are updated synchronously with the value network parameters.

[0194] In this implementation, value network updates and parameter network updates are performed alternately. After sampling a batch of samples from the experience replay pool, the value network parameters are updated first, followed by the parameter network parameters. Furthermore, to improve training stability, each of the value network and parameter network corresponds to a target network. The parameters of the target network are not directly updated via gradient descent, but rather through a soft update at preset steps, performed at a small ratio from the current network parameters to the target network parameters. Specifically, the target network parameter values ​​are updated by multiplying the current target network parameter value by 1, subtracting the soft update coefficient, and then adding the current network parameter value multiplied by the soft update coefficient. The soft update coefficient is small, for example, 0.005, ensuring smooth and slow changes in the target network parameters and providing a stable reference benchmark for target value calculation.

[0195] In this embodiment, the above-mentioned processes of experience collection, sampling, loss calculation and parameter update are executed cyclically in each decision time slot. As the training rounds progress, the parameters of the feature extraction network, parameter network and value network gradually converge to a state that can output a high reward value strategy, ultimately enabling the control device to autonomously generate an energy-saving and safe proactive cooling control strategy under dynamic IT load and outdoor environmental conditions.

[0196] In some implementations, the step of calculating the temperature violation penalty value based on the difference between the computer room temperature data and a preset temperature safety threshold includes:

[0197] Step S1631: Determine the difference between the computer room temperature data and the preset temperature safety threshold.

[0198] In this embodiment, the average temperature of the computer room at the end of the current time slot can be read from the combined adjusted computer room temperature data collected in step S161. This temperature value is then subtracted from a preset temperature safety threshold. The difference between the computer room temperature data and the preset temperature safety threshold is used as the basic measure of the degree of temperature violation. A positive difference indicates that the computer room temperature has exceeded the safety threshold, posing a thermal safety risk. A negative difference indicates that the computer room temperature is within the safe range. A difference of zero indicates that the computer room temperature is exactly equal to the safety threshold.

[0199] Step S1632: Input the difference into the Softplus function and output the temperature violation penalty value; wherein, the Softplus function is a smooth function that monotonically increases with the difference, the output approaches zero when the difference is negative and decreasing, and the output continues to increase when the difference is positive and increasing.

[0200] In this embodiment, the calculation process of the Softplus function can be as follows: First, perform an exponential operation on the input value to obtain a power value with the natural constant e as the base and the input value as the exponent. Then, add this power value to 1 to obtain a sum. Finally, take the natural logarithm of this sum, and the result of the logarithmic operation is the output value of the Softplus function. The above calculation process can be expressed as the output being equal to 1 plus the natural logarithm of the sum of the input powers to the base e.

[0201] In one possible and specific implementation, this implementation alternately updates QNet and xNet by minimizing the Bellman error and maximizing the Q value, and its complete training process is shown in the table below.

[0202] In some embodiments, the step of outputting the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume, and water pump frequency includes:

[0203] Step S151: Select the discrete action corresponding to the maximum expected return value and obtain the continuous control parameters corresponding to the discrete action.

[0204] In this embodiment, step S14 outputs a scalar expected return value for each of the three candidate discrete actions in the discrete action set. The control device can compare these three expected return values, determine the one with the largest value, and determine the discrete action corresponding to the largest value as the discrete action to be executed in the current decision time slot.

[0205] In this embodiment, after determining the selected discrete action, the control device can extract the set of continuous control parameters corresponding to the selected discrete action from the three sets of continuous control parameters output by the parameter network. This set of continuous control parameters includes two components: the first component corresponds to the terminal air volume, and the second component corresponds to the water pump frequency.

[0206] It should be noted that step S151 only performs the maximum value operation during the inference phase. During the training phase, to encourage the agent to fully explore the environment, the control device randomly selects discrete actions with a certain probability according to a greedy strategy, rather than taking the maximum value. The exploration probability is relatively high in the early stage of training, gradually decays as training progresses, and eventually converges to the deterministic strategy of taking the maximum value.

[0207] Step S152: Determine the updated value of the chiller unit outlet water temperature based on discrete actions; wherein, when the discrete action is to decrease, the current outlet water temperature is decreased by a preset temperature step; when the discrete action is to increase, the current outlet water temperature is increased by a preset temperature step; when the discrete action is to maintain, the current outlet water temperature is kept unchanged.

[0208] In this embodiment, the discrete action set includes three operation types, each corresponding to a different adjustment direction of the chiller unit's outlet water temperature. When the discrete action is selected as "decrease," the control device reads the chiller unit's outlet water temperature value for the current time slot from the multivariate time series state vector, subtracts a preset temperature step size from the current outlet water temperature value, and uses the difference as the updated outlet water temperature value. When the discrete action is selected as "increase," the current outlet water temperature value is added to the preset temperature step size, and the sum is used as the updated outlet water temperature value. When the discrete action is selected as "maintain," no addition or subtraction operation is performed; the current outlet water temperature value is directly used as the updated value, i.e., the current outlet water temperature remains unchanged.

[0209] In one possible and specific implementation, the preset temperature step size is 0.5 degrees Celsius, corresponding to the smallest adjustment granularity of the chiller unit's outlet water temperature. This step size value matches the accuracy of the chiller unit controller's temperature setting interface, meaning the chiller unit controller can receive temperature setting commands in units of 0.5 degrees Celsius. The current outlet water temperature value is directly read from the current time component of the chiller unit's outlet water temperature sequence in the multivariate time series state vector obtained in step S11.

[0210] In this embodiment, the adjustable range of the chiller unit's outlet water temperature is 6 to 12 degrees Celsius, with 13 discrete levels in 0.5-degree Celsius increments. The decrease and increase operations in the discrete action set correspond to progressively moving between these discrete levels, rather than directly setting arbitrary temperature values. This discrete-level temperature regulation method is consistent with the setting interface of actual chiller unit controllers; that is, industrial chillers often change the outlet water temperature setting through level or relay switching, rather than accepting continuously changing temperature setpoints. This embodiment uses a three-choice discrete action representation method precisely to natively match this physical control interface characteristic and avoid control interface mismatch problems that arise after relaxing discrete temperature setpoints into continuous variables.

[0211] Step S153: When the terminal air supply volume value exceeds the preset allowable air supply volume range, constrain the terminal air supply volume value to the boundary value of the preset allowable air supply volume range.

[0212] In this embodiment, the initial value of the terminal air volume output by the parameter network is an unconstrained scalar value, which may exceed the physical operating boundaries of the terminal fans of the computer room air conditioning system during training. Therefore, step S153 performs a constraint operation on this initial value. Specifically, the initial value of the terminal air volume is compared with the lower limit and upper limit of the preset allowable air volume range. If the initial value of the terminal air volume is less than the lower limit of the allowable range, it is forcibly set to the lower limit of the allowable range. If the initial value of the terminal air volume is greater than the upper limit of the allowable range, it is forcibly set to the upper limit of the allowable range. If the initial value of the terminal air volume is within the allowable range, the original value remains unchanged.

[0213] Step S154: When the pump frequency value exceeds the preset frequency allowable range, constrain the pump frequency value to the boundary value of the preset frequency allowable range.

[0214] In this embodiment, step S154 can be consistent with the constraint processing logic of step S153.

[0215] Step S155: When the updated value of the chiller unit's outlet water temperature exceeds the preset temperature allowable range, the updated value is constrained to the boundary value of the preset temperature allowable range.

[0216] In this embodiment, the updated chiller outlet water temperature calculated in step S152 based on discrete actions and a preset temperature step size may exceed the physical operating temperature range of the chiller after accumulation or subtraction operations. Therefore, step S155 also needs to perform a constraint operation on this updated value.

[0217] Step S156: Output the terminal air volume, water pump frequency, and chiller outlet water temperature after constraint processing, so as to jointly adjust the chiller outlet water temperature, terminal air volume, and water pump frequency.

[0218] In this embodiment, the terminal airflow value after constraint processing in step S153, the water pump frequency value after constraint processing in step S154, and the chiller outlet water temperature value after constraint processing in step S155 are assembled into a joint control command package, which is then sent to each execution device through the control communication interface of the refrigeration system. Specifically, the chiller outlet water temperature value is sent to the chiller controller as a new outlet water temperature setpoint. The terminal airflow value is sent to the inverter driver of the computer room air conditioning terminal as a setpoint for fan speed or airflow. The water pump frequency value is sent to the inverter driver of the chilled water pump as a setpoint for pump speed. All three commands are sent synchronously within the same decision time slot, and each device begins to respond and operate according to the set parameters in the next control cycle after receiving the command.

[0219] like Figure 4 This invention provides a collaborative energy-saving control device for a data center cooling system, comprising:

[0220] The acquisition module is used to acquire a multivariate time series state vector including the current time and N past times; wherein the multivariate time series state vector includes an IT load sequence and a thermodynamic state sequence; wherein the thermodynamic state sequence includes a computer room temperature sequence, a chiller unit outlet water temperature sequence, and an outdoor ambient temperature sequence.

[0221] An extraction module is used to process multivariate time series state vectors through a feature extraction network to obtain high-order state feature vectors. The feature extraction network includes a convolutional path, a long short-term memory path with a time attention mechanism, and an autoregressive linear path. The convolutional path extracts local coupling features between variables in the state vector. The long short-term memory path and the time attention mechanism capture long-term dependencies in state evolution and generate a context vector. The autoregressive linear path performs a linear transformation on the state vector to extract linear trends. The high-order state feature vector is obtained by fusing the context vector and the output of the autoregressive linear path.

[0222] The generation module is used to generate continuous control parameters corresponding to each discrete action in the preset discrete action set by using a parameter network based on a high-order state feature vector. The discrete action set includes three discrete actions corresponding to the decrease, maintenance, and increase of the chiller unit outlet water temperature, respectively. The continuous control parameters include the terminal air volume and the water pump frequency.

[0223] The determination module is used to determine the expected return value of each discrete action and the corresponding continuous control parameter of the hybrid action based on the high-order state feature vector, discrete action and the continuous control parameter corresponding to the discrete action through the value network.

[0224] The output module is used to select the discrete action corresponding to the maximum expected return value, and output the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume and water pump frequency.

[0225] According to an embodiment of the present invention, an electronic device is provided; please refer to... Figure 5 The electronic device in this embodiment may include one or more of the following components: a processor, a network interface, memory, non-volatile memory, and one or more application programs, wherein the one or more application programs may be stored in non-volatile memory and configured to be executed by one or more processors, and the one or more programs are configured to perform the methods as described in the foregoing method embodiments.

[0226] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for coordinated energy-saving control of a data center cooling system, characterized in that, include: Obtain a multivariate time series state vector including the current time and N past times; wherein the multivariate time series state vector includes an IT load sequence and a thermodynamic state sequence; wherein the thermodynamic state sequence includes a computer room temperature sequence, a chiller unit outlet water temperature sequence, and an outdoor ambient temperature sequence; A high-order state feature vector is obtained by processing the multivariate time series state vector through a feature extraction network. The feature extraction network includes a convolutional path, a long short-term memory path with a time attention mechanism, and an autoregressive linear path. The convolutional path is used to extract local coupling features between variables in the state vector. The long short-term memory path and the time attention mechanism are used to capture long-term dependencies in state evolution and generate a context vector. The autoregressive linear path is used to perform a linear transformation on the state vector to extract linear trends. The high-order state feature vector is obtained by fusing the context vector and the output of the autoregressive linear path. For each discrete action in the preset discrete action set, the continuous control parameters corresponding to the discrete action are generated by the parameter network based on the high-order state feature vector; wherein, the discrete action set includes three discrete actions corresponding to the decrease, maintenance and increase of the chiller unit outlet water temperature respectively; wherein, the continuous control parameters include the terminal air volume and the water pump frequency; The expected return value of each discrete action and the corresponding continuous control parameter is determined by using a value network based on high-order state feature vectors, discrete actions, and the continuous control parameters corresponding to the discrete actions. Select the discrete action corresponding to the maximum expected return value, and output the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume and water pump frequency.

2. The method of claim 1, wherein, The step of processing the multivariate time series state vector through a feature extraction network to obtain a high-order state feature vector includes: The multivariate time series state vector is input into the convolution path, and the local coupling features between variables are extracted by the convolution kernel sliding along the time axis to obtain the convolution feature matrix; The convolutional feature matrix is ​​input into a long short-term memory pathway with a time attention mechanism. The hidden state at each historical moment is obtained through the long short-term memory network. The correlation score between the hidden state at each historical moment and the hidden state at the last moment is calculated through the time attention mechanism. The hidden states at each historical moment are weighted and summed based on the correlation score to generate a context vector. The multivariate time series state vector is input into the autoregressive linear path, and the linear trend is extracted through linear transformation to obtain the autoregressive output vector; By fusing the context vector and the autoregressive output vector, a higher-order state feature vector is obtained.

3. The method of claim 2, wherein, The step of generating continuous control parameters corresponding to each discrete action in a preset set of discrete actions using a parameter network based on a high-order state feature vector includes: The high-order state feature vector is concatenated with the discrete action identifier to obtain the joint input vector of the parameter network; The parameter network is combined with the input vector and input to a fully connected network for nonlinear mapping, outputting the continuous control parameters corresponding to the discrete action.

4. The method of claim 3, wherein, The step of determining the expected return value of a hybrid action consisting of each discrete action and its corresponding continuous control parameters based on a value network using a high-order state feature vector, discrete actions, and the corresponding continuous control parameters includes: The high-order state feature vector, discrete action identifier, and the continuous control parameter corresponding to the discrete action are concatenated to obtain the joint input vector of the value network. The joint input vector of the value network is input into a fully connected network and nonlinearly mapped to output the scalar expected return value corresponding to the hybrid action.

5. The method of claim 1, wherein, After outputting the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume, and pump frequency, the following is also included: Acquire energy consumption data of the refrigeration system and temperature data of the computer room after joint adjustment; Based on the energy consumption data of the refrigeration system and the IT load data in the multivariate time series state vector, the energy efficiency value is calculated. Calculate the penalty value for temperature violation based on the difference between the data of the computer room temperature and the preset temperature safety threshold; Calculate the physical constraint violation penalty value based on whether the continuous control parameters exceed the preset operating boundary; The reward value is obtained by subtracting the product of the energy use efficiency value and the first weighting coefficient, the product of the temperature violation penalty value and the second weighting coefficient, and the physical constraint violation penalty value from the preset positive reward constant. Based on the reward value, the parameters of the feature extraction network, parameter network, and value network are iteratively updated.

6. The method of claim 5, wherein, The step of calculating the temperature violation penalty value based on the difference between the computer room temperature data and the preset temperature safety threshold includes: Determine the difference between the computer room temperature data and the preset temperature safety threshold; The difference is input into the Softplus function, which outputs the temperature violation penalty value. The Softplus function is a smooth function that increases monotonically with the difference. When the difference is negative and decreasing, the output approaches zero, and when the difference is positive and increasing, the output continues to increase.

7. The method of claim 1, wherein, The step of outputting the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume, and water pump frequency includes: Select the discrete action corresponding to the maximum expected return value and obtain the continuous control parameters corresponding to the discrete action; The updated value of the chiller unit's outlet water temperature is determined based on discrete actions; wherein, when the discrete action is to decrease, the current outlet water temperature is decreased by a preset temperature step size; when the discrete action is to increase, the current outlet water temperature is increased by a preset temperature step size; when the discrete action is to maintain, the current outlet water temperature is kept unchanged. When the terminal air supply volume exceeds the preset allowable air supply volume range, the terminal air supply volume will be constrained to the boundary value of the preset allowable air supply volume range. When the pump frequency exceeds the preset allowable frequency range, the pump frequency will be constrained to the boundary value of the preset allowable frequency range. When the updated value of the chiller unit's outlet water temperature exceeds the preset temperature allowable range, the updated value will be constrained to the boundary value of the preset temperature allowable range. The system outputs the terminal air volume, water pump frequency, and chiller outlet water temperature after constraint processing, so as to jointly regulate the chiller outlet water temperature, terminal air volume, and water pump frequency.

8. A data center refrigeration system coordinated energy saving control apparatus, characterized by, include: The acquisition module is used to acquire a multivariate time series state vector including the current time and N past times; wherein the multivariate time series state vector includes an IT load sequence and a thermodynamic state sequence; wherein the thermodynamic state sequence includes a computer room temperature sequence, a chiller unit outlet water temperature sequence, and an outdoor ambient temperature sequence. An extraction module is used to process multivariate time series state vectors through a feature extraction network to obtain high-order state feature vectors. The feature extraction network includes a convolutional path, a long short-term memory path with a time attention mechanism, and an autoregressive linear path. The convolutional path extracts local coupling features between variables in the state vector. The long short-term memory path and the time attention mechanism capture long-term dependencies in state evolution and generate a context vector. The autoregressive linear path performs a linear transformation on the state vector to extract linear trends. The high-order state feature vector is obtained by fusing the context vector and the output of the autoregressive linear path. The generation module is used to generate continuous control parameters corresponding to each discrete action in the preset discrete action set by using a parameter network based on a high-order state feature vector. The discrete action set includes three discrete actions corresponding to the decrease, maintenance, and increase of the chiller unit outlet water temperature, respectively. The continuous control parameters include the terminal air volume and the water pump frequency. The determination module is used to determine the expected return value of each discrete action and the corresponding continuous control parameter of the hybrid action based on the high-order state feature vector, discrete action and the continuous control parameter corresponding to the discrete action through the value network. The output module is used to select the discrete action corresponding to the maximum expected return value, and output the discrete action and its corresponding continuous control parameters to jointly regulate the chiller outlet water temperature, terminal air volume and water pump frequency.

9. An electronic device, comprising: include: A memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors to cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images