Data center global temperature control optimization method and system based on hybrid reinforcement learning

Abstract

The invention discloses a data center global temperature control optimization method based on online-offline hybrid reinforcement learning, and the method comprises the steps: constructing an interpretable global state space fusing physical information, and guaranteeing that a state variable has a clear physical meaning; a combined action space is defined, and cooperative control of the cold source side and the tail end side is achieved; designing a multi-objective reward function based on physical model driving, and comprehensively considering energy consumption, temperature stability, carbon emission and physical optimization indexes; obtaining a basic security policy through offline pre-training, and extracting reliable behaviors from historical data by using a conservative Q learning algorithm; strategy optimization is achieved through online safety fine adjustment, and gradual adjustment is conducted under multiple constraints to adapt to real-time changes; and finally, deploying an optimization strategy to realize closed-loop control of the system, and establishing a continuous learning mechanism to cope with long-term drift. According to the method, a complete offline-online mixed learning system is established, and the energy efficiency of the data center temperature control system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data center energy-saving control, and particularly relates to a data center global temperature control optimization method and system based on online-offline hybrid reinforcement learning, an electronic device and a storage medium. BACKGROUND

[0002] As the core of modern digital infrastructure, data centers have increasingly prominent energy consumption problems. With the rapid development of AI, cloud computing and big data, this proportion is expected to further significantly increase. This not only increases operating costs, but also poses challenges to global energy sustainability and carbon emission targets. In the composition of data center energy consumption, the proportion of temperature control systems is relatively high, and it usually becomes one of the main energy consumption units. The HVAC system is responsible for maintaining the stable temperature and humidity of the server to ensure the reliable operation of the equipment, but its energy efficiency optimization faces many difficulties. Traditional control methods perform poorly in dealing with complex and variable environments, and the overshoot rate of PID control is relatively high, resulting in obvious energy waste. Existing technologies often rely on fixed settings or manual experience, and when the load changes and the pipe network working conditions fluctuate, they cannot be dynamically adjusted, causing redundant operation of the water pump and insufficient flow in some terminals, ultimately affecting the cooling effect and overall energy efficiency. This inefficiency not only amplifies energy consumption, but also may cause equipment overheating or failure, increasing maintenance burden.

[0003] Existing differential pressure control and optimization methods generally lack real-time monitoring and adaptive adjustment capabilities for pipe network hydraulic characteristics, and are usually based on static models, ignoring pipe resistance and load mutations, resulting in response lag, cooling fluctuations and reduced stability. At the same time, pure offline reinforcement learning faces the problem of distribution shift, and historical operation data cannot fully cover special scenarios such as extreme high temperatures in summer, equipment performance degradation, and sudden IT load, resulting in serious performance degradation of the strategy trained based on historical data after actual deployment. This distribution shift problem makes it difficult for pure offline learning methods to adapt to changes in the real operating environment. Although pure online reinforcement learning can continuously adapt to system changes, its exploration process has high safety risks, and these defects make it difficult for the system to balance energy saving and comfort, especially when complex subsystems are coupled, the optimization space is lost more obviously. However, the high-dimensional action vector output by traditional reinforcement learning methods cannot be directly related to the specific device state, and the decision-making process lacks transparency, forming a trust crisis caused by black box decision-making.

[0004] In addition, the existing reinforcement learning method has defects in the state space and reward function design, usually directly using the original sensor data, resulting in high feature dimension, high redundancy, and lack of clear physical meaning. The reward function design is too simple, only considering surface indicators such as energy consumption and temperature, ignoring key physical quantities such as thermodynamic optimal COP, Darcy head, and pinch temperature, resulting in control decisions that violate physical laws and cannot achieve truly optimal control. The existing method controls and optimizes the cold source system and the terminal air conditioning system independently, ignoring the coupling characteristics between the subsystems, resulting in optimization space loss of the total energy consumption of the system, forming a clear "energy-saving island" phenomenon. The lack of model adaptability further exacerbates the problem, and the drift phenomenon such as equipment performance aging and load mode change during system operation causes the control strategy to gradually fail. To maintain system performance, the retraining cost is high, which brings a heavy burden to operation. The rule-based control method relies heavily on expert experience and often fails when faced with complex and variable working conditions. PID control performs well in linear systems, but for temperature control systems with nonlinear, large delay, and multivariable coupling characteristics, its overshoot is serious. Model predictive control (MPC) can theoretically provide optimal control, but it relies on accurate thermodynamic models, which is costly to construct, and the model error is usually high.

[0005] In summary, based on the above analysis, there is an urgent need for an innovative technical solution that can effectively solve the above problems, and achieve global optimization and continuous adaptive control of the data center temperature control system under the premise of ensuring system safety. SUMMARY

[0006] The purpose of the present application is to solve the shortcomings of traditional methods in safety, adaptability and explainability, and to provide a data center global temperature control optimization method based on online-offline hybrid reinforcement learning for realizing efficient, safe and reliable operation of data center heating, ventilation and air conditioning system through systematic data processing, model training and online optimization process, as well as a system, electronic device and storage medium that can realize the above method.

[0007] In order to achieve the above purpose, the present application discloses a data center global temperature control optimization method based on hybrid reinforcement learning, comprising the following steps: Step S1. Constructing an explainable global state space integrating physical information: based on the operating physical mechanism of the data center heating, ventilation and air conditioning system, converting the original sensor data into state characteristic variables with clear physical meaning, including system sensible heat ratio, virtual load rate, system approach subcooling degree and device comprehensive efficiency index; Step S2. Define the joint action space of the cold source side and the terminal side: Establish a collaborative control system that includes the control actions of the cold source side equipment and the control actions of the terminal side equipment. The cold source side actions include the chiller unit outlet water temperature setpoint, the cooling water pump frequency setpoint and the cooling tower fan speed. The terminal side actions include the air supply temperature setpoint of the computer room air conditioner, the water valve opening degree and the fan speed adjustment amount. Step S3. Design a multi-objective reward function driven by a physical model: Taking into account the energy consumption, temperature compliance, control action stability, carbon emissions, and physical information constraints of the data center HVAC system, construct a global optimization objective function that includes multiple sub-rewards to improve system performance while meeting safety constraints; Step S4. Offline pre-training stage: Train the initial control policy using historical running data, and use the conservative Q-learning algorithm to introduce a conservative regularization term in the objective function to prevent overestimation of the value function and ensure that the trained policy is within the range of historical safe behavior. Step S5. Online security fine-tuning stage: Under multiple security constraints, the offline strategy is progressively optimized based on real-time interactive data. The trust domain strategy optimization algorithm is used to limit the policy update range to ensure the stability of the policy update and make the policy adapt to real-time operating conditions. Step S6. Strategy Deployment and Online Control: Deploy the trained and optimized control strategy to the data center HVAC system to achieve real-time closed-loop control of the cold source and terminal equipment, and establish a continuous learning mechanism to cope with environmental drift during long-term operation.

[0008] Furthermore, in step S1, the construction of the interpretable global state space includes: calculating the system sensible heat ratio: based on the total cooling capacity and latent cooling capacity of the data center HVAC system, using the formula... Calculate, where Total cooling capacity Latent cooling capacity; Virtual load factor calculation: Based on the principle of thermodynamic energy balance, using the formula... Calculations are performed, where c is the specific heat capacity of water, m is the mass flow rate of chilled water, ΔT is the temperature difference between the supply and return water, and Prated is the rated capacity of the equipment; the system's near subcooling is calculated based on the thermodynamic characteristics of the refrigeration cycle, using the formula... Calculation, where This is the condensation temperature. The evaporation temperature. The theoretical optimal approximation; Calculation of the overall equipment efficiency index: based on the equipment performance curve, using the formula... Calculation, where For the actual operating efficiency of the equipment, For design operating conditions efficiency.

[0009] Furthermore, in step S3, the multi-objective reward function consists of multiple weighted sub-rewards; The sub-rewards include: an energy consumption reward R that reflects the total energy consumption of the system. energy Temperature compliance reward R, reflecting temperature control deviation temperature Reward R reflects the stability of the controlled action and the range of change in the action. stability Carbon emission incentives R that reflect the impact of carbon emissions carbon And the reward R based on physical information physics The total reward value is obtained by linearly combining each sub-reward multiplied by its corresponding weight coefficients w1, w2, w3, w4, and w5.

[0010] Furthermore, in step S4, the offline pre-training stage uses historical running data to train the basic security policy. The algorithm used is the conservative Q-learning algorithm, which avoids overestimation of value by introducing a conservative regularization term into the value objective, thereby ensuring that the trained control policy is reliable and effective within the range of historical security behavior.

[0011] Furthermore, in step S5, the online security fine-tuning stage optimizes and adjusts the initial control strategy in real time while meeting preset security constraints. The trust domain strategy optimization algorithm is used to limit the magnitude of each strategy update to ensure the stability of the strategy improvement process and enable the optimized control strategy to adapt to the real-time operational changes of the data center HVAC system.

[0012] Furthermore, it also includes data acquisition and preprocessing steps: sensors are deployed at key locations in the data center HVAC system to collect raw operating data containing parameters such as temperature, pressure, flow rate and power consumption in real time, and the raw data is cleaned, standardized and anomaly detected.

[0013] Furthermore, it also includes an anomaly handling mechanism: when sensor data is detected to be missing or abnormal, the system uses predicted values ​​based on historical data or default control strategies to maintain the normal operation of the data center HVAC system.

[0014] This invention also discloses a data center global temperature control optimization system based on hybrid reinforcement learning, which, based on any of the aforementioned data center global temperature control optimization methods based on hybrid reinforcement learning, includes: a data acquisition and processing module: used to acquire sensor data in real time at key locations of the data center HVAC system and process the acquired raw data to calculate physical information features; a physical feature calculation module: used to calculate state feature quantities such as the system sensible heat ratio and virtual load rate of the data center HVAC system based on a physical model; an offline training module: used to train the control strategy offline using the historical operating data to obtain a basic safety strategy; an online learning module: used to fine-tune and optimize the basic safety strategy online under the condition of meeting safety constraints; a control execution module: used to convert the optimized control strategy into control signals for each controlled device in the data center HVAC system; and a safety monitoring module: used to monitor the operating status of the data center HVAC system in real time and trigger preset protection measures to ensure system operation safety when an anomaly occurs.

[0015] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the computer program, it implements the data center global temperature control optimization method based on hybrid reinforcement learning as described in any of the preceding claims.

[0016] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it is able to implement the data center global temperature control optimization method based on hybrid reinforcement learning as described in any of the preceding claims.

[0017] Compared with the prior art, the present invention has the following significant advantages: 1. Compared to existing designs, this method achieves global, efficient, and adaptive optimization control of data center HVAC systems. Specifically, it first constructs an interpretable global state space based on the physical mechanism of data center HVAC systems, including the system sensible heat ratio, virtual load rate, system near-subcooling degree, and equipment comprehensive efficiency index, allowing reinforcement learning strategies to directly address real physical states rather than simply raw sensor values. Building upon this, through a joint action space between the cold source and terminal sides, the chiller outlet water temperature, cooling water pump frequency, cooling tower fan speed, and the air supply temperature, water valve opening, and fan speed adjustment of the computer room air conditioning are unified into a single action vector, avoiding local optima caused by independent control of each subsystem. Subsequently, using a multi-objective reward function driven by a physical model as the core, energy consumption, temperature compliance, and other factors are integrated into the overall optimization control. The smoothness of control actions, carbon emissions, and physical information constraints are integrated into the same optimization objective, ensuring that the control effect no longer solely pursues energy saving but rather balances multiple objectives. In step S4, conservative Q-learning is used to perform offline pre-training on historical operating data, first learning basic safety strategies within the scope of historical safe behavior to avoid the risks brought by direct online exploration. In step S5, online safety fine-tuning is performed under multiple safety constraints in conjunction with trust domain strategy optimization, limiting the policy update range so that the policy can gradually adapt to real-time operating conditions without drastic fluctuations. Finally, through the policy deployment and continuous learning mechanism in step S6, the above-mentioned optimization strategy is applied to the actual system in a closed loop, and the strategy is continuously updated using new data. It is precisely by relying on the synergy of the above-mentioned (1) to (6) technical means that this invention achieves global optimization and long-term adaptive capability of data center HVAC systems while ensuring safety. This is a technical effect that is difficult to achieve by traditional fixed rule-based or single online learning methods.

[0018] 2. This invention, through the design of "physical information fusion + joint control + multi-objective reward", makes the optimization results not only energy-saving in numerical terms, but also reasonable and reliable at the physical level.

[0019] Specifically, in terms of physical information, the system sensible heat ratio, virtual load rate, system near subcooling degree and equipment comprehensive efficiency index are used as state characteristics to enable the intelligent agent to accurately perceive the current load nature, equipment load level and refrigeration cycle operating margin, thereby spontaneously distinguishing complex scenarios such as high humidity load, heavy load or equipment performance degradation in the control strategy. In terms of joint control, the coordinated operation of chillers, cooling towers, water pumps, and computer room air conditioners is achieved through the joint action space of the cold source side and the terminal side. This breaks the traditional fragmented model of "one logic for the cold source and another logic for the terminal", avoiding the "energy-saving island" where energy saving on the cold source side leads to increased air volume or water volume compensation on the terminal side, which in turn increases the total energy consumption of the system. In terms of multi-objective rewards, through a comprehensive balance of energy consumption, temperature compliance, action stability, carbon emissions, and physical information rewards, the strategy reduces total power consumption while maintaining stable cabinet air intake temperature, smooth control actions, reduced carbon emissions, and meeting thermodynamic and hydraulic constraints.

[0020] Based on the aforementioned specific technical means, this invention achieves a balance between energy efficiency, temperature control stability, and physical feasibility within a single control strategy. Compared with existing technologies that rely solely on experience-based settings or single-objective optimization, it significantly improves the overall operational performance of data center HVAC systems.

[0021] 3. In the path of generating and deploying control strategies, this invention uses an overall mechanism of "offline conservative pre-training + online security fine-tuning + progressive deployment + continuous learning and anomaly fallback" to make this hybrid reinforcement learning scheme truly applicable to high-reliability data center scenarios.

[0022] Specifically, in the offline phase, conservative Q-learning is adopted, and conservative regularization terms are introduced to train the basic strategy on historical safe operation data to avoid optimistic estimation of unseen state-action pairs, thereby reducing the risk of strategy deployment from the source. In the online phase, trust domain strategy optimization is introduced under multiple safety constraints. KL divergence constraints are used to limit the differences between new and old strategies. Combined with constraints such as the magnitude of action changes, temperature of key areas, and equipment power, the strategy update process is guaranteed to be controllable and predictable. At the deployment level, a gradual deployment process is adopted, which first verifies the strategy in the simulation environment and shadow mode, and then gradually expands the control scope until full control is achieved, minimizing the impact of the new strategy on the production system. At the same time, with the integrated continuous performance monitoring and retraining trigger mechanism, as well as the exception handling mechanism that automatically switches to the default safety strategy when sensor data is missing or abnormal, the system can maintain an interpretable, reversible, and evolvable operating state in the face of long-term environmental drift, equipment aging, or sudden operating conditions. Thanks to the aforementioned systematic strategy generation, updating, and deployment technologies, this invention achieves the engineering implementation of intelligent control strategies while ensuring the long-term stable and safe operation of data center HVAC systems. This is something that traditional single-tuning control methods or reinforcement learning methods without safety constraints cannot achieve simultaneously. Attached Figure Description

[0023] Figure 1 This is a control flowchart of the data center global temperature control optimization method based on hybrid reinforcement learning, as proposed in this invention. Figure 2This is a system schematic diagram of the global temperature control optimization method for data centers based on hybrid reinforcement learning, as proposed in this invention. Figure 3 This is a flowchart of the state space construction process for the data center global temperature control optimization method based on hybrid reinforcement learning, as described in this invention. Figure 4 This is a schematic diagram illustrating the multi-objective reward function calculation of the global temperature control optimization method for data centers based on hybrid reinforcement learning, as presented in this invention. Figure 5 This is a flowchart of the offline-online combined training process for the data center global temperature control optimization method based on hybrid reinforcement learning, as described in this invention. Detailed Implementation

[0024] The following specific embodiments further illustrate the implementation of the present invention, but do not constitute a limitation on the scope of protection of the present invention.

[0025] Example 1: Overall Flow of a Data Center Global Temperature Control Optimization Method Based on Hybrid Reinforcement Learning This embodiment provides a complete data center global temperature control optimization method based on hybrid reinforcement learning. Through a systematic data processing, model training and online optimization process, it solves the shortcomings of traditional methods in terms of security, adaptability and interpretability, and realizes the efficient, safe and reliable operation of data center HVAC systems.

[0026] The technical solution of this embodiment includes: constructing an interpretable global state space that integrates physical information; defining a joint action space; designing a multi-objective reward function driven by a physical model; obtaining basic security policies through offline pre-training; optimizing policies through online security fine-tuning; and deploying optimized policies to achieve closed-loop system control. The design of the state space integrating physical information enhances the interpretability of the model. The offline pre-training stage extracts the foundation of security policies from historical data, and the online security fine-tuning stage progressively optimizes these policies to adapt to real-time changes. Simultaneously, the multi-objective reward function drives energy efficiency improvement, and finally, progressive deployment and continuous learning mechanisms ensure stable system operation. This approach aims to reduce energy consumption and minimize deployment risks, making it suitable for large-scale data centers to promote green computing and intelligent operation and maintenance.

[0027] like Figure 1 As shown, an embodiment of the present invention provides a global temperature control optimization method for data centers based on hybrid reinforcement learning. This method addresses the problems of poor interpretability and insufficient adaptability of traditional methods through physical information fusion and a hybrid learning framework, improving the energy efficiency of HVAC systems and is applicable to data centers and other similar thermal systems. The core steps include the following: Step S1: Construct an interpretable global state space that integrates physical information to characterize the operating conditions and equipment status of the data center HVAC system; Step S2: Define the joint action space between the cold source side and the terminal side, and establish a collaborative control system between the cold source system and the terminal system; Step S3: Design a multi-objective reward function driven by a physics model, which integrates energy consumption, temperature, motion stability, carbon emissions, and physical constraints into the optimization framework. Step S4: In the offline pre-training stage, the basic security strategy is trained using the conservative Q-learning algorithm by utilizing historical running data and the state space, action space and reward function. Step S5: In the online security fine-tuning stage, the basic security policy is deployed to the actual system. Under multiple security constraints, the trust domain policy optimization algorithm is used to make small adjustments to the policy so that the policy can adapt to real-time operating conditions. Step S6: Deploy and control the optimized strategy online. Through progressive deployment and continuous learning mechanisms, achieve safe, adaptive and efficient operation of the data center HVAC system throughout its entire lifecycle.

[0028] Example 2: Construction of an interpretable global state space incorporating physical information like Figure 3 As shown in this embodiment, step S1 is specifically described, which involves constructing an interpretable global state space that integrates physical information.

[0029] First, based on the physical mechanisms of data center operation, raw sensor data is transformed into feature parameters with clear physical meaning to enhance the physical rationality and interpretability of state representation. This specifically includes the following key steps: S11. Calculation of System Sensible Heat Ratio (SHR) The sensible heat ratio of the system is calculated using data related to cooling capacity and humidity. Total cooling capacity. The latent cooling capacity is calculated from the chilled water flow rate and the supply and return water temperature difference. It was calculated from the change in air humidity.

[0030] The system sensible heat ratio is defined as: .

[0031] The sensible heat ratio (SHR) can characterize the proportion of sensible heat load to latent heat load in the current load, thus reflecting the humidity load level. Depending on the needs, the statistical characteristics of the SHR can be estimated based on historical data to obtain the confidence interval of the SHR, which can be used to characterize the uncertainty and volatility of the load type.

[0032] S12. Calculation of Virtual Load Factor (VLR) The virtual load factor reflects the proportion of the current system load relative to the rated capacity of the equipment. It is calculated as follows: In the formula, This refers to the specific heat capacity of chilled water. This refers to the mass flow rate of chilled water. To account for the temperature difference between supply and return water, This refers to the rated cooling capacity of the target equipment. This feature can identify whether the current system is under light load, partial load, or high load, providing load intensity information for strategic decision-making.

[0033] S13. The calculation of near-subcooling (ASC) of the system characterizes the degree of deviation of the refrigeration cycle operating point from the thermodynamic optimum. ASC is introduced to define this near-subcooling. In the formula, This is the condensation temperature. The evaporation temperature. This represents the theoretical optimal approximation. An ASC value close to 0 indicates that the operation is close to the optimal condition, while a larger deviation indicates a lower refrigeration cycle efficiency.

[0034] S14. Calculation of Equipment Overall Efficiency Index (EEI) To reflect the degradation of equipment performance, the Equipment Overall Efficiency Index (EEI) is introduced, defined as: In the formula, For the actual operating efficiency of the equipment, EEI is used to quantify the degree of performance degradation of equipment relative to its new state, based on design operating conditions.

[0035] S15. Construction of State Vectors By combining the aforementioned physical characteristics with some key raw monitoring data (such as typical cabinet inlet air temperature, chiller unit load rate, main loop flow rate, system power, etc.), a physically interpretable state vector is constructed. This state vector reflects both the system's thermal and humidity load characteristics and equipment performance status, while also taking into account the core operational information required for control.

[0036] Example 3: Cold Source-Terminal Joint Action Space and Coordinated Control System In this embodiment, step S2 in embodiment 1 is further optimized by defining the joint action space of the cold source side and the terminal side to achieve coordinated control at the system level.

[0037] Step S21. Cold source side action definition The cold source side includes equipment such as chillers, cooling towers, and water pumps. Its key control actions include, but are not limited to: chiller outlet water temperature setpoint, chilled water pump frequency or speed setpoint, cooling water pump frequency or speed setpoint, and cooling tower fan speed or on / off status.

[0038] By adjusting the above actions, the cooling capacity, heat exchange efficiency, and hydraulic conditions of the cold source system can be changed.

[0039] Step S22. Definition of End-of-Line Action The terminal side mainly includes the computer room air conditioning, air supply system, etc., and its key control actions include, but are not limited to: The setpoint of the air supply temperature of the computer room air conditioner, the opening degree of the chilled water valve at the end, and the speed or air volume adjustment of the terminal fan.

[0040] By adjusting the above actions, the supply air temperature, supply air volume, and terminal heat exchange capacity can be changed, thereby affecting the cabinet intake air temperature and the local environmental control effect.

[0041] Step S23. Construction of the joint action space By combining the control variables on the cold source side and the terminal side in a predetermined order, a joint action vector is formed, and the joint action space is the Cartesian product of all control action variables. Through a unified representation of the joint action space, the combined impact of the cold source and the terminal on overall energy efficiency and temperature control quality can be considered at the strategy level, avoiding the local optima caused by separate control of the cold source and the terminal in traditional methods.

[0042] Step S24. Cooperative Control Strategy: In the reinforcement learning strategy output phase, the agent outputs a joint action vector based on the current state vector, which includes control commands for all key devices on both the cold source and terminal sides. Through joint optimization, the cold source outlet water temperature, pump speed, terminal air supply temperature, valve opening, etc., can be comprehensively adjusted while meeting the cabinet temperature constraints, enabling the entire system to operate at a higher overall energy efficiency level.

[0043] The aforementioned joint action space and collaborative control mechanism ensure the linkage and coordination between the cold source system and the terminal air conditioning system, avoiding the situation where energy saving in a single subsystem leads to a decrease in overall energy efficiency.

[0044] Example 4: Design of a multi-objective reward function driven by a physics model like Figure 4 As shown, in this embodiment, step S3 in embodiment 1 is optimized to design a multi-objective reward function driven by a physical model, which is used to evaluate the quality of the control strategy.

[0045] The overall form of the multi-objective reward function: Multiple objectives, including energy consumption, temperature control, motion stability, carbon emissions, and physical information constraints, are weighted and combined. The overall form of the reward function R is as follows: In the formula, to This represents the weighting coefficient of each sub-reward, and the sum of the weights is 1.

[0046] The calculation methods for each sub-item reward are as follows: Energy consumption reward , In the formula, The total system power consumption can be further decomposed into P_chiller + P_pump + P_fan + P_tower, and verified through real-time power metering; therefore, This is the sum of the power of all related equipment, including chillers, water pumps, cooling towers, and computer room air conditioners. The higher the power, the heavier the penalty. Temperature compliance bonus reflects the deviation of the rack intake air temperature from the set value, and is defined as: ; In the formula This refers to the rack intake air temperature. To set the temperature, a soft threshold is introduced to progressively penalize temperature deviations approaching the upper limit. The larger the deviation, the lower the reward, ensuring that the temperature is controlled within a safe range.

[0047] Stability rewards are used to constrain the magnitude of changes in control commands, avoiding frequent and large-scale adjustments, and are defined as follows: ; In the formula For the current moment's action, This refers to the action performed at the previous moment.

[0048] Carbon emission rewards In the formula, CI represents the real-time carbon intensity.

[0049] Physical information reward item: To ensure the control strategy meets the physical rationality requirements of thermodynamics and hydraulics, a physical information reward item is introduced: In the formula, , , This represents the weighting coefficient for the sub-item.

[0050] The calculation methods for each sub-item of the Physical Information Award include: Theoretical COP Award. ,in ; Pipeline pressure loss penalty ,in f is the friction factor, L is the pipe length, D is the diameter, v is the velocity, and g is the gravity. Low loss coefficient, taking into account pipe roughness and velocity distribution; Heat exchanger pinch temperature penalty The multi-objective reward function serves as the global optimization objective function, used to minimize the total energy consumption of the system. To simultaneously meet temperature compliance and physical constraints, the optimization objective can be expressed as: In addition, weight It can be dynamically adjusted through Bayesian optimization or genetic algorithms.

[0051] By designing the multi-objective reward function described above, the strategy optimization not only focuses on energy consumption and temperature, but also takes into account control stability, carbon emissions and physical rationality, guiding the system to approach the overall optimal operating state.

[0052] Example 5: Implementation of the offline pre-training stage In this embodiment, based on implementation 1, step S4 is further optimized and explained, and the implementation method of the offline pre-training stage is given.

[0053] S41. Data Preprocessing: Clean and preprocess historical operational data, including outlier detection, missing data imputation, and normalization. Organize the raw monitoring data into a four-tuple of state-action-reward-next state according to time order to construct an offline dataset.

[0054] S42. Network Structure Design: Construct a deep neural network for fitting the Q-value function. The input is the state vector constructed in Example 2, and the output is the Q-value of each action in the joint action space. The network may include several hidden layers, using appropriate nonlinear activation functions.

[0055] S43. Conservative Q-Learning Algorithm: During offline training, the Conservative Q-Learning (CQL) algorithm is adopted. Its basic idea is to add a conservative regularization term to the standard Q-Learning loss function, lowering the Q-value of state-action pairs that are absent or rarely appear in the dataset. This suppresses overestimation of unknown regions by the value function, thereby improving the safety and robustness of the offline training strategy.

[0056] S44. Experience Replay and Parameter Update: Offline data is stored in an experience replay buffer, and training is performed using batch sampling. In each training batch, the gradient is calculated based on the objective function learned by conservative Q-learning, and the network parameters are updated using optimization algorithms (such as gradient descent methods). A target network can be set and updated periodically to improve training stability.

[0057] S45. Extraction of Basic Safety Strategy: After training convergence, a basic safety strategy is extracted from the Q-network. This could be, for example, selecting the action with the largest Q-value for a given state, or employing a soft-maximum strategy with a temperature parameter. This strategy serves as the initial control strategy for the online phase, ensuring its behavior is confined to the vicinity of historical safety data distribution.

[0058] Through the above steps, the offline pre-training phase learns a set of basic control strategies that perform well within a safe range using historical data without interfering with the actual system operation, laying the foundation for subsequent online safety fine-tuning.

[0059] Further preferably, the 3σ principle is used to identify outlier data, time series interpolation is used to fill in missing values, and minimum-maximum normalization is used to map the data to the [0,1] interval. Next is the network structure design. A deep Q-network is constructed, consisting of an input layer, three hidden layers, and an output layer. The input layer dimension is consistent with the state space dimension, the number of neurons in the hidden layers are 256, 512, and 256 respectively, and the output layer dimension is consistent with the action space dimension. The ReLU activation function is used, and the linear activation function is used in the output layer. Then, a conservative Q-learning algorithm is implemented. A conservative regularization term is added to the standard Q-learning loss function. The specific loss function is as follows: Here, α is a conservative coefficient, set to 0.5, to effectively prevent overestimation of the value of unseen state-action pairs. This can be further mitigated by combining with a dual-Q network. The training process is then optimized. A priority-based empirical replay mechanism is employed, determining sampling priority based on the magnitude of the temporal difference error.

[0060] Sampling priority is determined based on the magnitude of the time difference error. In the formula, Let ε be the TD error and ε be a small constant. The Adam optimizer is used for parameter updates, with a learning rate set to 0.001. The target network parameters are updated every 1000 training steps to ensure training stability. Finally, policy extraction is performed. A deterministic policy is obtained by maximizing the action-value function, or a stochastic policy is obtained through the Boltzmann distribution. Policy performance is evaluated on the validation set, and the policy that performs best in terms of cumulative reward, temperature compliance rate, and energy consumption is selected.

[0061] Example 6: Implementation of the Online Security Fine-Tuning Phase like Figure 5 As shown, in this embodiment, step S5 in embodiment 1 is further optimized and explained, and the implementation method of the online security fine-tuning stage is given.

[0062] S51. Safety Constraint Design: During the online phase, to ensure the safe operation of the system, multiple safety constraints are set for control actions and system states, including but not limited to: limiting the change in control actions between adjacent time periods to no more than a preset ratio; limiting the intake air temperature of critical cabinets to no more than the upper safety limit or lower safety limit; and limiting the power of major equipment to no more than a certain percentage of the rated power.

[0063] When the actions output by the policy violate the above constraints, the actions can be pruned or the security policy can be triggered to take over.

[0064] S52. Trust Domain Policy Optimization: Under the premise of satisfying security constraints, a policy optimization method with trust domain constraints is adopted to fine-tune the basic security policy online. By imposing constraints on the differences between the old and new policies under the same state (e.g., limiting the KL divergence to no more than a preset threshold), the step size of each policy update is controlled, avoiding excessively large policy changes and ensuring the continuity and stability of system operation.

[0065] S53. Online Data Acquisition and Incremental Update: During system operation, new state-action-reward-next state samples are collected in real time and appended to the online data buffer. At a certain frequency, or when performance improvement is detected, the online data is used to incrementally update the strategy, allowing the strategy to gradually adapt to the latest operating conditions and equipment status.

[0066] S54. Learning rate and update frequency adjustment: Adaptively adjust the learning rate and update frequency according to changes in policy performance: When performance continues to improve after several updates, the learning rate or update frequency can be appropriately increased to accelerate convergence; when performance fluctuates or deteriorates temporarily, the learning rate or update frequency can be reduced to enhance training stability.

[0067] S56. Security Monitoring and Rollback Mechanism: During the online phase, key performance indicators such as energy consumption level, temperature compliance rate, and control stability are continuously monitored. When a significant decline in policy performance is detected or a security threshold is triggered, a rollback mechanism can be activated to restore the policy to a better historical version, while simultaneously recording abnormal data for subsequent offline analysis and retraining.

[0068] Through the above design, the online safety fine-tuning stage achieves a balance between the adaptive capability of reinforcement learning and the safety and reliability of industrial systems, enabling the control strategy to be continuously optimized without sacrificing safety margins.

[0069] To further optimize the system, a triple safety guarantee mechanism is established: motion space constraints limit the change in motion between adjacent moments to no more than 20%; state space constraints ensure that the temperature in critical areas does not exceed 28°C; and performance constraints prevent equipment from operating under overload conditions, with power not exceeding 90% of the rated value.

[0070] Secondly, trust domain policy optimization is employed. A trust domain policy optimization method is used to limit the policy update magnitude, ensuring that the KL divergence between the new and old policies does not exceed a set threshold δ=0.01. The optimization problem is formulated as follows: Then, adaptive learning rate adjustment is implemented. The learning rate is dynamically adjusted based on changes in policy performance, with an initial learning rate set at 0.0001. When policy performance continues to improve, the learning rate is appropriately increased; when performance fluctuates, the learning rate is decreased to ensure training stability. Finally, a safety monitoring mechanism is implemented. Key system parameters are monitored in real time, and a multi-level early warning system is established. When conditions are detected, such as temperature exceeding safety thresholds in critical areas, device power consumption exceeding rated values, or control action changes exceeding limits, the safety controller is immediately triggered to ensure system safety. Example 7: Strategy Deployment and Online Control In this embodiment, based on implementation 1, step S6 is further optimized and explained, and the implementation method of strategy deployment and online control is given.

[0071] S61. Strategy Verification: Before deploying in the real system, the trained strategy is first verified in a simulation environment to check its performance under different load levels, outdoor weather conditions, and equipment states. After successful simulation verification, the strategy is run in the actual system in shadow mode, meaning the strategy calculates control commands but does not directly act on the equipment; it is only used for performance comparison analysis with the existing control scheme.

[0072] S62. Gradual Deployment: After verifying the performance of the strategy in shadow mode to meet expectations, a gradual deployment approach is adopted: In the first stage, the new strategy is enabled only in certain time periods or in certain data center areas, while the existing control strategy is still used in the remaining time periods or areas; in the second stage, the control scope and time percentage of the new strategy are gradually expanded; in the third stage, after verifying the stability and advantages of the new strategy in long-term operation, the system is switched to the new strategy for global control.

[0073] This phased deployment approach can effectively reduce the risks associated with launching new strategies.

[0074] S63. Continuous Learning and Performance Monitoring: After strategy deployment, continuously monitor changes in system energy consumption indicators, temperature compliance rate, equipment operating status, and strategy output actions. When significant changes are detected in system operating conditions, load structure, or equipment performance, a new offline training cycle or online fine-tuning process can be triggered to continuously update the control strategy and ensure that system performance remains at an optimal level over the long term.

[0075] S64. Human-Machine Collaboration and Anomaly Handling: Provides a visualization and interpretation interface for control strategies, presenting key physical characteristics and strategy decision-making basis to operations and maintenance personnel in a graphical manner for easy human understanding and review. When abnormal operating conditions occur or strategy outputs do not conform to common engineering sense, operations and maintenance personnel can intervene to adjust the control strategy or switch to a safe mode (such as rolling back to the traditional control strategy) to ensure the safe operation of the data center HVAC system.

[0076] Through the above-mentioned deployment and online control mechanism, the control strategy provided by this invention can be reliably implemented in engineering practice and maintain safety, stability and high energy efficiency during long-term operation.

[0077] Example 8: A Global Temperature Control Optimization Method for Data Centers Based on Online-Offline Hybrid Reinforcement Learning like Figure 1 As shown, an embodiment of the present invention provides a global temperature control optimization method for data centers based on online-offline hybrid reinforcement learning. This method aims to achieve safe, efficient, and adaptive system control through a systematic offline training and online optimization process. By employing physical information fusion and a hybrid learning framework, this method addresses the poor interpretability and insufficient adaptability of traditional methods, improving the energy efficiency of temperature control systems. It is applicable to data centers and other similar thermal systems. The core steps are as follows: Figure 1 The overall system architecture diagram is shown below, including the following steps: Step S1: Construct an interpretable global state space that incorporates physical information; Step S2: Define the joint action space; Step S3: Design a multi-objective reward function driven by a physics model; Step S4: Offline pre-training phase; Step S5: Online security fine-tuning phase; Step S6: Strategy deployment and online control.

[0078] In this embodiment, as Figure 3 As shown in the state space construction flowchart, the system collects raw data through sensors, calculates physical characteristics, and forms an interpretable state space. Figure 4 The reward function calculation diagram is shown below. Multi-objective rewards integrate energy consumption, temperature, and physical parameters to ensure optimization rationality. For example... Figure 5 The offline-online combined training flowchart is shown, which uses historical data to train the basic strategy.

[0079] Based on step S1, an interpretable global state space incorporating physical information is constructed, specifically including the following sub-steps: Step S101: Deploy high-precision sensors at key locations in the data center temperature control system to collect raw data in real time, including physical quantities such as temperature, pressure, flow rate, and power consumption. The data collection frequency is 5 minutes to ensure that the data captures system dynamics without placing excessive burden on the system. The sensors include a PT100 temperature sensor, a differential pressure transmitter, and an electromagnetic flow meter.

[0080] Step S102: Preprocess the collected data, including missing value imputation, standardization, and anomaly detection, to generate a structured data vector. Missing values ​​are imputed using linear interpolation: x(t) = Standardization uses mean-standard deviation normalization: = Where μ and σ are the mean and standard deviation of feature x, respectively; anomaly detection uses the Z-score method: If |Z|>3, then replace it with the most recent valid value. The preprocessed data vector is defined as s={ , where n is the feature dimension. The historical dataset contains 1000 sets of working conditions, which are used for training.

[0081] Step S103: Calculate characteristic quantities based on the physical model. Taking the system's sensible heat ratio as an example, the specific calculation process is as follows: obtain the cold water flow rate m and the supply and return water temperature difference through a cold water flow meter and a temperature sensor. Calculate the total cooling capacity Where c is the specific heat capacity of water (4.184 kJ / kg·K); the latent cooling capacity is calculated by acquiring changes in air humidity using a humidity sensor. ,in Where is the air mass flow rate, and w is the humidity ratio. The latent heat of vaporization is 2257 kJ / kg; the final calculation... And add variance estimation To quantify uncertainty.

[0082] Step S104: Verify the rationality of the feature data. Set a reasonable range check for each feature (e.g., SHR∈[0.7,1.0]). When the feature value exceeds the range, the data review process is automatically triggered. All feature data is stored in a time-series database, supporting real-time querying and historical analysis.

[0083] Step S2: Define the joint action space, which includes the following sub-steps: Step S201: Establish the cold source side actions, including the chiller outlet water temperature setpoint; cooling water pump frequency setpoint; and cooling tower fan speed. Adaptively adjust the discrete level according to the ambient temperature.

[0084] Step S202: Establish terminal-side actions, including the setpoint of the air conditioning supply temperature in the computer room; water valve opening; and fan speed adjustment. Apply these actions in different zones to match the load of different computer rooms.

[0085] Step S203: Discretize the action space, where the size of the action space is |A| = |A_cold source|·|A_terminal|≈10^4. Manage the complexity through factorization and ensure that the action changes satisfy the constraints. .

[0086] Step S3, design a multi-objective reward function driven by a physics model, which includes the following sub-steps: Step S301: Calculate the basic reward items, including ,in ; (a, b, c are fitting coefficients); , Step S302: Calculate the physical information reward item, including ,in (Temperature is in Kelvin), corrected to , f is the friction factor, L is the pipe length, D is the diameter, v is the velocity, and g is the gravity. Low loss coefficient; .

[0087] Step S303: Merge reward values. The weight The cumulative reward is maximized through Bayesian optimization; each term is standardized using the min-max method to ensure the range [0,1].

[0088] Step S304: Dynamically adjust the weights, for example, increase them during the high-temperature period of summer. Prioritize temperature compliance and integrate external weather data for forecasting adjustments. This reward function can be extended to device lifetime prediction, improving long-term optimization.

[0089] Step S4, the offline pre-training stage, specifically includes the following sub-steps: Step S401: Data preparation. Extract data from historical databases, perform cleaning and feature engineering, and divide the data into training set (70%), validation set (15%) and test set (15%), stratified according to load level.

[0090] Step S402: Model training, initialize the deep Q-network (input layer dimension = state dimension, hidden layer 256-512-256, activation...) (dropout=0.2), using mini-batch gradient descent with 512 data points per batch, and prioritizing empirical replay. ,in For TD error, Small constant.

[0091] Step S403: Loss function calculation, α=0.5, τ=1.0; a double Q network is used to prune overestimations. The training curve is monitored to stop training early. Step S404: Model evaluation, calculate cumulative reward on the validation set, prevent overfitting through early stopping mechanism, stop when the validation loss has no improvement for 20 consecutive epochs; monitor overfitting by the training-validation loss difference being <10%.

[0092] Step S405: Policy extraction, π(a|s)=exp(Q(s,a) / τ) / ∑exp(Q(s,a') / τ); finally, the Pareto front multi-objective optimization is selected.

[0093] Step S5, the online security fine-tuning stage, specifically includes the following sub-steps: Step S501: Safety constraint check, motion constraints State constraints Performance constraint: power ≤ 0.9 kW (rated value), determined by projection. Forced execution. This step ensures that the update does not violate physical limits, combined with real-time simulation. Predict potential risks.

[0094] Step S502: Policy update, using Trust Domain Policy Optimization (TRPO) to optimize the issue table. Solving the surrogate target by approximation using conjugate gradients ,in Let be the probability ratio, and A be the advantage function. This update balances exploration and stability, with an initial γ = 0.99.

[0095] Step S503: Adjust the learning rate dynamically based on changes in policy performance. , where ζ=0.1, For performance changes (based on cumulative reward changes); use cosine annealing. As an alternative, this ensures a balance between convergence speed and stability.

[0096] Step S504: Performance monitoring, real-time calculation of immediate rewards and key metrics such as temperature deviation. When performance degrades, a rollback to the previous model is triggered; multi-level early warning systems are established, and the safety controller immediately intervenes if device overload is detected. This mechanism supports incremental learning, collecting new data every 5 minutes to adjust parameters.

[0097] Step S6, Policy Deployment and Online Control, specifically includes the following sub-steps: Step S601: Strategy verification, calculate energy efficiency savings rate Temperature compliance rate The benchmark and PID were compared using a paired t-test. Robustness was verified in multi-condition simulations, such as load surges and sensor failure scenarios.

[0098] Step S602: Progressive Deployment, Shadow Mode Computing The transition is controlled by KPI thresholds (such as shadow accuracy > 95%), compared with the baseline.

[0099] Step S603: Continuous learning, using a circular buffer to store 10^6 samples, retraining when performance drops by more than 5%, and drift detection. It also integrates federated learning to share knowledge across multiple data centers.

[0100] Step S604: Human-machine collaboration, using SHAP values ​​to visualize feature importance. , where f is the model output; the interface includes a counterfactual analysis "what-if" simulation.

[0101] Step S605: Exception handling, missing data is filled with historical averages, and the default strategy is switched when the model is abnormal; the differential pressure setpoint, temperature error and running parameters of each cycle are recorded in the log and saved as a CSV file.

[0102] This invention achieves global optimization of data center temperature control systems through online-offline hybrid reinforcement learning and physical information fusion, significantly improving energy efficiency and safety. The method has wide applicability and can promote the green, energy-saving and intelligent development of the data center field.

[0103] Example 9: A Global HVAC Optimization System for Data Centers Based on Online-Offline Hybrid Reinforcement Learning like Figure 2 As shown, this embodiment provides a data center global HVAC optimization system based on online-offline hybrid reinforcement learning. The system includes: The main body of the data center HVAC system is used to provide cooling and maintain the ambient temperature of the data center; An optimized control device that is communicatively connected to the HVAC system is used to execute each step of the data center global temperature control optimization method based on hybrid reinforcement learning described in Embodiments 1 to 8 of the present invention.

[0104] The optimized control device is configured to: collect the operating data of the data center HVAC system and convert it into a state space that integrates physical information; perform offline pre-training and online safety fine-tuning based on the joint action space of the cold source side and the terminal side and a multi-objective reward function to generate an optimized control strategy; and issue control commands to equipment such as chillers, water pumps, cooling towers and computer room air conditioners according to the control strategy to achieve global optimized control of the data center HVAC system.

[0105] Example 10. Electronic device This embodiment provides an electronic device for executing the data center global temperature control optimization method based on hybrid reinforcement learning as described in Embodiments 1 to 8 of the present invention.

[0106] The electronic device may include a processor and a memory, the memory storing a computer program that can run on the processor, the processor being configured to: Collect and preprocess operational data of HVAC systems in data centers to construct an interpretable state space that integrates physical information; Define the joint action space of the cold source side and the terminal side, and evaluate the strategy based on a multi-objective reward function; During the offline pre-training phase, conservative Q-learning is performed using historical running data to obtain the basic security policy. During the online security fine-tuning phase, the basic security strategy is progressively updated under multiple security constraints; Based on the updated control strategy, control commands are issued to relevant equipment in the data center HVAC system to achieve global optimized control of the data center HVAC system.

[0107] Example 11: Computer-readable storage medium This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by the processor of an electronic device, it implements the data center global temperature control optimization method based on hybrid reinforcement learning as described in Embodiments 1 to 8 of this invention.

[0108] When the computer program is executed, it is specifically used to: collect operating data of the data center HVAC system and construct a state space that integrates physical information; define a joint action space between the cold source side and the terminal side; perform offline pre-training and online safety fine-tuning based on a multi-objective reward function to generate or update control strategies; and issue control commands to relevant equipment in the data center HVAC system according to the control strategies, thereby realizing global optimization and continuous adaptive control of the data center HVAC system.

[0109] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A data center global temperature control optimization method based on hybrid reinforcement learning, characterized in that, Includes the following steps: Step S1. Construct an interpretable global state space that integrates physical information: Based on the operating physical mechanism of the data center HVAC system, the raw sensor data is converted into state characteristic variables with clear physical meaning, including the system sensible heat ratio, virtual load rate, system near subcooling degree, and equipment comprehensive efficiency index; Step S2. Define the joint action space of the cold source side and the terminal side: Establish a collaborative control system that includes the control actions of the cold source side equipment and the control actions of the terminal side equipment. The cold source side actions include the chiller unit outlet water temperature setpoint, the cooling water pump frequency setpoint and the cooling tower fan speed. The terminal side actions include the air supply temperature setpoint of the computer room air conditioner, the water valve opening degree and the fan speed adjustment amount. Step S3. Design a multi-objective reward function driven by a physical model: Taking into account the energy consumption, temperature compliance, control action stability, carbon emissions, and physical information constraints of the data center HVAC system, construct a global optimization objective function that includes multiple sub-rewards; Step S4. Offline pre-training stage: Train the initial control policy using historical running data, and use the conservative Q-learning algorithm to introduce a conservative regularization term in the objective function to prevent overestimation of the value function and ensure that the trained policy is within the range of historical safe behavior. Step S5. Online security fine-tuning stage: Under multiple security constraints, the offline strategy is progressively optimized based on real-time interactive data. The trust domain strategy optimization algorithm is used to limit the policy update range to ensure the stability of the policy update and make the policy adapt to real-time operating conditions. Step S6. Strategy Deployment and Online Control: Deploy the trained and optimized control strategy to the data center HVAC system to achieve real-time closed-loop control of the cold source and terminal equipment, and establish a continuous learning mechanism.

2. The data center global temperature control optimization method based on hybrid reinforcement learning according to claim 1, characterized in that, In step S1, the construction of the interpretable global state space includes: calculating the system sensible heat ratio: based on the total cooling capacity and latent cooling capacity of the data center HVAC system, using the formula... Calculate, where This represents the total cooling capacity. Latent cooling capacity; Virtual load factor calculation: Based on the principle of thermodynamic energy balance, using the formula... Calculations are performed, where c is the specific heat capacity of water, m is the mass flow rate of chilled water, ΔT is the temperature difference between the supply and return water, and Prated is the rated capacity of the equipment; the system's near subcooling is calculated based on the thermodynamic characteristics of the refrigeration cycle, using the formula... Calculation, where This is the condensation temperature. The evaporation temperature. The theoretical optimal approximation; Calculation of the overall equipment efficiency index: based on the equipment performance curve, using the formula... Calculation, where For the actual operating efficiency of the equipment, For design operating conditions efficiency.

3. The data center global temperature control optimization method based on hybrid reinforcement learning according to claim 1, characterized in that, In step (3), the multi-objective reward function consists of multiple weighted sub-rewards; The sub-rewards include: an energy consumption reward R that reflects the total energy consumption of the system. energy Temperature compliance reward R, reflecting temperature control deviation temperature Reward R reflects the stability of the controlled action and the range of change in the action. stability Carbon emission incentives R that reflect the impact of carbon emissions carbon And the reward R based on physical information physics The total reward value is obtained by linearly combining each sub-reward multiplied by its corresponding weight coefficients w1, w2, w3, w4, and w5.

4. The data center global temperature control optimization method based on hybrid reinforcement learning according to claim 1, characterized in that, In step S4, the offline pre-training stage uses historical running data to train the basic security policy. The algorithm used is the conservative Q-learning algorithm, which avoids overestimation of value by introducing a conservative regularization term into the value objective, thereby ensuring that the trained control policy is reliable and effective within the range of historical security behavior.

5. The data center global temperature control optimization method based on hybrid reinforcement learning according to claim 1, characterized in that, In step S5, the online security fine-tuning stage optimizes and adjusts the initial control strategy in real time while meeting preset security constraints. The trust domain strategy optimization algorithm is used to limit the magnitude of each strategy update to ensure the stability of the strategy improvement process and enable the optimized control strategy to adapt to the real-time operational changes of the data center HVAC system.

6. The data center global temperature control optimization method based on hybrid reinforcement learning according to claim 1, characterized in that, It also includes data acquisition and preprocessing steps: sensors are deployed at key locations in the data center HVAC system to collect raw operating data containing parameters such as temperature, pressure, flow rate and power consumption in real time, and the raw data is cleaned, standardized and anomaly detected.

7. The data center global temperature control optimization method based on hybrid reinforcement learning according to claim 1, characterized in that, It also includes an anomaly handling mechanism: when sensor data is detected to be missing or abnormal, the system uses predicted values ​​based on historical data or default control strategies to maintain the normal operation of the data center HVAC system.

8. A global temperature control optimization system for data centers based on hybrid reinforcement learning, characterized in that, The data center global temperature control optimization method based on hybrid reinforcement learning according to any one of claims 1-7 includes: a data acquisition and processing module for real-time acquisition of sensor data at key locations of the data center HVAC system and processing of the acquired raw data to calculate physical information features; a physical feature calculation module for calculating state feature quantities such as the system sensible heat ratio and virtual load rate of the data center HVAC system based on a physical model; an offline training module for offline training of the control strategy using the historical operating data to obtain a basic safety strategy; an online learning module for online fine-tuning and optimization of the basic safety strategy under the condition of meeting safety constraints; a control execution module for converting the optimized control strategy into control signals for each controlled device in the data center HVAC system; and a safety monitoring module for real-time monitoring of the operating status of the data center HVAC system and triggering preset protection measures to ensure system operation safety when an anomaly occurs.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the data center global temperature control optimization method based on hybrid reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program can implement the data center global temperature control optimization method based on hybrid reinforcement learning as described in any one of claims 1-7.

Images