A central air conditioning cold source side energy efficiency optimization method and device

Abstract

This invention discloses a method and apparatus for optimizing the energy efficiency of a central air conditioning system's cold source side. By constructing a Transformer-XL state transition model, a safe and efficient offline training environment is provided for reinforcement learning, avoiding the energy waste and safety risks associated with online training. A truncated quantile critic algorithm with an integrated action extension mechanism is constructed to effectively suppress Q-value overestimation bias under dynamic loads, while fully exploring the system's energy-saving potential. A reward function designed with energy efficiency as its core achieves multi-objective collaborative optimization, including temperature control and efficient equipment operation. The optimal scheduling strategy obtained through offline training improves the energy efficiency ratio of the cold source side system, reduces average power consumption, and ensures temperature control accuracy meets engineering requirements. Furthermore, the strategy exhibits strong robustness to the design details of the reward function, solving the technical problems of poor reinforcement learning training security, susceptibility to control fluctuations or failures, and weak multi-objective coordination capabilities in existing central air conditioning cold source side energy efficiency optimization methods.

Description

Technical Field

[0001] This invention relates to the field of energy efficiency optimization technology, and in particular to a method and apparatus for optimizing energy efficiency on the cold source side of a central air conditioning system. Background Technology

[0002] Traditional cooling source-side scheduling methods, such as PID control (proportional-integral-derivative control), have significant limitations: fixed control parameters cannot adapt to dynamic operating environments, easily leading to "overcooling" or "undercooling" phenomena. Furthermore, they often prioritize the stability of a single parameter, lacking overall optimization of energy consumption, comfort, and equipment lifespan, resulting in low system energy efficiency. While Model Predictive Control (MPC) demonstrates superior performance to PID control in cooling source-side control, it relies on accurate mechanistic models and substantial computational resources, making it difficult to apply on a large scale in complex global optimization of cooling sources.

[0003] Reinforcement learning (RL) offers a novel approach to the dynamic scheduling optimization of central air conditioning systems. By learning optimal strategies through interaction with the environment, it demonstrates significant energy-saving potential for HVAC systems. Existing research on the application of reinforcement learning on the cooling source side, such as combining Q-learning with LQR to optimize variable air volume systems and using DDPG to achieve demand response and energy efficiency management in central air conditioning, has verified its effectiveness in maintaining comfort while reducing energy consumption. However, the application of current reinforcement learning methods on the cooling source side still faces the following challenges:

[0004] 1. Insufficient training efficiency and security: Model-free reinforcement learning algorithms often rely on online interactive training with the real environment, resulting in long convergence cycles, high energy consumption during the exploration phase, and large fluctuations in control performance, making them difficult to apply directly to engineering practice;

[0005] 2. Poor generalization ability in dynamic scenarios: Under dynamic loads, the algorithm is prone to overestimating the Q value and making overly optimistic decisions when historical data is not covered, which can lead to control fluctuations or failures. There is insufficient research on robustness and security mechanisms in continuous control scenarios.

[0006] 3. Weak multi-objective coordination capability: Most algorithms overemphasize the single objective of "energy saving", reducing energy consumption by reducing cooling capacity or widening the temperature control range. This leads to the temperature of the chilled water return main pipe deviating from the set value and the indoor comfort decreasing. It cannot simultaneously meet the core engineering requirements of improving energy efficiency ratio and precise temperature control.

[0007] Therefore, providing a cold source-side energy efficiency optimization method that balances training safety, dynamic generalization, and multi-objective coordination, while improving the system energy efficiency of the cold source side under the premise of ensuring temperature control accuracy and equipment operation stability, and at the same time making the learned scheduling strategy robust and adaptable to the deployment needs of actual engineering, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] This invention provides a method and apparatus for optimizing the energy efficiency of a central air conditioning cold source side, which solves the technical problems of poor reinforcement learning training security, easy to cause control fluctuations or failures, and weak multi-objective coordination ability in existing central air conditioning cold source side energy efficiency optimization methods.

[0009] In view of this, the first aspect of the present invention provides a method for optimizing the energy efficiency of a central air conditioning system on the cold source side, comprising:

[0010] The scheduling optimization problem of the central air conditioning energy side is modeled as a Markov decision process, which is defined as a quintuple consisting of a state space, an action space, a reward function, a state transition probability, and a discount factor.

[0011] A state transition model based on Transformer-XL is constructed and used as a high-fidelity offline training environment for reinforcement learning. The state transition model based on Transformer-XL is used to accurately predict the state at the next moment based on the current state and the current action, and to fit the state transition probability.

[0012] A truncated quantile critic algorithm with an integrated action expansion mechanism is constructed. The truncated quantile critic algorithm with an integrated action expansion mechanism includes priority experience replay, N-step reward, network architecture optimization, targeted exploration of noise, and penalty-free action expansion mechanism.

[0013] Establish a reward function with energy efficiency as the core, which includes temperature control reward, energy efficiency ratio improvement reward, equipment high-efficiency operation reward, system coordination reward, energy consumption suppression reward and operation stability reward;

[0014] Based on the truncated quantile critic algorithm with Transformer-XL state transition model and fusion action extension mechanism, the agent is trained offline using the reward function, so that the agent learns the optimal energy efficiency scheduling strategy on the cold source side.

[0015] The optimal energy efficiency scheduling strategy output by the trained agent is deployed to the control system on the cooling source side of the central air conditioning system, and the scheduling instructions are output to optimize the energy efficiency of the cooling source side of the central air conditioning system.

[0016] Optionally, the parameters in the state space include outdoor temperature, outdoor humidity, chilled water return main temperature, chilled water supply main temperature difference, instantaneous flow rate of the flow meter, chiller power, chilled water pump power, cooling water pump power, and the power of the two cooling towers.

[0017] The parameters of the action space include the chilled water temperature setpoint of the main unit, the chilled water pump frequency feedback, the cooling water pump frequency feedback, and the frequency feedback of the two cooling towers.

[0018] Optionally, the process of constructing a state transition model based on Transformer-XL includes:

[0019] Collect actual operation monitoring data of the central air conditioning cold source side for a preset time period, and divide the actual operation monitoring data into training set and test set;

[0020] The structure of the Transformer-XL state transition model consists of an input projection layer, a relative position encoding module, stacked Transformer-XL blocks, and an output projection layer.

[0021] The Transformer-XL state transition model is trained using the training set. The performance of the Transformer-XL state transition model is evaluated using mean squared error, mean absolute error, and coefficient of determination. The generalization ability of the Transformer-XL state transition model is verified using the test set, resulting in the trained Transformer-XL state transition model.

[0022] Optionally, the reward function is:

[0023] ;

[0024] in, The total reward of the reward function. As a reward for temperature control, Incentives for improving energy efficiency ratio Incentives for efficient operation of chilled water pumps Incentives for efficient operation of cooling water pumps To coordinate rewards within the system, As a power penalty, As a penalty for smooth operation, a reward is given for efficient operation of the chilled water pump. Rewards for efficient operation of cooling water pumps Both serve as rewards for efficient equipment operation and as power penalties. Energy consumption suppression reward, motion smoothness penalty This is a reward for operational stability.

[0025] Optionally, based on the truncated quantile critic algorithm using the Transformer-XL state transition model and fused action extension mechanism, the agent is trained offline using a reward function, enabling the agent to learn the optimal energy efficiency scheduling strategy on the cold source side, including:

[0026] Initialize the actor network, critic network, and target critic network of the truncated quantile critic algorithm for the fusion action extension mechanism, and keep the parameters of the target critic network consistent with those of the critic network;

[0027] Initialize the priority experience replay buffer, the N-step buffer, and the exploration noise; set the core hyperparameters of the algorithm.

[0028] Perform 2500 training iterations, with 200 steps per iteration. At the beginning of each iteration, reset the offline training environment to obtain the initial state and reset the N-step buffer until the training ends.

[0029] After each iteration, the exploration noise is attenuated by a noise attenuation factor of 0.999, and the importance sampling coefficient is updated until the total reward stabilizes at 7.0 × 10⁻⁶. 6 ~7.4×10 6 Within the interval, the algorithm converges, and the trained actor network is obtained as the optimal energy efficiency scheduling strategy, enabling the agent to learn the optimal energy efficiency scheduling strategy on the cold source side.

[0030] Optionally, the discount factor can be set to 0.99.

[0031] A second aspect of the present invention provides a central air conditioning cold source side energy efficiency optimization device, comprising:

[0032] The problem transformation unit is used to model the scheduling optimization problem of the central air conditioning energy side as a Markov decision process. The Markov decision process is defined as a quintuple consisting of a state space, an action space, a reward function, a state transition probability, and a discount factor.

[0033] The modeling unit is used to build a state transition model based on Transformer-XL. The state transition model based on Transformer-XL is used as a high-fidelity offline training environment for reinforcement learning. The state transition model based on Transformer-XL is used to accurately predict the state at the next moment based on the current state and the current action, and to fit the state transition probability.

[0034] Algorithm building unit: used to build the truncated quantile commentator algorithm with integrated action expansion mechanism. The truncated quantile commentator algorithm with integrated action expansion mechanism includes priority experience replay, N-step reward, network architecture optimization, directed exploration noise and penalty-free action expansion mechanism;

[0035] The reward function construction unit is used to establish a reward function with energy efficiency as the core. The reward function includes temperature control reward, energy efficiency ratio improvement reward, equipment high-efficiency operation reward, system coordination reward, energy consumption suppression reward and operation stability reward.

[0036] The training unit is used to train the agent offline using the reward function based on the truncated quantile critic algorithm based on the Transformer-XL state transition model and the fusion action extension mechanism, so that the agent learns the optimal energy efficiency scheduling strategy on the cold source side.

[0037] The output unit is used to deploy the optimal energy efficiency scheduling strategy output by the trained agent to the control system on the cooling source side of the central air conditioning system, and outputs scheduling instructions to optimize the energy efficiency of the cooling source side of the central air conditioning system.

[0038] Optionally, the parameters in the state space include outdoor temperature, outdoor humidity, chilled water return main temperature, chilled water supply main temperature difference, instantaneous flow rate of the flow meter, chiller power, chilled water pump power, cooling water pump power, and the power of the two cooling towers.

[0039] The parameters of the action space include the chilled water temperature setpoint of the main unit, the chilled water pump frequency feedback, the cooling water pump frequency feedback, and the frequency feedback of the two cooling towers.

[0040] Optionally, the process of constructing a state transition model based on Transformer-XL includes:

[0041] Collect actual operation monitoring data of the central air conditioning cold source side for a preset time period, and divide the actual operation monitoring data into training set and test set;

[0042] The structure of the Transformer-XL state transition model consists of an input projection layer, a relative position encoding module, stacked Transformer-XL blocks, and an output projection layer.

[0043] The Transformer-XL state transition model is trained using the training set. The performance of the Transformer-XL state transition model is evaluated using mean squared error, mean absolute error, and coefficient of determination. The generalization ability of the Transformer-XL state transition model is verified using the test set, resulting in the trained Transformer-XL state transition model.

[0044] Optionally, the reward function is:

[0045] ;

[0046] in, The total reward of the reward function. As a reward for temperature control, Incentives for improving energy efficiency ratio Incentives for efficient operation of chilled water pumps Incentives for efficient operation of cooling water pumps To coordinate rewards within the system, As a power penalty, As a penalty for smooth operation, a reward is given for efficient operation of the chilled water pump. Rewards for efficient operation of cooling water pumps Both serve as rewards for efficient equipment operation and as power penalties. Energy consumption suppression reward, motion smoothness penalty This is a reward for operational stability.

[0047] As can be seen from the above technical solutions, the light-load control method for a symmetrical LCC-type bidirectional wireless power transfer system for battery energy storage provided by the present invention has the following advantages:

[0048] The energy efficiency optimization method for the cold source side of central air conditioning provided by this invention constructs a high-fidelity Transformer-XL state transition model, providing a safe and efficient offline training environment for reinforcement learning, fundamentally avoiding the energy waste and safety risks of online training. It proposes a truncated quantile critic algorithm that integrates an action expansion mechanism, especially the penalty-free action expansion mechanism, which effectively suppresses the overestimation bias of Q-values ​​under dynamic loads while fully exploring the system's energy-saving potential. The reward function is designed with energy efficiency as the core, achieving multi-objective collaborative optimization of temperature control, efficient equipment operation, and other objectives. The optimal scheduling strategy obtained through offline training improves the energy efficiency ratio of the cold source side system by approximately 104.4%, reduces average power consumption by approximately 42.2%, and meets engineering requirements for temperature control accuracy. Furthermore, the strategy exhibits strong robustness to the design details of the reward function, significantly improving the reliability and practicality of engineering deployment. This solves the technical problems of poor reinforcement learning training security, susceptibility to control fluctuations or failures, and weak multi-objective coordination capabilities in existing central air conditioning cold source side energy efficiency optimization methods. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a method for optimizing the energy efficiency of a central air conditioning system on the cold source side, as provided in an embodiment of the present invention.

[0051] Figure 2This is a schematic diagram of the TQC algorithm network architecture provided in an embodiment of the present invention;

[0052] Figure 3 This is a flowchart of the TQC algorithm training sub-process according to an embodiment of the present invention;

[0053] Figure 4 The box plot and probability density histogram of the chilled water pump frequency distribution in the early, middle and late phases during the training process of this embodiment of the invention are shown.

[0054] Figure 5 This is a schematic diagram of the structure of a central air conditioning cold source side energy efficiency optimization device provided in an embodiment of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0057] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0058] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0059] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0060] For easier understanding, please refer to Figure 1 This invention provides an embodiment of a method for optimizing the energy efficiency of a central air conditioning system on the cold source side, comprising:

[0061] Step 101: Model the scheduling optimization problem of the central air conditioning energy side as a Markov decision process. Define the Markov decision process as a quintuple consisting of a state space, an action space, a reward function, a state transition probability, and a discount factor.

[0062] It should be noted that the scheduling optimization problem of the central air conditioning energy side is modeled as a Markov Decision Process (MDP). A Markov Decision Process is defined as a quintuple consisting of a state space S, an action space A, a reward function R, state transition probabilities P, and a discount factor γ. .

[0063] The parameters of the state space S include outdoor temperature, outdoor humidity, chilled water return main pipe temperature, chilled water supply main pipe temperature difference, instantaneous flow rate of the flow meter, chiller power, chilled water pump power, cooling water pump power, and the power of the two cooling towers.

[0064] The parameters of the action space A include the chilled water temperature setpoint of the main unit, the chilled water pump frequency feedback, the cooling water pump frequency feedback, and the frequency feedback of the two cooling towers.

[0065] The state transition probability P is fitted by the subsequently constructed Transformer-XL state transition model.

[0066] Reward function R: used to provide immediate feedback on actions.

[0067] Discount factor γ: with a value of 0.99, emphasizes long-term cumulative rewards and avoids the algorithm making short-sighted decisions that sacrifice device lifespan.

[0068] Step 102: Construct a state transition model based on Transformer-XL. Use the state transition model based on Transformer-XL as a high-fidelity offline training environment for reinforcement learning. The state transition model based on Transformer-XL is used to accurately predict the state at the next moment based on the current state and the current action, and to fit the state transition probability.

[0069] It should be noted that the core functionality of the state transition model based on Transformer-XL is based on the current state. and perform actions Accurately predict the state of the next moment Modeling mapping relationships This provides a reliable offline training environment for reinforcement learning. The construction process of the state transition model based on Transformer-XL includes:

[0070] 1) Collect actual operation monitoring data of the central air conditioning cooling source side for a preset time period, and divide the actual operation monitoring data into training set and test set. Specifically, collect actual operation monitoring data of the central air conditioning cooling source side of a commercial building from May to August of a certain year, with a sampling interval of 1 hour, obtaining approximately 3000 sets of valid samples, which are randomly divided into training set (2100 sets) and test set (900 sets) in a 7:3 ratio. Normalize all input variables to [0,1] to eliminate the influence of dimensions and avoid slow model training or non-convergence caused by differences in the dimension and order of magnitude of dependent variables.

[0071] 2) The structure of the Transformer-XL state transition model consists of an input projection layer, a relative position encoding module, stacked Transformer-XL blocks, and an output projection layer. The input projection layer maps the concatenated "state + action" vector to 128 dimensions (consistent with the Transformer-XL model's dimensions). The relative position encoding module uses the classic relative position encoding method of Transformer-XL to capture temporal sequence information. Ten stacked Transformer-XL blocks are used, with eight multi-head attention heads and a 256-dimensional hidden layer in the feedforward network. A memory sequence is introduced to store historical state features to enhance long-term dependency modeling capabilities. The output projection layer maps the encoded features to a 10-dimensional next-time state vector, with the output dimension consistent with the state space.

[0072] 3) The Transformer-XL state transition model was trained using the training set. The performance of the Transformer-XL state transition model was evaluated using mean squared error (MSE), mean absolute error (MAE), and coefficient of determination (R²), and the generalization ability of the Transformer-XL state transition model was verified using the test set, resulting in the trained Transformer-XL state transition model. Specifically, the Adam optimizer was used with a learning rate of 0.001, 100 training epochs, and a batch size of 32. After 10 independent training iterations, the model's performance metrics were: mean squared error (MSE) = 0.3917 ± 0.1881, mean absolute error (MAE) = 0.2962 ± 0.0830, and coefficient of determination (R²) = 0.9979 ± 0.0007. The R² value is close to 1 with minimal fluctuation, indicating that the model can stably capture the dynamic evolution of multiple variables and long time series on the cold source side, meeting the accuracy requirements for state transition prediction.

[0073] Fitting state transition probabilities based on the Transformer-XL state transition model: The Transformer-XL state transition model uses the current state s t and the currently executing action a t Given the input, the true state s at the next time step. t+1 Using labels, the system is trained by minimizing the mean squared error loss function to learn a deterministic state transition mapping f, i.e., s. t+1 =f(s t ,a t ), fitting the state transition probability P, including: state transition probability P(s) t+1 |s t , a t ) indicates that in state s t Next, execute action a t Then transition to the next state s t+1 The probability of state transition; given the current state and actions of the central air conditioning cold source side, its next state is uniquely determined by the system's physical characteristics, and the state transition probability P degenerates into the form of a Dirac delta function, i.e., P(s t+1 |s t , a t ) = 1 if and only if s t+1 =f(s t ,a t If the value is 0, then the value is 0; otherwise, it is 0. Under the Markov decision process framework, this deterministic state transition mapping corresponds to the degenerate distribution of the state transition probability P, that is, by minimizing the mean square error loss function, the deterministic mapping f is learned, and the equivalent fitting of the state transition probability P is achieved.

[0074] Step 103: Construct a truncated quantile commentator algorithm with an integrated action expansion mechanism. The truncated quantile commentator algorithm with an integrated action expansion mechanism includes priority experience replay, N-step reward, network architecture optimization, directed exploration of noise, and a penalty-free action expansion mechanism.

[0075] It should be noted that the Truncated Quantile Critics (TQC) algorithm, which integrates a fusion action expansion mechanism, combines priority experience replay, N-step reward, network architecture optimization, directed exploration noise, and a penalty-free action expansion mechanism to obtain a target reinforcement learning algorithm adapted to cold source-side scheduling. Specifically:

[0076] Priority Experience Replay: Based on the TD error, the sampling probability of the sample is dynamically adjusted so that the experience with high information value is learned more frequently, thereby improving the efficiency of sample utilization and accelerating model convergence. The priority index α is set to 0.7.

[0077] N-step reward: Extends the 1-step reward to a 4-step reward. By accumulating rewards for 4 consecutive steps and combining them with the state value estimation in the 4th step, a more stable and accurate learning target is generated, reducing the estimation bias caused by delay effects such as temperature changes.

[0078] Network architecture: The standard ReLU activation function is replaced with LeakyReLU to prevent neuron death, and a BatchNorm1d layer is added to alleviate the gradient vanishing problem and improve the training stability of deep networks.

[0079] Directional exploration noise: Replace isotropic Gaussian noise with directional Gaussian noise, and apply positive bias Gaussian noise to specific actions such as chilled water pump frequency. ~N(0,0.15) incorporates domain knowledge to guide the agent to explore physically efficient action space regions, avoiding ineffective exploration.

[0080] Penalty-free action extension mechanism: The action extension component is retained, and the penalty weight is set to 0. For each batch of sampled states s and actions... Build extended action set A exp The extended action set includes the original action a and the policy noise action. Disturbance actions = +δ(δ~N(0,0.3))Uniform random action ~Uniform(0,1), Mirroring action The extended action set is input into the critic network to update the runtime statistics of the BatchNorm layer, enhancing the critic network's ability to evaluate the action space and its generalization, effectively suppressing Q-value overestimation bias under dynamic loads, and without imposing conservative constraints on performance optimization. Here, s represents the state sampled from the priority experience replay buffer, a represents the original action corresponding to s, and π(s) represents the action output by the actor network in the current state s. Both δ and are Gaussian noise, Uniform(0,1) represents a uniform distribution in the interval [0,1], and A exp It is an extended action set that includes the original action and four derived actions.

[0081] The architecture of the Truncated Quantile Critics' Algorithm (TQC algorithm) provided in this invention is as follows: Figure 2 As shown, support is provided from three dimensions: exploration diversity, value estimation accuracy, and sample learning efficiency, ensuring that the agent can efficiently explore the optimal scheduling strategy under dynamic load scenarios on the cold source side, while effectively suppressing Q-value overestimation bias. The input of the truncated quantile critic algorithm provided in this invention is: actor network. Critics Network Target Commentator Network Priority index Importance sampling coefficient Actor's online learning rate Critics' Network Learning Rate Discount factor γ, soft update coefficient Batch size B, buffer capacity M, minimum buffer size Given N steps with length n and the number of quantiles K, the output is the trained actor network. (Optimal energy efficiency scheduling strategy).

[0082] Step 104: Establish a reward function with energy efficiency as the core. The reward function includes temperature control reward, energy efficiency ratio improvement reward, equipment high-efficiency operation reward, system coordination reward, energy consumption suppression reward, and operation stability reward.

[0083] It should be noted that in this invention, a multi-objective reward function is constructed, denoted as:

[0084] ;

[0085] in, The total reward of the reward function. As a reward for temperature control, Incentives for improving energy efficiency ratio Incentives for efficient operation of chilled water pumps Incentives for efficient operation of cooling water pumps To coordinate rewards within the system, As a power penalty, As a penalty for smooth operation, a reward is given for efficient operation of the chilled water pump. Rewards for efficient operation of cooling water pumps Both serve as rewards for efficient equipment operation and as power penalties. Energy consumption suppression reward, motion smoothness penalty This is a reward for operational stability.

[0086] Temperature control reward Maintain the temperature of the chilled water return main pipe at around the set value of 11℃. Scores are awarded in segments based on temperature deviation. When the deviation is ≤0.5℃, the maximum reward is 3.0. When the deviation exceeds the threshold, the reward decreases linearly. A weight of 0.3 is used to achieve appropriate temperature control.

[0087] Energy efficiency ratio improvement reward To maximize the system's energy efficiency ratio (COP), the calculation method is COP² × 35, where COP = Q. ch / P total Q ch The cooling capacity (kW) on the cold source side, P total The total operating power (kW) of all equipment on the cold source side is used to amplify the incentive for high energy efficiency by square amplification, with the highest weight of 35 highlighting the core optimization goal of prioritizing energy efficiency.

[0088] High-efficiency operation reward for chilled water pumps : Guide the chilled water pump to operate in the high-efficiency frequency range. The maximum reward for the core high-efficiency range of 45~48Hz is 40.0. The secondary reward ranges are 42~45Hz and 48~50Hz. The basic reward range is 30~42Hz. A penalty is applied when the frequency is <30Hz or >50Hz, with a weight of 5.0.

[0089] Cooling water pump high-efficiency operation reward : Guide the cooling water pump to operate in the high-efficiency frequency range. The maximum reward for the core high-efficiency range of 47~49Hz is 30.0. The secondary reward ranges are 45~47Hz and 49~50Hz. A penalty is applied when the frequency is <45Hz or >50Hz, with a weight of 4.0.

[0090] System coordination reward (i.e., cooling tower synchronization reward) To promote frequency synchronization between the two cooling towers and avoid efficiency loss, a score is awarded based on the frequency difference. When the frequency difference is less than 3Hz, the maximum reward is 40.0. When the frequency difference is between 3 and 5Hz, the reward decreases linearly. When the frequency difference is greater than 5Hz, it becomes a penalty. A weight of 8 is used to achieve coordinated operation of the system.

[0091] Power penalty Directly suppress the high power consumption of chilled water pumps and cooling water pumps. The calculation method is actual power of equipment × 0.001, which is used as a negative item in the total reward calculation with a weight of 0.2 to achieve moderate energy consumption suppression.

[0092] Smoothness of movement penalty To prevent large fluctuations in control commands and improve equipment operational stability, the calculation method is 3.0 × (current action - previous action)², which is directly deducted from the total reward.

[0093] Step 105: Based on the truncated quantile critic algorithm based on the Transformer-XL state transition model and the fusion action extension mechanism, the agent is trained offline using the reward function, so that the agent learns the optimal energy efficiency scheduling strategy on the cold source side.

[0094] It should be noted that, based on the Transformer-XL state transition model constructed in step 102 and the TQC algorithm in step 103, the agent is trained offline using the multi-objective reward function designed in step 104. The training iterations consist of 2500 rounds, with 200 steps per round. Exploration noise gradually decreases with each training iteration, achieving a training logic of sufficient exploration in the early stages and stable policy in the later stages. When the total reward stabilizes at 7.0 × 10... 6 ~7.4×10 6 When the interval is reached, the algorithm converges, and the trained actor network is obtained as the optimal scheduling strategy.

[0095] like Figure 3 As shown, based on the truncated quantile critic algorithm using the Transformer-XL state transition model and fusion action extension mechanism, the agent is trained offline using a reward function to learn the optimal energy efficiency scheduling strategy for the cold source side, including:

[0096] S1. Initialize the actor network, critic network, and target critic network of the truncated quantile critic algorithm provided in this invention, and keep the parameters of the target critic network consistent with those of the critic network.

[0097] S2. Initialize the priority experience replay buffer, the N-step buffer, and the exploration noise, and set the core hyperparameters of the algorithm. Specifically, initialize the priority experience replay buffer D and the N-step buffer B. n 1. Explore the noise σ and set the core hyperparameters of the algorithm: 2 commentator networks, 51 quantiles, value range vmin~vmax -10~10, actor network learning rate 0.003, commentator network learning rate 0.001, discount factor γ=0.99, soft update coefficient τ=0.0005, N-step length n=4, replay buffer capacity 1×10 6Batch size 128, initial exploration noise standard deviation 0.12, minimum exploration noise standard deviation 0.03, noise attenuation coefficient 0.999, Transformer-XL memory length 10, model dimension 128.

[0098] S3. Perform 2500 training iterations, with 200 steps per iteration. At the beginning of each iteration, reset the offline training environment to obtain the initial state and reset the N-step buffer B. n Until the training ends. Specifically:

[0099] S301. Output the motion from the actor network and superimpose exploration noise to obtain the actual executed motion. , ;

[0100] S302. Perform actions in the Transformer-XL state transition model. The next state s', the immediate reward r, and the training end flag done are observed.

[0101] S303, will Add N-step buffer If the buffer is ready, construct an N-step transfer sample, assign an initial priority p to the sample, and store it in the priority experience replay buffer D.

[0102] S304. If the sample size of the priority experience replay buffer D is greater than... Based on priority Sample small batches of samples and calculate importance sampling weights. ;

[0103] S305, Critics Network Update: For each sample j, the next action is obtained through the actor network. Calculate the minimum output of the target critic network. Calculate the time-difference objective ,Will Projecting onto the quantile support set yields For each critic network, compute its output. loss function Update the critic network parameters using gradient descent The priority of samples in buffer D is updated based on the TD error.

[0104] S306, Actor Network Update: Output the action of the current sample based on the actor network. Calculate the expected output of the critic network. Take the minimum value loss function -Entropy regularization updates the actor network parameters θ through gradient descent;

[0105] S307, Target Commentator Network Soft Update: According to Formula Update the target commentator network parameters to reduce the relevance of parameter updates.

[0106] S4. After each iteration, the exploration noise σ is decreased by a noise attenuation coefficient of 0.999, and the importance sampling coefficient β is updated until the total reward stabilizes at 7.0 × 10⁻⁶. 6 ~7.4×10 6 Within the interval, the algorithm converges, and the trained actor network is obtained as the optimal energy efficiency scheduling strategy, enabling the agent to learn the optimal energy efficiency scheduling strategy on the cold source side.

[0107] Step 106: Deploy the optimal energy efficiency scheduling strategy output by the trained agent to the control system on the central air conditioning cold source side, and output scheduling instructions to optimize energy efficiency scheduling on the central air conditioning cold source side.

[0108] It should be noted that the optimal energy efficiency scheduling strategy output by the trained agent is deployed to the integrated monitoring and management platform on the cold source side of the central air conditioning system, replacing the traditional PID control strategy. At the same time, the performance monitoring module monitors key indicators such as the system energy efficiency ratio, chilled water return temperature, and system average power in real time, so as to achieve real-time feedback on the scheduling effect.

[0109] The energy efficiency optimization method for the cold source side of central air conditioning provided by this invention constructs a high-fidelity Transformer-XL state transition model, providing a safe and efficient offline training environment for reinforcement learning, fundamentally avoiding the energy waste and safety risks of online training; it constructs a truncated quantile critic algorithm with an integrated action extension mechanism, effectively suppressing the overestimation bias of Q-value under dynamic load, while fully tapping the energy-saving potential of the system; the reward function is designed with energy efficiency as the core, realizing multi-objective collaborative optimization of temperature control, efficient equipment operation, etc.; the optimal scheduling strategy obtained through offline training improves the energy efficiency ratio of the cold source side system by about 104.4%, reduces the average power consumption by about 42.2%, and meets the temperature control accuracy requirements of engineering. Moreover, the strategy has strong robustness to the design details of the reward function, greatly improving the reliability and practicality of engineering deployment, and solving the technical problems of poor reinforcement learning training security, easy to cause control fluctuations or failures, and weak multi-objective coordination ability of existing central air conditioning cold source side energy efficiency optimization methods.

[0110] To verify the effectiveness of the method of this invention, DDPG, SAC, TD3 and the original TQC algorithm were selected for comparison. All comparison algorithms integrate PER and a 4-step reward mechanism and are trained on the same experimental platform (Python+PyTorch). The historical operating data of a central air conditioner in September 2024 was used as the test set. The comparison was carried out from three dimensions: energy efficiency ratio (COP), temperature control accuracy and average power consumption. The results are shown in Table 1.

[0111] Table 1

[0112]

[0113] From Table 1, we can obtain:

[0114] The average COP of the method of this invention is 9.6951, which is 104.4% higher than the historical operating data (4.7433) and 96.2% higher than the original TQC (4.9425), significantly higher than DDPG (4.9710), SAC (4.7902), and TD3 (4.8852). The average temperature of the chilled water return main pipe of the method of this invention is 11.5789℃, with a deviation of 0.58℃ from the set value of 11℃, which is within the acceptable engineering range of ±1℃. Moreover, the temperature fluctuation is stable in the range of 11~12℃, which is far better than the large fluctuation of 9~16℃ in the historical operating data. The average power consumption of the method of the present invention is 132.9404kW, which is 42.2% lower than the historical operating data (229.9487kW) and 40.0% lower than the original TQC (221.6309kW), and significantly lower than DDPG (220.9579kW), SAC (223.4502kW) and TD3 (223.4103kW).

[0115] The results of the ablation experiment are shown in Table 2, which verify that the synergistic effect of the Transformer-XL state transition model and the penalty-free action extension mechanism is the key to performance improvement.

[0116] Table 2

[0117]

[0118] The experimental results of CQL (Conservative Q-Learning) weight sensitivity are shown in Table 3, which verify that the penalty term limits the potential for energy efficiency optimization, and the absence of a penalty term is the optimal configuration.

[0119] Table 3

[0120]

[0121] The results of the reward function sensitivity experiment are shown in Table 4, which verify the strong robustness of the strategy. After adjusting the fine-grained incentive of the reward function, the core performance indicators of the strategy do not change significantly.

[0122] Table 4

[0123]

[0124] Action distribution analysis, such as Figure 4 The results show that the frequency of the chilled water pump gradually converges from the conservative exploration of 30-35Hz in the early stage of training to the efficient range of 45-50Hz in the later stage, and is always within the safety boundary, which verifies the safety and optimality of the strategy.

[0125] For easier understanding, please refer to Figure 5 This invention provides an embodiment of a central air conditioning cold source side energy efficiency optimization device, comprising:

[0126] The problem transformation unit is used to model the scheduling optimization problem of the central air conditioning energy side as a Markov decision process. The Markov decision process is defined as a quintuple consisting of a state space, an action space, a reward function, a state transition probability, and a discount factor.

[0127] The modeling unit is used to build a state transition model based on Transformer-XL. The state transition model based on Transformer-XL is used as a high-fidelity offline training environment for reinforcement learning. The state transition model based on Transformer-XL is used to accurately predict the state at the next moment based on the current state and the current action, and to fit the state transition probability.

[0128] Algorithm building unit: used to build the truncated quantile commentator algorithm with integrated action expansion mechanism. The truncated quantile commentator algorithm with integrated action expansion mechanism includes priority experience replay, N-step reward, network architecture optimization, directed exploration noise and penalty-free action expansion mechanism;

[0129] The reward function construction unit is used to establish a reward function with energy efficiency as the core. The reward function includes temperature control reward, energy efficiency ratio improvement reward, equipment high-efficiency operation reward, system coordination reward, energy consumption suppression reward and operation stability reward.

[0130] The training unit is used to train the agent offline using the reward function based on the truncated quantile critic algorithm based on the Transformer-XL state transition model and the fusion action extension mechanism, so that the agent learns the optimal energy efficiency scheduling strategy on the cold source side.

[0131] The output unit is used to deploy the optimal energy efficiency scheduling strategy output by the trained agent to the control system on the cooling source side of the central air conditioning system, and outputs scheduling instructions to optimize the energy efficiency of the cooling source side of the central air conditioning system.

[0132] In one embodiment, the parameters of the state space include outdoor temperature, outdoor humidity, chilled water return main temperature, chilled water supply main temperature difference, instantaneous flow rate of the flow meter, chiller power, chilled water pump power, cooling water pump power, and the power of the two cooling towers.

[0133] The parameters of the action space include the chilled water temperature setpoint of the main unit, the chilled water pump frequency feedback, the cooling water pump frequency feedback, and the frequency feedback of the two cooling towers.

[0134] In one embodiment, the process of constructing a state transition model based on Transformer-XL includes:

[0135] Collect actual operation monitoring data of the central air conditioning cold source side for a preset time period, and divide the actual operation monitoring data into training set and test set;

[0136] The structure of the Transformer-XL state transition model consists of an input projection layer, a relative position encoding module, stacked Transformer-XL blocks, and an output projection layer.

[0137] The Transformer-XL state transition model is trained using the training set. The performance of the Transformer-XL state transition model is evaluated using mean squared error, mean absolute error, and coefficient of determination. The generalization ability of the Transformer-XL state transition model is verified using the test set, resulting in the trained Transformer-XL state transition model.

[0138] In one embodiment, the reward function is:

[0139] ;

[0140] in, The total reward of the reward function. As a reward for temperature control, Incentives for improving energy efficiency ratio Incentives for efficient operation of chilled water pumps Incentives for efficient operation of cooling water pumps To coordinate rewards within the system, As a power penalty, As a penalty for smooth operation, a reward is given for efficient operation of the chilled water pump. Rewards for efficient operation of cooling water pumps Both serve as rewards for efficient equipment operation and as power penalties. Energy consumption suppression reward, motion smoothness penalty This is a reward for operational stability.

[0141] The central air conditioning cold source side energy efficiency optimization device provided in this invention is used to execute the central air conditioning cold source side energy efficiency optimization method provided in this invention. Its principle and the technical effect obtained are the same as the central air conditioning cold source side energy efficiency optimization method provided in this invention, and will not be described again here.

[0142] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing energy efficiency on the cold source side of a central air conditioning system, characterized in that, include: The scheduling optimization problem of the central air conditioning energy side is modeled as a Markov decision process, which is defined as a quintuple consisting of a state space, an action space, a reward function, a state transition probability, and a discount factor. A state transition model based on Transformer-XL is constructed and used as a high-fidelity offline training environment for reinforcement learning. The state transition model based on Transformer-XL is used to accurately predict the state at the next moment based on the current state and the current action, and to fit the state transition probability. A truncated quantile critic algorithm with an integrated action expansion mechanism is constructed. The truncated quantile critic algorithm with an integrated action expansion mechanism includes priority experience replay, N-step reward, network architecture optimization, targeted exploration of noise, and penalty-free action expansion mechanism. Establish a reward function with energy efficiency as the core, which includes temperature control reward, energy efficiency ratio improvement reward, equipment high-efficiency operation reward, system coordination reward, energy consumption suppression reward and operation stability reward; Based on the truncated quantile critic algorithm with Transformer-XL state transition model and fusion action extension mechanism, the agent is trained offline using the reward function, so that the agent learns the optimal energy efficiency scheduling strategy on the cold source side. The optimal energy efficiency scheduling strategy output by the trained agent is deployed to the control system on the cooling source side of the central air conditioning system, and the scheduling instructions are output to optimize the energy efficiency of the cooling source side of the central air conditioning system.

2. The method for optimizing the energy efficiency of the central air conditioning cold source side according to claim 1, characterized in that, The parameters in the state space include outdoor temperature, outdoor humidity, chilled water return main temperature, chilled water supply main temperature difference, instantaneous flow rate of the flow meter, chiller power, chilled water pump power, cooling water pump power, and the power of the two cooling towers. The parameters of the action space include the chilled water temperature setpoint of the main unit, the chilled water pump frequency feedback, the cooling water pump frequency feedback, and the frequency feedback of the two cooling towers.

3. The method for optimizing the energy efficiency of the central air conditioning cold source side according to claim 1, characterized in that, The process of building a state transition model based on Transformer-XL includes: Collect actual operation monitoring data of the central air conditioning cold source side for a preset time period, and divide the actual operation monitoring data into training set and test set; The structure of the Transformer-XL state transition model consists of an input projection layer, a relative position encoding module, stacked Transformer-XL blocks, and an output projection layer. The Transformer-XL state transition model is trained using the training set. The performance of the Transformer-XL state transition model is evaluated using mean squared error, mean absolute error, and coefficient of determination. The generalization ability of the Transformer-XL state transition model is verified using the test set, resulting in the trained Transformer-XL state transition model.

4. The method for optimizing the energy efficiency of the central air conditioning cold source side according to claim 1, characterized in that, The reward function is: ； in, The total reward of the reward function. As a reward for temperature control, Incentives for improving energy efficiency ratio Incentives for efficient operation of chilled water pumps Incentives for efficient operation of cooling water pumps To coordinate rewards within the system, As a power penalty, As a penalty for smooth operation, a reward is given for efficient operation of the chilled water pump. Rewards for efficient operation of cooling water pumps Both serve as rewards for efficient equipment operation and as power penalties. Energy consumption suppression reward, motion smoothness penalty This is a reward for operational stability.

5. The method for optimizing the energy efficiency of the central air conditioning cold source side according to claim 1, characterized in that, Based on the truncated quantile critic algorithm using the Transformer-XL state transition model and fused action extension mechanism, the agent is trained offline using a reward function to learn the optimal energy efficiency scheduling strategy for the cold source side, including: Initialize the actor network, critic network, and target critic network of the truncated quantile critic algorithm for the fusion action extension mechanism, and keep the parameters of the target critic network consistent with those of the critic network; Initialize the priority experience replay buffer, the N-step buffer, and the exploration noise; set the core hyperparameters of the algorithm. Perform 2500 training iterations, with 200 steps per iteration. At the beginning of each iteration, reset the offline training environment to obtain the initial state and reset the N-step buffer until the training ends. After each iteration, the exploration noise is attenuated by a noise attenuation factor of 0.999, and the importance sampling coefficient is updated until the total reward stabilizes at [value missing]. Within the interval, the algorithm converges, and the trained actor network is obtained as the optimal energy efficiency scheduling strategy, enabling the agent to learn the optimal energy efficiency scheduling strategy on the cold source side.

6. The method for optimizing the energy efficiency of the central air conditioning cold source side according to claim 1, characterized in that, The discount factor is set to 0.

99.

7. A central air conditioning cold source side energy efficiency optimization device, characterized in that, include: The problem transformation unit is used to model the scheduling optimization problem of the central air conditioning energy side as a Markov decision process. The Markov decision process is defined as a quintuple consisting of a state space, an action space, a reward function, a state transition probability, and a discount factor. The modeling unit is used to build a state transition model based on Transformer-XL. The state transition model based on Transformer-XL is used as a high-fidelity offline training environment for reinforcement learning. The state transition model based on Transformer-XL is used to accurately predict the state at the next moment based on the current state and the current action, and to fit the state transition probability. Algorithm building unit: used to build the truncated quantile commentator algorithm with integrated action expansion mechanism. The truncated quantile commentator algorithm with integrated action expansion mechanism includes priority experience replay, N-step reward, network architecture optimization, directed exploration noise and penalty-free action expansion mechanism; The reward function construction unit is used to establish a reward function with energy efficiency as the core. The reward function includes temperature control reward, energy efficiency ratio improvement reward, equipment high-efficiency operation reward, system coordination reward, energy consumption suppression reward and operation stability reward. The training unit is used to train the agent offline using the reward function based on the truncated quantile critic algorithm based on the Transformer-XL state transition model and the fusion action extension mechanism, so that the agent learns the optimal energy efficiency scheduling strategy on the cold source side. The output unit is used to deploy the optimal energy efficiency scheduling strategy output by the trained agent to the control system on the cooling source side of the central air conditioning system, and outputs scheduling instructions to optimize the energy efficiency of the cooling source side of the central air conditioning system.

8. The central air conditioning cold source side energy efficiency optimization device according to claim 7, characterized in that, The parameters in the state space include outdoor temperature, outdoor humidity, chilled water return main temperature, chilled water supply main temperature difference, instantaneous flow rate of the flow meter, chiller power, chilled water pump power, cooling water pump power, and the power of the two cooling towers. The parameters of the action space include the chilled water temperature setpoint of the main unit, the chilled water pump frequency feedback, the cooling water pump frequency feedback, and the frequency feedback of the two cooling towers.

9. The central air conditioning cold source side energy efficiency optimization device according to claim 7, characterized in that, The process of building a state transition model based on Transformer-XL includes: Collect actual operation monitoring data of the central air conditioning cold source side for a preset time period, and divide the actual operation monitoring data into training set and test set; The structure of the Transformer-XL state transition model consists of an input projection layer, a relative position encoding module, stacked Transformer-XL blocks, and an output projection layer. The Transformer-XL state transition model is trained using the training set. The performance of the Transformer-XL state transition model is evaluated using mean squared error, mean absolute error, and coefficient of determination. The generalization ability of the Transformer-XL state transition model is verified using the test set, resulting in the trained Transformer-XL state transition model.

10. The central air conditioning cold source side energy efficiency optimization device according to claim 7, characterized in that, The reward function is: ； in, The total reward of the reward function. As a reward for temperature control, Incentives for improving energy efficiency ratio Incentives for efficient operation of chilled water pumps Incentives for efficient operation of cooling water pumps To coordinate rewards within the system, As a power penalty, As a penalty for smooth operation, a reward is given for efficient operation of the chilled water pump. Rewards for efficient operation of cooling water pumps Both serve as rewards for efficient equipment operation and as power penalties. Energy consumption suppression reward, motion smoothness penalty This is a reward for operational stability.

Images