A deep reinforcement learning driven dynamic precooling and preheating optimization method for air conditioning system

Abstract

The application relates to the air conditioning technical field, in particular to a deep reinforcement learning driven air conditioning system dynamic precooling and preheating optimization method, which comprises the following steps: collecting current environment state information in real time and obtaining exogenous disturbance prediction data, so as to train a multi-step prediction model to perform multi-step iterative prediction on a candidate control action sequence, and obtain a future state trajectory; based on the future state trajectory, a composite evaluation function is used to score the control action sequence; under the state evolution constraint generated by the multi-step prediction model, a deep reinforcement learning algorithm is used to solve an action sequence which optimizes the composite evaluation function, and the first control action of the action sequence is executed; subsequently, the above process is repeated in a rolling time domain manner, and the multi-step prediction model is updated in real time by using the latest observation data, so that adaptive dynamic precooling and preheating control is realized. The application aims to realize adaptive dynamic precooling and preheating control of an air conditioning system.

Description

Technical Field

[0001] This application relates to the field of air conditioning technology, specifically to a deep reinforcement learning-driven method for optimizing dynamic precooling and preheating of air conditioning systems. Background Technology

[0002] Dynamic precooling and preheating of air conditioning systems refers to proactively adjusting the indoor temperature to a level lower or higher than the normal set value before the building actually enters its high-load period. This utilizes the building envelope (walls, floors, furniture, etc.) to temporarily store cooling or heating energy. When the peak arrives, the output of the air conditioning units is reduced or suspended, allowing the building envelope to slowly release the stored cooling or heating energy, thereby smoothing the power consumption curve, reducing instantaneous power peaks, and maintaining indoor temperature and humidity within a comfortable range.

[0003] Traditional air conditioning systems typically operate on an on-demand start-stop model, resulting in high electricity costs during peak electricity price periods or at the moment of demand response events. Furthermore, the temperature lag caused by building thermal inertia necessitates that the unit start a certain amount of time in advance to be effective. By pre-cooling and pre-heating, the high-electricity-price window of energy injection can be shifted to a low-electricity-price window, reducing electricity costs and grid peak pressure without compromising comfort. Simultaneously, it avoids frequent start-stop cycles of the air conditioning unit, thereby extending the equipment's lifespan.

[0004] The energy-saving peak-shaving effect of precooling and preheating depends on a series of time-series decisions regarding how far in advance, how much in advance, and how long the preheating lasts. However, the thermodynamics of buildings is affected by multiple uncertainties such as real-time weather, opening and closing of doors and windows, number of people, and heat dissipation of equipment. It has the characteristics of nonlinearity, large lag, and strong coupling, making it difficult for traditional rule or model predictive control to achieve online adjustment.

[0005] Therefore, the technical problem to be solved by this application is: in the pre-cooling and pre-heating of an air conditioning system, how to determine the optimal timing and intensity of pre-cooling and pre-heating through an online rolling optimization strategy driven by deep reinforcement learning, so as to reduce peak power consumption and electricity costs without sacrificing comfort. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems, thereby resolving the existing issues.

[0007] The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems proposed in this application adopts the following technical solution:

[0008] One embodiment of this application provides a deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems, which includes the following steps:

[0009] Real-time acquisition of current environmental state information and acquisition of exogenous disturbance prediction data are used to train a multi-step prediction model to perform multi-step iterative prediction of candidate control action sequences and obtain future state trajectories.

[0010] Based on the future state trajectory, the control action sequence is scored using a composite evaluation function, which includes energy consumption cost, thermal comfort deviation cost, and peak demand penalty cost.

[0011] Under the state evolution constraints generated by the multi-step prediction model, a deep reinforcement learning algorithm is used to solve for the action sequence that optimizes the composite evaluation function, and the first control action is executed. The above process is then repeated in a rolling time domain manner, and the multi-step prediction model is updated in real time using the latest observation data to achieve adaptive dynamic pre-cooling and pre-heating control.

[0012] Preferably, the multi-step prediction model is a gray box dynamics model, which includes: firstly, using a first-order RC thermal network to generate explicit Euler discrete equations, and then using a residual network to fit the deviation between the model prediction value and the true value to compensate for the unmodeled state.

[0013] Preferably, the multi-step prediction model takes the environmental state estimation vector, air conditioning control action vector, and exogenous disturbance prediction vector of the current time for the next preset step as input, and outputs the environmental state estimation vector for the next step, so that the optimizer can generate a candidate control action sequence based on the multi-step prediction model and the composite reward function.

[0014] Preferably, the environmental status information includes room temperature, wall temperature, humidity, and air conditioning power; after each step, the environmental status information is readjusted using the deviation between the new sensor readings and the model output.

[0015] Preferably, the exogenous disturbance prediction data includes: weather forecast data, personnel occupancy prediction, equipment heat dissipation prediction, and time-varying electricity price and demand response signals. The prediction data is sent to the local controller at the beginning of the control cycle through an external prediction module.

[0016] Preferably, the candidate control action sequence is a control sequence of length H generated at the current time t, whose first element is executed in the next control cycle, and then solved again in a rolling time domain manner to form a closed-loop control.

[0017] Preferably, the iterative prediction of the multi-step prediction model adopts a chain-like expansion method, and the state trajectory is calculated recursively step by step within the prediction time domain H, with the initial state assigned by the sensor measured value.

[0018] Preferably, the energy consumption cost is calculated by using the instantaneous power from the predicted state-action pair obtained from rollout and estimated in conjunction with time-varying electricity prices; the thermal comfort deviation cost is calculated by the portion of the deviation between the predicted room temperature and the set value that exceeds the tolerance range; and the peak demand penalty cost is a soft penalty term for the maximum power in the predicted trajectory exceeding a threshold.

[0019] Preferably, the composite evaluation function is converted into a reinforcement learning reward form by using a negative sign, the optimization objective is to maximize the cumulative reward, and the long-term cost is weighted using a discount factor γ∈[0.95, 1].

[0020] Preferably, the solution process of the composite evaluation function adopts the stochastic shooting method or the full-space gradient method, and performs Monte Carlo sampling on the uncertainty of exogenous disturbances, taking the expected reward as the optimization objective to achieve robust rolling optimization control.

[0021] This application has at least the following beneficial effects:

[0022] This application formalizes the precooling and preheating control problem of building air conditioning systems as a model-driven Markov decision process. Using a predictive model as the environmental transfer function, it rapidly rolls out a multi-step trajectory before each decision. The policy gradient is calculated using a reward function that incorporates electricity costs, comfort levels, and peak load penalties. The optimal control sequence is then solved using zero-order optimization or the gradient method, executing the control action of its first element and performing rolling replanning. Its beneficial effects include: by using peak load penalties and multi-predictive joint optimization, it proactively injects cooling or heating energy into low-price periods in advance, significantly reducing peak demand and minimizing power overruns during DR events; by utilizing time-varying electricity prices and thermal inertia for coordinated scheduling, it effectively reduces electricity costs, decreases equipment start-up and shutdown frequency, extends the lifespan of air conditioning units, reduces maintenance costs, and ensures that room temperature deviations remain within the tolerance range. Attached Figure Description

[0023] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems provided in this application. Detailed Implementation

[0025] To further illustrate the technical means and effects adopted by this application to achieve the intended purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0027] The following description, in conjunction with the accompanying drawings, details the specific scheme of the deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems provided in this application.

[0028] One embodiment of this application provides a deep reinforcement learning-driven method for optimizing the dynamic precooling and preheating of an air conditioning system.

[0029] Specifically, a deep reinforcement learning-driven method for optimizing the dynamic precooling and preheating of air conditioning systems is provided below. Please refer to [link / reference]. Figure 1 The method includes the following steps:

[0030] S1: Real-time acquisition of current environmental state information and acquisition of exogenous disturbance prediction data, which are used to train a multi-step prediction model to perform multi-step iterative prediction of candidate control action sequences and obtain future state trajectories.

[0031] The thermal capacity of the building envelope causes a delay of several minutes or even hours in the power response of the air conditioning system to room temperature (i.e., the room temperature reaching the preset temperature). In addition, random window opening, personnel movement, and equipment start-up and shutdown introduce uncertainties from exogenous disturbances. If the optimizer only relies on the current instantaneous state or uses a single-step linear optimization model, it cannot take into account the dynamic coupling relationship in the pre-cooling and pre-heating process, resulting in inaccurate pre-cooling and pre-heating start-up timing and ineffective transfer of peak load.

[0032] This application employs an interpretable gray box dynamics model, which performs H-step forward predictions before each decision, thereby embedding the relationship between future disturbances, control sequences, and room temperature changes into the optimization variable space as a state transition constraint for subsequent reward evaluation and strategy search. This transforms the originally lagging and uncertain building thermal response into a predictable and optimizable deterministic multi-stage problem.

[0033] Specifically, this application utilizes the data-physics hybrid modeling concept to construct a multi-step prediction model. The multi-step prediction model adopts a gray box dynamics model: the building thermal balance equation is discretized into a low-order state-space form, and the environmental state vector and exogenous disturbance vector are used as inputs. The nonlinear disturbances and spatial heterogeneity not characterized by linear lumped parameters are learned through a residual neural network to obtain a single-step transfer function. With this function as the core, a deterministic or random rollout (trajectory expansion) is performed along the time chain within the control period to form a multi-step virtual trajectory, thereby reducing the dimension of the high-order distributed parameter thermal system to a computable mapping of current state-control-disturbance-future state.

[0034] The multi-step prediction model can be expressed as follows: , , Where H represents the total number of prediction steps, the prediction time domain duration is typically 30-120 minutes, and it needs to cover the building's thermal time constant, i.e., the product of thermal resistance and heat capacity, to ensure that the heat storage and release process is fully incorporated into the optimization, according to the controlled sampling period. (The value is 5 minutes) Conversion, H is the ratio of the prediction time domain duration to the control sampling period. This represents the actual value measured by the sensor at the starting time t of the rolling time-domain optimization. The initial state of the predicted trajectory is directly assigned. This ensures that the entire multi-step rollout starts from the latest observation, preventing model errors from accumulating over time.

[0035] This indicates that the multi-step prediction model has a certain probability of predicting the future at time t. The environmental state estimation vector for each step is obtained through recursive rollout, initially using real sensor readings, including but not limited to room temperature, wall temperature, humidity, and air conditioning power. After each step, the deviation between the new sensor readings and the model output is used for bias correction or to update the model parameters. In the next cycle, the rollout is restarted, forming a closed-loop adaptive loop.

[0036] This indicates the future at time t. The air conditioning control action vector at time t is the candidate control action sequence output by the optimizer. Specifically, at the current time t, the optimizer generates an action sequence of length H based on the multi-step prediction model and the composite reward function. The entire sequence is fed into the model for H-step rollout to calculate the cumulative reward. The first element of the action sequence is used for control. Each time a sampling period progresses, the optimizer re-solves the problem using the latest measured state to generate a new control action sequence, forming a closed-loop control. Initial values ​​can be obtained through random or grid sampling within the allowable control set.

[0037] This indicates the future at time t. The prediction vector for exogenous disturbances is provided once by the external prediction module before the start of the control period. Specifically, it includes: a meteorological sub-vector, which is generated by pulling deterministic or ensemble forecast data from the meteorological station API, analyzing outdoor dry-bulb temperature, relative humidity, horizontal solar radiation, and wind speed for the next 0-2 hours, and then calculating them according to the control period. Perform linear interpolation; use subvectors for occupancy and equipment heat dissipation, utilizing building access control, Sensors, Wi-Fi / Bluetooth probes, or meeting room reservation systems use Kalman filters or LSTM regressors to output the expected number of people and the expected plug-in load of equipment every 5 minutes in the future; electricity price and demand response signal sub-vectors are read from the grid's public interface and the day-ahead or real-time TOU electricity price and DR event flags are directly mapped to vectors, and if the signals are piecewise constants, they remain constant within the event window.

[0038] The three predictions mentioned above are concatenated into a vector sequence based on the same time base, and then sent to the local controller once at the beginning of each control cycle through a message queue as a deterministic or random feedforward input for model rollout. In actual operation, it can be updated on a rolling basis with new forecasts to achieve a closed loop for disturbance prediction.

[0039] The multi-step prediction model has the following structure: a linear skeleton (grey box), firstly written as an explicit Euler discrete equation based on a first-order RC thermal network, where the elements of the system matrix A, B, and constant term c are directly given by the physical priors of building thermal resistance R, heat capacity C, and heat transfer coefficient UA, ensuring interpretability and stability; then, a residual network (data-driven) is used to fit the deviation between the model's predicted values ​​and the actual values, compensating for unmodeled states, i.e., using a small fully connected network to compensate for nonlinear effects such as unmodeled dynamics, non-uniform spatial temperature, and duct hysteresis, with a structure of 2-3 fully connected layers, each with 64-128 hidden units, and using the tanh activation function for the output; finally, the output of the multi-step prediction model. It is the sum of the linear part and the residual network.

[0040] For model parameters During training, the offline phase utilizes historical building data to minimize error, employing the AdamW optimizer with early stopping to prevent overfitting. In the online phase, whenever a new batch of data arrives, gradient descent is used for 5-10 steps to adjust the algorithm. Lightweight fine-tuning can be performed to keep the model drift bounded, or recursive least squares can be used to update the linear part to maintain real-time performance.

[0041] Compared to pure black-box networks, which are prone to temperature drift or energy conservation violations during long-term rollout, the linear skeleton of this prediction model enforces the macroscopic constraint that the product of heat capacity and temperature change is approximately equal to the difference between heat gain and heat loss, thus making it more stable in extrapolation or long-term prediction.

[0042] S2: Based on the future state trajectory, the control action sequence is scored using a composite evaluation function, which includes energy consumption cost, thermal comfort deviation cost, and peak demand penalty cost.

[0043] The optimization goals of building air conditioning systems are, on the one hand, to minimize electricity costs under time-varying electricity prices, on the other hand, to maintain indoor thermal comfort, and on the other hand, to avoid instantaneous power exceeding limits under demand response constraints.

[0044] Traditional methods treat peak loads as hard constraints or ignore them directly, resulting in the inability to obtain an optimal feasible solution or insufficient peak shaving effect. This application transforms the worst instantaneous load into a soft penalty in the reward by using a differentiable peak penalty term. This allows the strategy to actively smooth the load curve during rolling optimization and advance or delay energy injection, thereby achieving a Pareto optimal trade-off between electricity cost savings and peak reduction. It also ensures that comfort costs are only incurred outside the set tolerance zone, thus maximizing economic efficiency and grid friendliness without sacrificing user experience.

[0045] In one embodiment, the composite evaluation function can be expressed as: Where J represents the total score of the action sequence, with a higher score indicating better performance. The negative sign is used to convert the cost into a reward for reinforcement learning. The optimizer objective is maxJ. This indicates the prediction step index.

[0046] Indicates the time of prediction The instantaneous energy cost is calculated by using the predicted state-action pairs obtained from the rollout to calculate the instantaneous electrical power, and then estimating it in conjunction with time-varying electricity prices. Specifically, it involves extracting energy from the rollout trajectory. and The instantaneous power is calculated by predicting state-action using physical mapping. First, the cooling or heating demand given by the action is divided by the coefficient of performance (COP) that varies with the operating conditions to obtain the compressor power. Then, the power consumption of the wind-side and water-side motors is calculated according to the third law of the fan and the similarity law of the water pump, respectively. The three items are added together to obtain the instantaneous power. Finally, the energy cost of this step is obtained by multiplying it by the time-varying electricity price of the corresponding time period.

[0047] This represents a preset comfort weight, used to convert temperature deviation into cost on the same scale as economy. The value ranges from 0.5 to 5, and in one embodiment, it is set to 1. Indicates the time of prediction The cost of indoor thermal comfort deviation, calculated from the portion of the deviation between the predicted room temperature and the set value exceeding the tolerance zone, specifically obtained from rollout. Predicted state vector The room temperature is extracted from this value, and the difference between this room temperature and the set value (the temperature set by the air conditioner) is taken as the absolute value. Then, the allowable tolerance range is subtracted. In one embodiment, this value is taken as... If the difference is not greater than 0, the comfort cost is 0; otherwise, it is penalized in a linear or quadratic form. The value is taken as the larger of 0 and the absolute value and the allowable tolerance zone difference, thereby converting the deviation from the comfort zone into an instantaneous cost on the same scale as the economic cost.

[0048] This indicates the preset peak penalty weight, which needs to be on a much larger scale than the preset peak penalty weight. This ensures that peak clipping plays a dominant role in the optimization process. The value range is 10 to 100, and in one embodiment, the value is 30. When the DR signal arrives, the peak clipping is temporarily boosted by 3 times to achieve event response. This represents the maximum instantaneous value of electric power that may occur within the entire prediction time domain. It is obtained by statistically analyzing the entire candidate control trajectory after rollout. Specifically, the instantaneous electric power at each step is calculated and the maximum value is taken to obtain the future peak demand estimate of the control sequence. This represents the power ceiling given by the grid peak threshold or DR event. The peak demand penalty cost is a soft penalty term for the maximum electrical power exceeding a threshold in the predicted trajectory. It should be noted that when... When the value is less than 0, it means that there is no peak demand penalty cost. That is, the maximum value of 0 is taken as the soft penalty term. In other words, the penalty is only applied when the predicted peak exceeds the threshold, so as to avoid hard constraints that make optimization infeasible.

[0049] S3: Under the state evolution constraints generated by the multi-step prediction model, a deep reinforcement learning algorithm is used to solve for the action sequence that optimizes the composite evaluation function, and the first control action is executed; then the above process is repeated in a rolling time domain manner, and the multi-step prediction model is updated in real time using the latest observation data to achieve adaptive dynamic pre-cooling and pre-heating control.

[0050] Large buildings not only have high thermal inertia but also high randomness of exogenous disturbances. In addition, the control time domain is long, and a purely data-driven strategy may require millions of interactions to converge. Traditional MCP is difficult to solve in real time due to its nonlinearity and high-dimensional action space. This application utilizes the multi-step prediction model in step S1 to quickly roll out multiple trajectories before each decision. The policy gradient is directly calculated through a composite evaluation function or the optimal action sequence is searched using zero-order optimization. After executing the first step, the model is rolled back.

[0051] The final output optimal control sequence is ,in This represents the optimal control sequence, with a length of H steps. The first element is executed, and the solution is rolled over again in the next cycle. The Cartesian product of the action space is represented by a multi-step sequence of elements, each of which is a candidate control sequence of length H, in which the search maximizes the cumulative reward after the prediction model rollout.

[0052] This means calculating the expectation of the uncertainty in the disturbance prediction, that is, averaging the impact of the uncertainty of exogenous disturbances such as future weather, occupancy, and electricity prices on the reward, so that the optimizer can obtain a robust solution. P scenarios (ranging from 64 to 256) are generated from the disturbance prediction through Monte Carlo or Latin hypercube sampling, and the mean of the samples is taken after the prediction model is rolled out.

[0053] Discount factor of Power of 1 The value ranges from 0.95 to 1, which is used to ensure that long-term costs are bounded. Because large buildings have large thermal inertia, in one embodiment the value is 0.99, which reduces the weight of long-term uncertainty on current decisions. That is, the further into the future, the smaller the impact of electricity costs or peak penalties on current actions.

[0054] This means inverting the instantaneous reward at each prediction step in the composite evaluation function, i.e. Due to peak penalty It is related to the maximum value of the entire trajectory and cannot be determined in a single step. Therefore, after the H-step rollout is completed, it is added to the immediate reward of the last step, and then the expectation is calculated for all scenarios.

[0055] Using stochastic shooting (CEM) or full-space gradient method (iLQR) per The solution is performed once, and the optimal control sequence is output in real time.

[0056] The above technical features constitute the preferred embodiment of this application, which has strong adaptability and the best implementation effect. Unnecessary technical features can be added or removed according to actual needs to meet the needs of different situations.

Claims

1. A deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems, characterized in that, The method includes the following steps: Real-time acquisition of current environmental state information and acquisition of exogenous disturbance prediction data are used to train a multi-step prediction model to perform multi-step iterative prediction of candidate control action sequences and obtain future state trajectories. Based on the future state trajectory, the control action sequence is scored using a composite evaluation function, which includes energy consumption cost, thermal comfort deviation cost, and peak demand penalty cost. Under the state evolution constraints generated by the multi-step prediction model, a deep reinforcement learning algorithm is used to solve for the action sequence that optimizes the composite evaluation function, and the first control action is executed. Then, the above process is repeated in a rolling time domain manner, and the multi-step prediction model is updated in real time using the latest observation data to achieve adaptive dynamic pre-cooling and pre-heating control. The multi-step prediction model is a gray box dynamics model, which includes: firstly, using a first-order RC thermal network to generate explicit Euler discrete equations, and then using a residual network to fit the deviation between the model prediction value and the true value, and to compensate for the unmodeled state. The multi-step prediction model takes the environmental state estimation vector, air conditioning control action vector, and exogenous disturbance prediction vector of the current time as input, and outputs the environmental state estimation vector of the next step, which is used by the optimizer to generate candidate control action sequences based on the multi-step prediction model and the composite reward function.

2. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 1, characterized in that, The environmental status information includes room temperature, wall temperature, humidity, and air conditioning power; after each step, the environmental status information is readjusted using the deviation between the new sensor readings and the model output.

3. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 1, characterized in that, The exogenous disturbance prediction data includes: weather forecast data, personnel occupancy prediction, equipment heat dissipation prediction, and time-varying electricity price and demand response signals. The prediction data is sent to the local controller once at the start of the control cycle through an external prediction module.

4. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 1, characterized in that, The candidate control action sequence is a control sequence of length H generated at the current time t. Its first element is executed in the next control cycle, and then it is solved again in a rolling time domain manner to form a closed-loop control.

5. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 1, characterized in that, The iterative prediction of the multi-step prediction model adopts a chain expansion method, which recursively calculates the state trajectory step by step in the prediction time domain H, and the initial state is assigned by the sensor measured value.

6. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 1, characterized in that, The energy consumption cost is calculated by rolling out the predicted state-action pair to obtain the instantaneous power and estimated by combining it with the time-varying electricity price; the thermal comfort deviation cost is calculated by the deviation of the predicted room temperature from the set value that exceeds the tolerance range; the peak demand penalty cost is a soft penalty term for the maximum power in the predicted trajectory exceeding the threshold.

7. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 1, characterized in that, The composite evaluation function is converted into a reinforcement learning reward form by using a negative sign. The optimization objective is to maximize the cumulative reward, and the long-term cost is weighted using a discount factor γ∈[0.95, 1].

8. The deep reinforcement learning-driven dynamic precooling and preheating optimization method for air conditioning systems as described in claim 7, characterized in that, The solution process of the composite evaluation function adopts the stochastic shooting method or the full-space gradient method, and performs Monte Carlo sampling on the uncertainty of exogenous disturbances, taking the expected reward as the optimization objective to achieve robust rolling optimization control.

Images