A Comprehensive Energy-Saving HVAC Control Method Based on Deep Reinforcement Learning

Abstract

The application discloses a comprehensive energy HVAC control method based on deep reinforcement learning, and has the characteristics that the method comprises the following steps: firstly, determining the control action space that can be taken by the HVAC system; then, collecting the state data of the building HVAC system; then, constructing a control sequence, updating the HVAC control action by a DRL algorithm; calculating the immediate reward and the maximum cumulative reward after the HVAC system takes the action; finally, approximating the Q value by using an artificial neural network, and selecting the optimal control action. The application expresses the HVAC operation process as a Markov decision process, develops the HVAC control method based on DRL, minimizes the building energy cost and maintains the comfortable temperature of the residents, further proposes a heuristic method for the effective control of a complex multi-zone system in order to obtain higher expansibility, and effectively reduces the building energy cost.

Description

Technical Field

[0001] This invention relates to the field of data-driven machine learning technology, and more particularly to a comprehensive energy HVAC control method based on deep reinforcement learning. Background Technology

[0002] Intelligent and flexible scheduling of HVAC systems offers significant potential for reducing energy building costs and improving grid energy efficiency and stability. To control the energy efficiency of building HVAC systems, many methods employ simplified building thermodynamic models to predict building temperature variations. However, the performance and reliability of these methods largely depend on the accuracy of the building thermodynamic model; furthermore, building temperature is influenced by numerous factors, including building structure and materials, the surrounding environment (such as ambient temperature, humidity, and solar radiation intensity), and internal heat increments from occupants, lighting systems, and other equipment. Therefore, in the absence of complete modeling, building temperature often exhibits stochastic behavior. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0004] In view of the problems existing in the current integrated energy HVAC control method based on deep reinforcement learning, this invention is proposed.

[0005] The purpose of this invention is to use deep reinforcement learning (DRL) technology to process large state spaces by establishing deep neural networks, in order to solve complex HVAC system control problems and effectively reduce energy costs.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A comprehensive energy HVAC control method based on deep reinforcement learning includes the following steps:

[0008] Step 1: Determine the control action range that the HVAC system can take;

[0009] Step 2: Collect status data of the building's HVAC system;

[0010] Step 3: Construct the control sequence, and use the DRL algorithm to update the HVAC control actions;

[0011] Step 4: Calculate the immediate and maximum cumulative returns after the HVAC system takes action;

[0012] Step 5: Use an artificial neural network to approximate the Q value and select the optimal control action.

[0013] As a preferred embodiment of the integrated energy HVAC control method based on deep reinforcement learning described in this invention, in step one, it is assumed that a building has z temperature zones and is equipped with a variable air flow volume (VAV) HVAC system. Each zone's VAV terminal box provides regulated air (typically at a constant temperature) with different discrete levels of airflow velocity, expressed as F = {f...} 1 ,f 2 ,…,f m Therefore, the entire motion space controlled by HVAC This includes all possible combinations of airflow velocity in each region, i.e., n = m z .

[0014] As a preferred embodiment of the integrated energy HVAC control method based on deep reinforcement learning described in this invention, in step two, the current time, regional temperature, and environmental disturbances are taken as the current system state, and state data is collected to determine the optimal control action; and combined with the current time information, the DRL algorithm can adapt to time-related activities, such as temperature requirements, electricity prices, resident activities, and equipment operation that change with time.

[0015] In addition to considering environmental disturbances, multi-step weather forecast data is also taken into account, not just the current ambient temperature and solar radiation; a shorter weather forecast data sequence is considered so that the DRL algorithm can capture environmental trends, perform proactive control, and adapt the system to changes over time.

[0016] As a preferred embodiment of the integrated energy HVAC control method based on deep reinforcement learning described in this invention, in step three, the DRL algorithm interacts with the building environment during operation and updates the control actions taken by the HVAC system according to the system state of the building; a single control step size Δt is used. c =kΔt s Represents the control frequency of the DRL algorithm; each Δt c Over time, the DRL algorithm will observe the building status. And update control action a t a t The expression is:

[0017]

[0018] Between two control time steps, the control actions operating the HVAC system remain consistent with the last updated actions. Δt sRepresents the building simulation or sensor sampling frequency, per Δt s At time, the building receives the control signal and enters the next state. t s t The expression is:

[0019]

[0020] at t-Δt s DRL observes the building status as Then, the control action to be taken in the next step will be updated to a. t Simultaneously take control actions To bring architecture to a new stage t At time t, take control action a. t Afterwards, architecture developed into a new stage.

[0021] As a preferred embodiment of the integrated energy HVAC control method based on deep reinforcement learning described in this invention, wherein: in step four, the objective of the DRL algorithm is to take a sequence of actions a1, a2, ..., a t}(a t ∈A), minimize the total energy cost while maintaining the temperature of each region within a satisfactory range; in state s t-1 Take action a t-1 Afterwards, architecture will develop into a new state. t The DRL algorithm will receive an immediate reward r. t The expression is:

[0022]

[0023] r t Including the previous control action a t-1 By considering the energy cost and the total penalty for violating the desired temperature, a trade-off can be found between minimizing the energy cost and maintaining the desired temperature. The return is maximized when the sum of the total energy cost and the total penalty is minimized.

[0024] Use the optimal value Q * (s t ,a t ) represents the state s t Take action a t The maximum cumulative return achievable is calculated recursively using the Bellman equation, where γ∈[0,1] is the decay factor controlling the window length when maximizing the return, expressed as:

[0025]

[0026] Because regional temperature is affected by various disturbances, the state transition of buildings is random and cannot be accurately measured. Therefore, this invention follows the Q-learning method to update the estimated values:

[0027]

[0028] Where η∈[0,1] represents the learning rate of the estimated value during training; in the MDP environment, the estimated value should gradually converge to the optimal value Q over time. * (s t ,a t ).

[0029] As a preferred embodiment of the integrated energy HVAC control method based on deep reinforcement learning described in this invention, in step five, an artificial neural network is used to approximate the Q value; under the neural network structure in the DRL framework, the Q value estimation of all control actions can be greatly improved by performing a forward pass calculation in the neural network, which greatly improves the efficiency when using the ∈-greedy strategy to select actions; the input value of the feature is the environmental state, and a linear rectified function is used as the activation function of the hidden layer. The linear layer infers the output action value to determine the optimal control action;

[0030] The mean squared error between the target Q-value and the neural network inference output is used as the loss function:

[0031]

[0032] Where n represents the number of possible control actions; the parameters in the neural network are updated using the mini-batch gradient descent method ω∶=ω-αΔω, where α is the learning rate, then:

[0033]

[0034] Consistent with the Q-learning update process, when using gradient descent, the target value Q... * (s t ,a t The Q-value can be estimated by the following equation, and is approximated by a neural network:

[0035]

[0036] State vector s t The possible combinations of each feature's values ​​form a very large state space, and the range of each value can vary greatly. To facilitate the learning process, the state vector is scaled to the range [0,1] before being input into the neural network. x represents a feature of the input state, and the minimum and maximum values ​​of each feature can be estimated from historical observations.

[0037]

[0038] For the output value, the linear layer infers the Q-value estimate from the hidden units; if the immediate reward function is used directly to calculate the target Q-value, it may result in a large variance in the target value; during backpropagation, the corresponding bias factor of the last linear layer may dominate the derivative of the loss function, which will prevent the weights of the previous layers from learning the optimal value; to address this limitation, the initial immediate reward is first shrunk by a factor ρ, and then pruned when the target is less than -1:

[0039]

[0040] This method compresses the initial target value with a large variance to the range of [-1,0], focusing on better control actions;

[0041] A state transition of one step size is represented by a tuple (s) t-1 ,a t-1 ,r t ,s t The target vector of the neural network is represented as target(s), which includes the previous state, previous action, immediate reward, and current state. t-1 ):

[0042]

[0043] Compare the target vector with the current inference output of the neural network, calculate the approximation error, and update the parameters in the neural network using RMSprop.

[0044] A multi-zone heuristic adaptive HVAC control method is proposed for application to the aforementioned integrated energy HVAC control method based on deep reinforcement learning. The method involves: training neural networks for each zone separately, with each neural network responsible for approximating the Q-value of its corresponding zone. At each time step, all networks receive the building's state and then determine the control actions for each zone. After executing the control actions, the penalty for each zone violating the desired temperature is calculated similarly to the immediate reward function, with the electricity cost being [calculated / implied]. i :

[0045]

[0046] Where cost represents the building's total electricity cost, u i This indicates the airflow rate in each area.

[0047] The beneficial effects of this invention are:

[0048] 1. The HVAC operation process is expressed as a Markov decision process, and a DRL-based HVAC control method is developed to minimize building energy costs and maintain comfortable temperatures for residents. To achieve greater scalability, a heuristic method is further proposed for the effective control of complex multi-zone systems, effectively reducing building energy costs.

[0049] 2. During building operation, the DRL-based HVAC control method learns effective control strategies based on input sensor data without relying on any thermodynamic model, thus avoiding the influence of random behavior of building temperature and the accuracy of thermodynamic models. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0051] Figure 1 This is a flowchart illustrating the integrated energy HVAC control method based on deep reinforcement learning proposed in this invention. Detailed Implementation

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0054] Reference Figure 1 As an embodiment of the present invention, a comprehensive energy HVAC control method based on deep reinforcement learning is provided, which includes the following steps:

[0055] Step 1: Determine the control action range that the HVAC system can take;

[0056] Specifically, suppose a building has z temperature zones and is equipped with a variable airflow volume (VAV) HVAC system. Each zone's VAV terminal box provides conditioned air (typically at a constant temperature) with different discrete levels of airflow velocity, expressed as F = f 1 ,f 2 ,…,fm Therefore, the entire motion space controlled by HVAC This includes all possible combinations of airflow velocity in each region, i.e., n = m z .

[0057] Step 2: Collect status data of the building's HVAC system;

[0058] Specifically, the current time, regional temperature, and environmental disturbances are taken as the current system state, and state data is collected to determine the optimal control action; and combined with the current time information, the DRL algorithm can adapt to time-related activities, such as temperature requirements, electricity prices, resident activities, and equipment operation that change over time.

[0059] In addition to considering environmental disturbances, multi-step weather forecast data is also taken into account, not just the current ambient temperature and solar radiation; a shorter weather forecast data sequence is considered so that the DRL algorithm can capture environmental trends, perform proactive control, and adapt the system to changes over time.

[0060] Step 3: Construct the control sequence, and use the DRL algorithm to update the HVAC control actions;

[0061] Specifically, the DRL algorithm interacts with the building environment during operation, updating the control actions taken by the HVAC system based on the building's system state; it uses a single control step size Δt. c =kΔt s Represents the control frequency of the DRL algorithm; each Δt c Over time, the DRL algorithm will observe the building status. And update control action a t a t The expression is:

[0062]

[0063] Between two control time steps, the control actions operating the HVAC system remain consistent with the last updated actions. Δt s Represents the building simulation or sensor sampling frequency, per Δt s At time, the building receives the control signal and enters the next state. t s t The expression is:

[0064]

[0065] at t-Δt s DRL observes the building status as Then, the control action to be taken in the next step will be updated to a. t Simultaneously take control actions To bring architecture to a new stage t At time t, take control action a. t Afterwards, architecture developed into a new stage.

[0066] Step 4: Calculate the immediate and maximum cumulative returns after the HVAC system takes action;

[0067] The goal of the DRL algorithm is to perform a sequence of actions {a1, a2, ..., a...} by taking a sequence of actions {a1, a2, ..., a...} t}(a t ∈A), minimize the total energy cost while maintaining the temperature of each region within a satisfactory range. In state s t-1 Take action a t-1 Afterwards, architecture will develop into a new state. t The DRL algorithm will receive an immediate reward r. t :

[0068]

[0069] r t Including the previous control action a t-1 By considering the energy cost and the total penalty for violating the desired temperature, a trade-off can be found between minimizing the energy cost and maintaining the desired temperature. The return is maximized when the sum of the total energy cost and the total penalty is minimized.

[0070] Use the optimal value Q * (s t ,a t ) represents the state s t Take action a t The maximum cumulative return that can be obtained is calculated recursively using the Bellman equation, where γ∈[0,1] is the decay factor that controls the window length when maximizing the return:

[0071]

[0072] Because regional temperature is affected by various disturbances, the state transition of buildings is random and cannot be accurately measured. Therefore, this invention follows the Q-learning method to update the estimated values:

[0073]

[0074] Where η∈[0,1] represents the learning rate of the estimated value during training; in the MDP environment, the estimated value should gradually converge to the optimal value Q over time. * (s t ,a t );

[0075] Step 5: Use an artificial neural network to approximate the Q value and select the optimal control action.

[0076] Specifically, within the neural network structure of the DRL framework, the Q-value estimation of all control actions can be significantly improved by performing a single forward pass computation within the neural network, greatly enhancing the efficiency of action selection using the ∈-greedy policy. The input value of the features is the environment state, and the Rectified Linear Unit (ReLU) function is used as the activation function for the hidden layers. The linear layers infer the output action value to determine the optimal control action.

[0077] The mean squared error between the target Q-value and the neural network inference output is used as the loss function:

[0078]

[0079] Where n represents the number of possible control actions; the parameters in the neural network are updated using the mini-batch gradient descent method ω∶=ω-αΔω, and α is the learning rate;

[0080]

[0081] Consistent with the Q-learning update process, when using gradient descent, the target value Q... * (s t ,a t The Q-value can be estimated by the following equation, and is approximated by a neural network:

[0082]

[0083] State vector s t The possible combinations of each feature's values ​​form a very large state space, and the range of each value can vary greatly. To facilitate the learning process, the state vector is scaled to the range [0,1] before being input into the neural network. x represents a feature of the input state, and the minimum and maximum values ​​of each feature can be estimated from historical observations.

[0084]

[0085] For the output value, the linear layer infers the Q-value estimate from the hidden units. However, directly using the immediate reward function to calculate the target Q-value can lead to a large variance in the target value. During backpropagation, the corresponding bias factor of the last linear layer may dominate the derivative of the loss function, which will prevent the weights of the earlier layers from learning the optimal value. To address this limitation, the initial immediate reward is first shrunken with a factor ρ, and then pruned when the target is less than -1:

[0086]

[0087] This method compresses the initial target value with a large variance to the range of [-1,0], focusing on better control actions.

[0088] A state transition of one step size is represented by a tuple (s) t-1 ,a t-1 ,r t ,s t The target vector of the neural network is represented as target(s), which includes the previous state, previous action, immediate reward, and current state. t-1 ):

[0089]

[0090] Compare the target vector with the current inference output of the neural network, calculate the approximation error, and update the parameters in the neural network using RMSprop.

[0091] It should be noted that, based on the current temperature and external environmental disturbances, the building HVAC system is operated to maintain the desired temperature in each area. The area temperature in the next time step is determined only by the current system state, environmental disturbances, and the regulated air input from the HVAC system, and is independent of the building's previous state. Therefore, the operation of HVAC control can be regarded as a Markov decision process.

[0092] The present invention also discloses a multi-zone heuristic adaptive HVAC control method that applies the above-mentioned deep reinforcement learning-based building HVAC system control method.

[0093] Let the cardinality of the action space be m. z The dimensions of the DRL algorithm increase with more areas and airflow levels, which greatly increases training time and reduces control performance. To address this, this invention proposes an adaptive HVAC control method to make the DRL algorithm applicable to multi-zone HVAC control.

[0094] The adaptive HVAC control method involves training separate neural networks for each zone, with each network responsible for approximating the Q-value of its corresponding zone, rather than using a single network to approximate the Q-value of all control actions in the building. At each time step, all networks receive the building's state and then determine the control actions for each zone. After executing the control actions, the penalty for each zone violating the desired temperature is calculated similarly to the immediate reward function, along with the electricity cost. i :

[0095]

[0096] Where cost represents the building's total electricity cost, ui This indicates the airflow rate in each area.

[0097] In summary, this invention expresses the HVAC operation process as a Markov decision process, develops a DRL-based HVAC control method, and simultaneously minimizes building energy costs and maintains comfortable temperatures for residents. To achieve greater scalability, a heuristic method is further proposed for the effective control of complex multi-zone systems. This effectively reduces building energy costs. During building operation, the DRL-based HVAC control method learns effective control strategies based on input sensor data without relying on any thermodynamic model, thus avoiding the influence of stochastic building temperature behavior and the accuracy of thermodynamic models.

[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A comprehensive energy heating, ventilation and air conditioning control method based on deep reinforcement learning, characterized in that, The method comprises the following steps: Step one: determining the control action space that the HVAC system can take; Step two: collecting state data of the building HVAC system; Step three: constructing a control sequence, and updating the HVAC control action by the DRL algorithm; Step four: calculating the immediate reward and the maximum cumulative reward after the HVAC system takes an action; Step five: using an artificial neural network to approximate the Q value and selecting the optimal control action; wherein in the step three, the DRL algorithm interacts with the building environment during the running process, and updates the control action taken by the HVAC system according to the system state of the building; a single control step is used representing the control frequency of the DRL algorithm, k represents a control decision period the sensor sampling period contained in the building the number of control actions, which defines the frequency of control action update relative to the frequency of state perception; each time, the DRL algorithm will observe the building state and update the control action , the expression of which is: ； wherein, a control policy function representing a deep reinforcement learning algorithm, the function being implemented by a trained artificial neural network; between two control time steps, the control action operating the HVAC system remains consistent with the last updated action; a building simulation or sensor sampling frequency, each time, the building receives a control signal and enters the next state , is expressed as: ； a dynamic function representing the building environment, i.e. a state transition model; in , the DRL observes the building state as , the control action to be taken in the next step is updated as ; the control action is taken , the building evolves to a new state ; at time t, the control action is taken , the building evolves to a new state ; In step four, the goal of the DRL algorithm is to take a series of actions. ( To minimize total energy costs while maintaining the temperature in each region within a satisfactory range; in the state Take action Afterwards, architecture will evolve into a new state. The DRL algorithm will then receive an immediate reward. The expression is: ； including the previous control action The total penalty of energy cost and violating the desired temperature, there is a trade-off between minimizing energy cost and maintaining the desired temperature, the return is maximized when the sum of total energy cost and total penalty is minimized; is the penalty term weight coefficient; it is a hyperparameter, used to balance the two goals: minimizing energy cost (cost) and minimizing the penalty for temperature deviation from the comfort interval, The larger the value, the more the algorithm will focus on maintaining temperature comfort; z represents the total number of regions divided in the building, and i represents the current region; represents the upper limit value of the desired temperature interval set for the i-th region at time step t, that is, the highest acceptable comfort temperature; represents the lower limit value of the desired temperature interval set for the i-th region at time step t, that is, the lowest acceptable comfort temperature; using the optimal value to represent the maximum cumulative reward that can be obtained in state taking action is computed recursively by the Bellman equation, where is a decay factor that controls the window length in maximizing the reward, expressed as: ； Since the state transition of the building is random and cannot be accurately measured due to the influence of various disturbances on the zone temperature, the method follows the Q-learning method to update the estimated value: ； wherein represents the learning rate of the estimate value during the training process; in the MDP environment, the estimate value should gradually converge to the optimal value over time ; In step five, the Q value is approximated by using an artificial neural network; under the neural network structure in the DRL framework, the Q value estimation of all control actions is calculated by performing a forward pass in the neural network, which greatly improves the efficiency when the action is selected by using the ϵ-greedy strategy, wherein the Q value estimation represents the approximate calculation result of the Q value by the neural network at a specific time; the input value of the feature is the environmental state, a linear rectifier function is used as the activation function of the hidden layer, and a linear layer is used to infer the output action value to determine the optimal control action.

2. The deep reinforcement learning based integrated energy HVAC control method of claim 1, wherein: In step one, assume a building has z temperature zones, equipped with variable air flow HVAC system, each zone's VAV terminal box provides conditioned air, with different discrete levels of air flow rate, denoted as where m is the total number of discrete air flow rate levels that individual zone VAV terminal box can adjust, thus the whole action space of HVAC control includes all possible combinations of air flow rate of each zone, the total number of n of all possible control actions of the whole system is calculated as .

3. The deep reinforcement learning based integrated energy HVAC control method of claim 2, wherein: In step two, the current time, zone temperature, and environmental disturbance are taken as the current system state to collect the state data for determining the optimal control action; and the current time information can make the DRL algorithm adapt to time-dependent activities, such as temperature requirements, electricity prices, household activities, and equipment operation that change over time; Among them, the multi-step weather forecast data is also considered for the environmental disturbance, not just the current environmental temperature and solar radiation; considering a shorter sequence of weather forecast data makes the DRL algorithm capture the trend of the environment, perform proactive control, and adapt to the system that changes over time.

4. The deep reinforcement learning based integrated energy HVAC control method of claim 1, wherein: The method further comprises: Using the mean square error between the target Q value and the neural network inference output as the loss function: ； where n represents the total number of all possible control actions of the whole system; the parameters in the neural network adopt a mini-batch gradient descent method updating, is a learning rate, is a set of all trainable parameters of the artificial neural network, then: ； In line with the Q-learning update procedure, when using gradient descent, the target value The Q-value can be approximated by a neural network as follows: ； state vector The combination of possible values for each feature in the state forms a very large state space, with each value ranging possibly over a large interval; to facilitate the learning process, the state vector is scaled to the interval [0, 1] before being input to the neural network, represents the input state for one feature, with the minimum and maximum values for each feature estimated from historical observations: ； For the output values, the linear layer extrapolates Q-value estimates from the hidden units; if the immediate return function is used directly to compute the target Q-values, it can result in target values with large variance; during backpropagation, the corresponding bias factor of the last linear layer can dominate the derivative of the loss function, which will prevent the weights of the earlier layers from learning optimal values; to address this limitation, which is caused by the high-variance returns and ultimately hinders the stable learning of the network, the following approach is used: first, scale the initial immediate returns by a factor shrink the initial immediate returns and then clip them when the target is less than -1 : ； wherein, , for training of the neural network, an updated "target Q value" is constructed according to the Bellman equation, which is a scalar combining the actual immediate reward received by the system and the estimate of the maximum potential value in the future; by this method, the initial target value with large variance is compressed into the range [-1, 0], focusing on better control actions; A one-step state transition is represented by a tuple including the previous state, the previous action, the immediate reward, and the current state; the target vector of the neural network is represented as : ； wherein, for constructing a target vector of the same dimension as the output of the neural network; comparing the target vector with the current inference output of the neural network, calculating an approximation error, and updating the parameters in the neural network using RMSprop.

5. A multi-zone heuristic adaptive HVAC control method applied to the deep reinforcement learning based integrated energy HVAC control method of claim 4, characterized in that: The neural networks for each zone are trained separately, each responsible for approximating the Q-values for its corresponding zone. At each time step, all networks will receive the state of the building and then determine the control action for each zone separately; after the control actions are executed, the calculation of the penalty for each zone violating the desired temperature is exactly the same as the temperature penalty summation term in the immediate reward function r t :​ ； wherein represents the total electricity cost of the building, represents the air flow of each zone.

Images