A Deep Reinforcement Learning-Based Adaptive Coordinated Control Method and Device for Building HVAC

Abstract

This invention discloses a method and apparatus for adaptive coordinated control of building HVAC systems based on deep reinforcement learning. The method involves: constructing an indoor temperature change model; describing indoor temperature changes using the indoor temperature change model; constructing a building thermodynamic model based on the indoor temperature change model; constructing a real-time dynamic control model for the building HVAC system based on the building thermodynamic model and combining the EnergyPlus and Python co-simulation framework; transforming the real-time dynamic control model of the building HVAC system into a Markov decision process model; constructing a deep reinforcement learning algorithm EB-TD3 based on set expert rules; and iteratively optimizing the Markov decision process model using the deep reinforcement learning algorithm EB-TD3 according to a set optimization objective, thereby achieving adaptive real-time dynamic control of the building HVAC system. This invention improves the control efficiency and stability of HVAC systems in complex environments, reduces energy consumption, and enhances thermal comfort.

Description

Technical Field

[0001] This invention relates to the field of building HVAC system optimization control technology, specifically to a building HVAC adaptive coordinated control method and device based on deep reinforcement learning. Background Technology

[0002] With rising building energy efficiency standards and the pursuit of low-carbon goals, building HVAC systems face increasingly severe challenges in optimizing energy efficiency and ensuring thermal comfort. Traditional HVAC system control methods rely heavily on fixed rules and experience, failing to flexibly respond to complex environmental changes and external disturbances. This results in insufficient optimization of building energy efficiency and difficulty in balancing energy efficiency and thermal comfort. Especially under complex operating conditions, HVAC systems need to effectively regulate indoor temperature and humidity to adapt to external weather changes, sudden load fluctuations, and equipment operational disturbances. Existing HVAC control strategies often neglect the impact of random disturbances on control effectiveness, leading to poor control accuracy and response speed.

[0003] Currently, deep reinforcement learning (DRL), as an intelligent optimization method, can learn optimal strategies through interaction with the environment and has shown good performance in many control problems. However, the control of HVAC systems faces significant challenges under complex operating conditions, especially under the influence of variable indoor and outdoor environmental conditions and random disturbances. Ensuring system stability, improving convergence speed, and maintaining indoor thermal comfort have always been research difficulties.

[0004] Therefore, how to invent a new adaptive control method for HVAC systems to improve the control efficiency and stability of HVAC systems in complex environments, reduce energy consumption, and enhance thermal comfort has become an urgent problem to be solved. Summary of the Invention

[0005] To this end, the present invention provides a building HVAC adaptive coordinated control method and device based on deep reinforcement learning. By combining deep reinforcement learning algorithms and expert knowledge, the control efficiency and stability of the HVAC system in complex environments are improved, energy consumption is reduced and thermal comfort is enhanced.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a building HVAC adaptive coordinated control method based on deep reinforcement learning, comprising:

[0007] Construct an indoor temperature change model; describe the indoor temperature change using the indoor temperature change model; construct a building thermodynamic model based on the indoor temperature change model;

[0008] Based on the aforementioned building thermodynamics model, and combined with the EnergyPlus and Python co-simulation framework, a real-time dynamic control model for the building HVAC system is constructed.

[0009] The real-time dynamic control model of the building HVAC system is transformed into a Markov decision process model.

[0010] Based on the established expert rules, a deep reinforcement learning algorithm EB-TD3 is constructed. According to the set optimization objective, the Markov decision process model is iteratively optimized through the deep reinforcement learning algorithm EB-TD3 to achieve adaptive real-time dynamic control of the building HVAC system.

[0011] As a preferred embodiment of the building HVAC adaptive coordinated control method based on deep reinforcement learning, the expression for the indoor temperature change model is as follows:

[0012]

[0013] In the formula, δt is the time increment; The temperature at the current time step; and These represent the temperatures of the first three time steps; O represents the approximation order, i.e., the error term.

[0014] As a preferred embodiment of the building HVAC adaptive coordinated control method based on deep reinforcement learning, the expression of the building thermodynamic model is as follows:

[0015]

[0016] C h =ρ air ×C ρ ×C T

[0017] In the formula, C h The heat capacity of indoor air; ρ air C is the density of air. ρ C is the specific heat capacity of air. T C is the volume of air. p Specific heat capacity at constant pressure; N l N represents the total number of hot zones. s N represents the total number of heat exchange surfaces. z h represents the total number of fluid branches. i A is the surface heat transfer coefficient; i T represents the surface area. si The room surface temperature; T h The room air temperature; T represents the mass flow rate of air entering from adjacent rooms. hi The temperature of the air in the adjacent room; The external air mass flow rate during infiltration; T ∞ The outside air temperature; It is the sum of external heat sources; This represents the sum of heat sources within the system.

[0018] As a preferred option for the adaptive coordinated control method of building HVAC based on deep reinforcement learning, the core elements of the Markov decision process model include: state space, action space, state transition probability, reward and discount factor;

[0019] The expression for the state space is:

[0020] S = {s0, s1, ..., s} t ,s t+1 ,...}

[0021] In the formula, S represents the state space; s t The state of the agent at time t;

[0022] The expression for the action space is:

[0023] A = {a0, a1, ..., a} n}

[0024] In the formula, A represents the action space; a n This indicates a specific control action;

[0025] The state transition probability represents the likelihood of a transition between states, reflecting the agent's ability to move from state s t Transfer to s t+1 The process;

[0026] The expression for the reward is:

[0027] R = {r0, r1, ..., r} t ,...,}

[0028] In the formula, R represents the reward; r t For an agent to be in state s at time t t And execute action a t The reward received at that time;

[0029] The discount factor is used to balance the importance of immediate rewards and future rewards.

[0030] As a preferred embodiment of the building HVAC adaptive coordinated control method based on deep reinforcement learning, in the process of iteratively optimizing the Markov decision process model using the deep reinforcement learning algorithm EB-TD3 according to the set optimization objective, the iterative optimization steps are as follows:

[0031] Initialize the parameters of the Actor network and the Critic network; based on the same parameters, construct the target network corresponding to the Actor network and the Critic network;

[0032] The current state is obtained through the Actor network, and a control action is determined based on the current state; noise is added when determining the control action to enhance the agent's exploration ability;

[0033] After each control action is executed, the environment provides a reward to the agent and transitions to the next state; the quadruple is stored in the experience replay buffer until the end of the round;

[0034] Mini-batch experiences are sampled from the experience replay buffer; the q-value of the target network is calculated using the mini-batch experiences;

[0035] The Critic network is updated according to its loss function.

[0036] The Actor network is updated based on the policy gradient theorem.

[0037] According to the soft update strategy, the target networks corresponding to the Actor network and the Critic network are updated.

[0038] This invention also provides a building HVAC adaptive coordinated control device based on deep reinforcement learning, which, based on the above-mentioned building HVAC adaptive coordinated control method based on deep reinforcement learning, includes:

[0039] The building thermodynamics model building module is used to construct an indoor temperature change model; describe indoor temperature changes through the indoor temperature change model; and construct a building thermodynamics model based on the indoor temperature change model.

[0040] The module for building a real-time dynamic control model for a building HVAC system is used to construct a real-time dynamic control model for a building HVAC system based on the building thermodynamic model and in conjunction with the EnergyPlus and Python co-simulation framework.

[0041] The Markov decision process model conversion module is used to convert the real-time dynamic control model of the building HVAC system into a Markov decision process model.

[0042] The module for constructing and iteratively optimizing the deep reinforcement learning algorithm EB-TD3 is used to construct the deep reinforcement learning algorithm EB-TD3 based on set expert rules; and to iteratively optimize the Markov decision process model through the deep reinforcement learning algorithm EB-TD3 according to the set optimization objective, so as to realize adaptive real-time dynamic control of the building HVAC system.

[0043] As a preferred embodiment of a building HVAC adaptive coordinated control device based on deep reinforcement learning, the expression for the indoor temperature change model in the building thermodynamics model construction module is as follows:

[0044]

[0045] In the formula, δt is the time increment; The temperature at the current time step; and These represent the temperatures of the first three time steps; O represents the approximation order, i.e., the error term.

[0046] As a preferred embodiment of a building HVAC adaptive coordinated control device based on deep reinforcement learning, the building thermodynamic model in the building thermodynamic model construction module is expressed as follows:

[0047]

[0048] C h =ρ air ×C ρ ×C T

[0049] In the formula, C h The heat capacity of indoor air; ρ air C is the density of air. ρ C is the specific heat capacity of air. T C is the volume of air. p Specific heat capacity at constant pressure; N l N represents the total number of hot zones. s N represents the total number of heat exchange surfaces. z h represents the total number of fluid branches. i A is the surface heat transfer coefficient; i T represents the surface area. si The room surface temperature; T h The room air temperature; T represents the mass flow rate of air entering from adjacent rooms. hi The temperature of the air in the adjacent room; The external air mass flow rate during infiltration; T ∞ The outside air temperature; It is the sum of external heat sources; This represents the sum of heat sources within the system.

[0050] As a preferred embodiment of a building HVAC adaptive coordinated control device based on deep reinforcement learning, the core elements of the Markov decision process model in the Markov decision process model transformation module include: state space, action space, state transition probability, reward, and discount factor.

[0051] The expression for the state space is:

[0052] S = {s0, s1, ..., s} t ,s t+1 ,...}

[0053] In the formula, S represents the state space; s t The state of the agent at time t;

[0054] The expression for the action space is:

[0055] A = {a0, a1, ..., a} n}

[0056] In the formula, A represents the action space; a n This indicates a specific control action;

[0057] The state transition probability represents the likelihood of a transition between states, reflecting the agent's ability to move from state s t Transfer to s t+1 The process;

[0058] The expression for the reward is:

[0059] R = {r0, r1, ..., r} t ,...,}

[0060] In the formula, R represents the reward; r t For an agent to be in state s at time t t And execute action a t The reward received at that time;

[0061] The discount factor is used to balance the importance of immediate rewards and future rewards.

[0062] As a preferred embodiment of the building HVAC adaptive coordinated control device based on deep reinforcement learning, the iterative optimization sub-module in the deep reinforcement learning algorithm EB-TD3 construction and iterative optimization module includes:

[0063] The network initialization submodule is used to initialize the parameters of the Actor network and the Critic network; based on the same parameters, it constructs the target network corresponding to the Actor network and the Critic network.

[0064] The control action determination submodule is used to obtain the current state through the Actor network and determine the control action based on the current state; noise is added when determining the control action to enhance the agent's exploration ability;

[0065] The iterative optimization submodule is used to provide a reward to the agent after each control action is executed and to transition to the next state; the quadruple is stored in the experience replay buffer until the end of the round;

[0066] The target network's q-value acquisition submodule is used to sample small batches of experience from the experience replay buffer; and to calculate the q-value of the target network using the small batches of experience.

[0067] The Critic network update submodule is used to update the Critic network according to the loss function of the Critic network;

[0068] The Actor network update submodule is used to update the Actor network based on the policy gradient theorem.

[0069] The target network update submodule is used to update the target networks corresponding to the Actor network and the Critic network according to a soft update strategy.

[0070] This invention has the following advantages: It constructs an indoor temperature change model; describes indoor temperature changes using this model; constructs a building thermodynamic model based on the model; constructs a real-time dynamic control model for the building HVAC system based on the thermodynamic model and a co-simulation framework using EnergyPlus and Python; transforms the real-time dynamic control model into a Markov decision process model; constructs a deep reinforcement learning algorithm EB-TD3 based on defined expert rules; and iteratively optimizes the Markov decision process model using the EB-TD3 algorithm according to the defined optimization objective, thereby achieving adaptive real-time dynamic control of the building HVAC system. This invention, by combining deep reinforcement learning algorithms and expert knowledge, can achieve adaptive adjustment of the HVAC system under complex environments and random disturbances, improving the flexibility and stability of the control strategy. Simultaneously, it can achieve a balance between energy saving and comfort, optimizing environmental parameters such as indoor temperature and humidity, ensuring energy saving while maintaining indoor thermal comfort, and meeting the dual needs of building energy efficiency and user comfort. In terms of control strategy, by introducing expert knowledge, temperature fluctuations caused during the exploration process are reduced, improving the convergence speed and training stability of the algorithm and ensuring long-term stable operation. The proposed control strategy can cope with complex changes in the building's internal and external environment, such as seasonal fluctuations and climate change, and dynamically adjust the operating parameters of the HVAC system to achieve optimal control effects. Through integration with simulation platforms such as EnergyPlus, the feasibility of the control method of this invention in practical applications has been verified, demonstrating its ability to effectively improve the operating efficiency of building HVAC systems. Attached Figure Description

[0071] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0072] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0073] Figure 1 This is a schematic diagram of the building HVAC adaptive coordinated control method based on deep reinforcement learning provided in Embodiment 1 of the present invention;

[0074] Figure 2 This is a schematic diagram of the building thermodynamic model modeling method in the building HVAC adaptive coordinated control method based on deep reinforcement learning provided in Embodiment 1 of the present invention;

[0075] Figure 3 This is a schematic diagram of the real-time dynamic control process implemented by the EnergyPlus model in the building HVAC adaptive coordinated control method based on deep reinforcement learning provided in Embodiment 1 of the present invention.

[0076] Figure 4 This is a schematic diagram of the Markov decision process model in the building HVAC adaptive coordinated control method based on deep reinforcement learning provided in Embodiment 1 of the present invention;

[0077] Figure 5 This is a schematic diagram of the architecture of the building HVAC adaptive coordination control device based on deep reinforcement learning provided in Embodiment 2 of the present invention. Detailed Implementation

[0078] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Example 1

[0080] See Figure 1 Embodiment 1 of the present invention provides a building HVAC adaptive coordinated control method based on deep reinforcement learning, comprising the following steps:

[0081] S1. Construct an indoor temperature change model; describe the indoor temperature change using the indoor temperature change model; construct a building thermodynamic model based on the indoor temperature change model;

[0082] S2. Based on the building thermodynamics model, and combined with the EnergyPlus and Python co-simulation framework, a real-time dynamic control model for the building HVAC system is constructed.

[0083] S3. Transform the real-time dynamic control model of the building HVAC system into a Markov decision process model;

[0084] S4. Based on the set expert rules, construct the deep reinforcement learning algorithm EB-TD3; according to the set optimization objective, iteratively optimize the Markov decision process model through the deep reinforcement learning algorithm EB-TD3 to realize adaptive real-time dynamic control of the building HVAC system.

[0085] In this embodiment, in step S1, an indoor temperature change model is constructed; the indoor temperature change is described using the indoor temperature change model; and a building thermodynamic model is constructed based on the indoor temperature change model.

[0086] The expression for the indoor temperature change model is as follows:

[0087]

[0088] In the formula, δt is the time increment; The temperature at the current time step; and These represent the temperatures of the first three time steps; O represents the approximation order, i.e., the error term.

[0089] In this embodiment, a building thermodynamics model is constructed based on the indoor temperature change model;

[0090] Specifically, for a building HVAC system, the modeling method is as follows: Figure 2 As shown, the geometric model was built using SketchUp, the thermal zones were constructed using OpenStudio, and the building's thermodynamic properties were assigned using EnergyPlus.

[0091] according to Figure 2 Therefore, there are five main factors influencing temperature changes in a building: the total convective heat load across all thermal zones, the convective heat transfer across room surfaces, heat transfer caused by air mixing between different thermal zones, heat transfer caused by external air infiltration, and the energy input to the air system. Thus, the expression for the building thermodynamic model is:

[0092]

[0093] C h =ρ air ×C ρ ×C T

[0094] In the formula, C h The heat capacity of indoor air; ρ air C is the density of air. ρ C is the specific heat capacity of air. T C is the volume of air. p Specific heat capacity at constant pressure; N l N represents the total number of hot zones. sN represents the total number of heat exchange surfaces. z h represents the total number of fluid branches. i A is the surface heat transfer coefficient; i T represents the surface area. si The room surface temperature; T h The room air temperature; T represents the mass flow rate of air entering from adjacent rooms. hi The temperature of the air in the adjacent room; The external air mass flow rate during infiltration; T ∞ The outside air temperature; It is the sum of external heat sources; This represents the sum of heat sources within the system.

[0095] In this embodiment, in step S2, based on the building thermodynamics model, a real-time dynamic control model of the building HVAC system is constructed by combining the EnergyPlus and Python co-simulation framework.

[0096] EnergyPlus simulations offer high accuracy. However, its real-time dynamic control implementation is limited by the ERL programming language used internally, thus restricting EnergyPlus's applicability as a simulation engine. To address this issue, secondary development was performed using EnergyPlus's Python interface, pyenergyplus. This allows for the definition of DRL algorithms in Python, enabling real-time dynamic control of the EnergyPlus model and further expanding its potential as a simulation engine. The specific implementation process is as follows: Figure 3 As shown.

[0097] In this embodiment, the modeling information section requires defining various parameters and the EnergyPlus model file. The EnergyPlus model file is obtained through the aforementioned modeling process. In the agent definition section, parameters such as meteorological information, simulation duration, and simulation time step need to be defined. Meteorological information is derived from typical meteorological year data, representing local meteorological characteristics. Simulation duration refers to the simulation time range. The simulation time step is δt. In defining the DRL algorithm, to achieve real-time dynamic control, the following need to be defined: control actions, state information, reward function, and expert knowledge. Control actions refer to controllable variables in the system, such as temperature setpoints and the on / off states of devices. These variables correspond to actuator setpoints in the EnergyPlus software. State information refers to state variables, i.e., information acquired by the system through sensors or other means. The reward function represents the system's optimization objective and is a specific decomposition of the optimization objective. Expert knowledge is used to guide and constrain the agent's exploratory behavior, thereby improving the algorithm's convergence speed and training stability.

[0098] In the simulation process section, simulation information and information defined in the DRL algorithm definition section will be used. Information from the DRL algorithm definition section will be passed to the Python environment for algorithm control. The control action information will be converted into actuator setpoints recognizable by EnergyPlus. Simulation information will then be transferred to EnergyPlus to complete simulation preparation.

[0099] In this embodiment, in step S3, the real-time dynamic control model of the building HVAC system is transformed into a Markov decision process model;

[0100] The core elements of the Markov decision process model include: state space, action space, state transition probability, reward, and discount factor.

[0101] The state space includes all potential states that the agent may exist in, and the expression for the state space is:

[0102] S = {s0, s1, ..., s} t ,s t+1 ,...}

[0103] In the formula, S represents the state space; s t The state of the agent at time t;

[0104] The action space comprises the set of all possible control actions that the agent can perform, and the expression for the action space is:

[0105] A = {a0, a1, ..., a} n}

[0106] In the formula, A represents the action space; a n This indicates a specific control action;

[0107] The state transition probability represents the likelihood of a transition between states, reflecting the agent's ability to move from state s t Transfer to s t+1 The process is implemented through a real-time dynamic model in EnergyPlus simulation.

[0108] The reward is defined by the user and represents the set of all possible rewards that the agent can obtain. The expression for the reward is:

[0109] R = {r0, r1, ..., r} t ,...,}

[0110] In the formula, R represents the reward; r t For an agent to be in state s at time t t And execute action a t The reward received at that time;

[0111] The discount factor is used to balance the importance of immediate rewards and future rewards.

[0112] In this embodiment, modeling the real-time dynamic control problem of the building's HVAC system as an MDP is crucial. The state variables are initially categorized into six main types: time, people, temperature, relative humidity, electricity, and weather. The state "time" includes "day type" and "current hour." The state "people" includes the number of people in each hot zone. The state "temperature" includes the temperature of each hot zone. The state "relative humidity" includes the relative humidity information for each hot zone. The state "electricity" includes the energy consumption of lighting, equipment, boilers, chillers, the HVAC system, and the building as a whole within each hot zone.

[0113] In the modeling of control actions, the control actions are defined as the heating and cooling temperature setpoints for each hot zone, as well as the on / off states of the boiler and chiller units. Given that the primary objective of a building HVAC control system is to minimize costs while maintaining thermal comfort, the reward function is constructed as a weighted sum of thermal comfort rewards and electricity price rewards, as shown in the following equation:

[0114] f = min(w1*PMV) reward +w2*Electricity reward )

[0115] In the formula, f represents the reward function corresponding to the optimization objective; w1 is the weighting coefficient of thermal comfort reward; w2 is the weighting coefficient of electricity price reward; PMVreward For thermal comfort bonus; Electricity reward This is for electricity price incentives.

[0116] The Predictive Average Votes (PMV) model, proposed by Fanger, is a widely used tool for assessing thermal comfort across various building types. Based on this, a dynamic management platform (MDP) for real-time control of building HVAC systems can be constructed, such as... Figure 4 As shown in the figure. Here, subscript 0 represents the initial state, and subscript t represents the final state.

[0117] In this MDP, the objective of real-time dynamic control of the building HVAC system is to maximize the cumulative reward, as shown in the following equation:

[0118] R t =r t +γr t+1 +γ 2 r t+2 +...

[0119] In the formula, γ is the discount factor.

[0120] Real-time dynamic control of a building's HVAC system should be able to implement appropriate control actions based on any given state to maximize cumulative rewards. In this process, the agent adopts an optimal strategy, denoted as π. * Accordingly, the general policy is defined as π. In the optimal policy π... * Under these conditions, the cumulative reward of the building HVAC control system is maximized, as shown in the following formula:

[0121]

[0122] Q(s,a) represents the action-value function, i.e., in state s t Next, execute action a t The expected cumulative reward afterward.

[0123] In this embodiment, in step S4, a deep reinforcement learning algorithm EB-TD3 is constructed based on the set expert rules; according to the set optimization objective, the Markov decision process model is iteratively optimized by the deep reinforcement learning algorithm EB-TD3 to realize adaptive real-time dynamic control of the building HVAC system.

[0124] Specifically, the control process of a building HVAC real-time control system for indoor temperature, thermal comfort, and electricity costs is transformed into a Markov decision process, which forms the basis for implementing the DRL algorithm. Furthermore, a deep reinforcement learning algorithm based on expert rules is proposed to achieve the best control effect.

[0125] In this embodiment, based on the set optimization objective, the Markov decision process model is iteratively optimized using the deep reinforcement learning algorithm EB-TD3 to achieve adaptive real-time dynamic control of the building HVAC system.

[0126] Specifically, the iterative optimization steps are as follows:

[0127] S41. Initialize the parameters of the Actor network and the Critic network; based on the same parameters, construct the target network corresponding to the Actor network and the Critic network;

[0128] Specifically, initialize the Actor network and the Critic network μ(s|θ) μ ) and Q(s,a|θ Q ), where θ μ and θ Q The parameters of the neural network are represented, and the corresponding target networks μ′ and Q′ are constructed, with their initial parameter values ​​being the same as those of the Actor and Critic networks. Finally, an experience replay buffer D is constructed to store the experience collected by the agent.

[0129] S42. Obtain the current state through the Actor network and determine the control action based on the current state; add noise when determining the control action to enhance the agent's exploration ability;

[0130] Specifically, the expression for the control action is:

[0131] a t =μ(s) t |θ μ )+N t

[0132] In the formula, a t To control the action; N t It is noise.

[0133] S43. After each control action is executed, the environment provides a reward to the agent and transitions to the next state; the quadruple is stored in the experience replay buffer until the end of the round;

[0134] Specifically, after each action is performed, the environment provides the agent with a reward r and transitions to the next state s. t+1 , the quadruple (s t ,a t ,r t ,s t+1 Stored in the experience replay buffer D until the end of the round.

[0135] S44. Sample a small batch of experience from the experience replay buffer; calculate the q value of the target network using the small batch of experience;

[0136] Specifically, the formula for calculating the q-value of the target network is:

[0137]

[0138] In the formula, y represents the target network Q-value, r represents the immediate reward, and γ represents the discount factor. θ' represents the output of the target Critic network. i Let π be the parameter of the i-th Critic target network. φ' is the Actor target network, φ' represents the network parameters, s' represents the state at the next time step, and ∈ represents the noise used for smoothing. This indicates taking the minimum value between the two networks.

[0139] S45. Update the Critic network according to its loss function;

[0140] Specifically, the loss function of the Critic network is:

[0141]

[0142] In the formula, L is the loss function; N is the batch size; y i Let q be the q-value of the target network in the i-th batch.

[0143] S46. Update the Actor network based on the policy gradient theorem;

[0144] Specifically, the update formula is as follows:

[0145]

[0146] In the formula, The gradient of the loss function; The gradient of the actor network; This represents the gradient of the critic network.

[0147] S47. Update the target networks corresponding to the Actor network and the Critic network according to the soft update strategy.

[0148] Specifically, the update formula is as follows:

[0149] θ Q′ ←τθ Q +(1-τ)θ Q′

[0150] θμ′ ←τθ μ +(1-τ)θ μ′

[0151] In the formula, τ represents the soft update factor.

[0152] In this embodiment, two expert rules are introduced in step S42. The first expert rule processes the reward by using the human thermal comfort PMV index and employing a segmented reward system. The range of [-0.5, 0.5] is the most comfortable, the range of abs[0.5, 1.5] is acceptable, and temperatures above this range are unacceptable. The expert rule establishes a buffer between the comfortable temperature and the extreme temperature, giving the reward function adaptive characteristics.

[0153] The second expert rule applies to action selection. In uninhabited buildings, more lenient rules are implemented, allowing the agent free exploration to improve adaptability. However, in inhabited buildings, expert rules are strictly enforced. Specifically, when the temperature deviates significantly from a comfort level, boilers and chillers are activated, and temperature setpoints are adjusted to restore the indoor temperature to an acceptable thermal comfort range. These expert rules ensure that the agent maintains its exploratory capabilities while ensuring thermal comfort. Furthermore, given the uncertainty of meeting room schedules and the sparsity of reward distribution, fixed temperature setpoints are used in both inhabited and uninhabited periods.

[0154] In one possible embodiment, a specific simulation implementation example is provided as follows:

[0155] Parameter settings:

[0156] Assume the simulation area is a 5km x 5km building in Tianjin, with dimensions of 17m, 12m, and 6m, containing three private offices, two open-plan offices, and a conference room. This area is divided into four thermal zones. The building envelope includes exterior walls, windows, doors, and interior walls, and their materials and thermal properties are shown in Table 1.

[0157] Material Name Thermal conductivity (W / m·K) Specific heat capacity (J / kg·K) Heavy concrete 1.95 900 polyurethane 0.0245 1590 concrete 1.11 920 Wood 0.15 1630 exterior windows 0.9 /

[0158] Table 1 Material parameters of building envelope

[0159] The single-person office is designed to accommodate one user. The population density of open-plan offices is 3.5 people / square meter, and the population density of meeting rooms is 2 people / square meter. On weekdays, the occupancy rate of these spaces gradually increases from 0.3 to a peak of 0.95 as people arrive, and then drops to 0 when no one is using them. Meeting room usage is irregular, with an average of 3 to 4 meetings held per week, each lasting 1 to 2 hours.

[0160] Regarding the HVAC system, the boiler and chiller have rated powers of 80kW and 40kW, respectively, with coefficients of performance (COPs) of 0.9 and 4.2. Both the boiler and chiller are electrically driven. Supply and return water temperatures, as well as the settings for fans and cooling towers, have been optimized to improve system performance. Comparative algorithms used in the experiments include a rule-based control algorithm and the baseline DRL algorithm TD3. Experimental scenarios cover typical winter and summer months.

[0161] Analysis of the results of the example:

[0162] The performance metrics for the algorithm include total cost, electricity cost, thermal comfort cost, energy consumption, and carbon dioxide emissions. The results of the example calculations are shown in Table 2.

[0163] method season Total cost (yuan) Electricity cost (yuan) Comfort cost (RMB) Energy consumption (kWh) CO2 (kg) Rule 19975.74 16493.92 3481.82 18208.07 18153.45 TD3 winter 16597.14 14124.33 2472.81 14119.68 14157.08 EB-TD3 16011.35 14016.61 1994.74 14052.78 14010.62 Rule 3497.16 2885.65 611.51 2793.96 2710.14 TD3 summer 2990.02 2730.17 259.85 2729.53 2647.64 EB-TD3 2618.54 2409.41 209.13 2517.67 2442.14

[0164] As shown in Table 2:

[0165] In winter, compared to rule-based control algorithms, the EB-TD3 algorithm reduced total cost by 19.84%, improved thermal comfort by 42.71%, reduced HVAC system energy consumption by 22.82%, and reduced monthly CO2 emissions by 4,142.83 kg. Compared to the baseline TD3 algorithm, the EB-TD3 algorithm reduced total cost by 3.53%, electricity cost by 0.76%, improved thermal comfort by 19.33%, reduced energy consumption by 1.03%, and reduced monthly CO2 emissions by 146.46 kg.

[0166] In summer, compared to rule-based control methods, the EB-TD3 algorithm reduced overall costs by 25.12%, electricity costs by 16.50%, thermal comfort by 65.80%, building HVAC system energy consumption by 9.89%, and monthly CO2 emissions by 268.00 kg. Compared to the baseline TD3 algorithm, the method proposed in this invention achieves a 12.42% reduction in total costs, an 11.75% reduction in electricity costs, a 19.52% reduction in thermal comfort costs, a 7.76% reduction in HVAC system energy consumption, and a 205.50 kg reduction in monthly CO2 emissions.

[0167] In summary, this invention constructs an indoor temperature change model; describes indoor temperature changes using this model; constructs a building thermodynamic model based on the indoor temperature change model; constructs a real-time dynamic control model for the building HVAC system based on the building thermodynamic model and using the EnergyPlus and Python co-simulation framework; transforms the real-time dynamic control model of the building HVAC system into a Markov decision process model; constructs a deep reinforcement learning algorithm EB-TD3 based on set expert rules; and iteratively optimizes the Markov decision process model using the deep reinforcement learning algorithm EB-TD3 according to the set optimization objective, thereby achieving adaptive real-time dynamic control of the building HVAC system. This invention, by combining deep reinforcement learning algorithms and expert knowledge, can achieve adaptive adjustment of the HVAC system under complex environments and random disturbances, improving the flexibility and stability of the control strategy. Simultaneously, it can achieve a balance between energy saving and comfort, optimizing environmental parameters such as indoor temperature and humidity, ensuring energy saving while maintaining indoor thermal comfort, and meeting the dual needs of building energy efficiency and user comfort. In terms of control strategy, by introducing expert knowledge, temperature fluctuations caused during the exploration process are reduced, improving the convergence speed and training stability of the algorithm and ensuring long-term stable operation. The proposed control strategy can cope with complex changes in the building's internal and external environment, such as seasonal fluctuations and climate change, and dynamically adjust the operating parameters of the HVAC system to achieve optimal control effects. Through integration with simulation platforms such as EnergyPlus, the feasibility of the control method of this invention in practical applications has been verified, demonstrating its ability to effectively improve the operating efficiency of building HVAC systems.

[0168] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0169] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] Example 2

[0171] See Figure 5Embodiment 2 of the present invention also provides a building HVAC adaptive coordinated control device based on deep reinforcement learning, comprising:

[0172] The building thermodynamics model construction module 001 is used to construct an indoor temperature change model; describe indoor temperature changes through the indoor temperature change model; and construct a building thermodynamics model based on the indoor temperature change model.

[0173] The building HVAC system real-time dynamic control model construction module 002 is used to construct a building HVAC system real-time dynamic control model based on the building thermodynamic model and combined with the EnergyPlus and Python co-simulation framework.

[0174] Markov decision process model conversion module 003 is used to convert the real-time dynamic control model of the building HVAC system into a Markov decision process model.

[0175] The EB-TD3 deep reinforcement learning algorithm construction and iterative optimization module 004 is used to construct the EB-TD3 deep reinforcement learning algorithm based on set expert rules; and to iteratively optimize the Markov decision process model through the EB-TD3 deep reinforcement learning algorithm according to the set optimization objective, so as to realize adaptive real-time dynamic control of the building HVAC system.

[0176] In this embodiment, the expression for the indoor temperature change model in the building thermodynamics model construction module 001 is as follows:

[0177]

[0178] In the formula, δt is the time increment; The temperature at the current time step; and These represent the temperatures of the first three time steps; O represents the approximation order, i.e., the error term.

[0179] In this embodiment, the expression of the building thermodynamic model in the building thermodynamic model construction module 001 is:

[0180]

[0181] C h =ρ air ×C ρ ×C T

[0182] In the formula, C h The heat capacity of indoor air; ρ air C is the density of air. ρ C is the specific heat capacity of air. T C is the volume of air.p Specific heat capacity at constant pressure; N l N represents the total number of hot zones. s N represents the total number of heat exchange surfaces. z h represents the total number of fluid branches. i A is the surface heat transfer coefficient; i T represents the surface area. si T represents the room surface temperature. h The room air temperature; T represents the mass flow rate of air entering from adjacent rooms. hi The temperature of the air in the adjacent room; The external air mass flow rate during infiltration; T ∞ The outside air temperature; It is the sum of external heat sources; This represents the sum of heat sources within the system.

[0183] In this embodiment, the core elements of the Markov decision process model in the Markov decision process model transformation module 003 include: state space, action space, state transition probability, reward, and discount factor.

[0184] The expression for the state space is:

[0185] S = {s0, s1, ..., s} t ,s t+1 ,...}

[0186] In the formula, S represents the state space; s t The state of the agent at time t;

[0187] The expression for the action space is:

[0188] A = {a0, a1, ..., a} n}

[0189] In the formula, A represents the action space; a n This indicates a specific control action;

[0190] The state transition probability represents the likelihood of a transition between states, reflecting the agent's ability to move from state s t Transfer to s t+1 The process;

[0191] The expression for the reward is:

[0192] R = {r0, r1, ..., r} t ,...,}

[0193] In the formula, R represents the reward; r t For an agent to be in state s at time t tAnd execute action a t The reward received at that time;

[0194] The discount factor is used to balance the importance of immediate rewards and future rewards.

[0195] In this embodiment, the iterative optimization submodule in the EB-TD3 deep reinforcement learning algorithm construction and iterative optimization module 004 includes:

[0196] The network initialization submodule 041 is used to initialize the parameters of the Actor network and the Critic network; based on the same parameters, it constructs the target network corresponding to the Actor network and the Critic network.

[0197] The control action determination submodule 042 is used to obtain the current state through the Actor network and determine the control action based on the current state; noise is added when determining the control action to enhance the agent's exploration ability;

[0198] The iterative optimization submodule 043 is used to provide a reward to the agent after each control action is executed, and to transition to the next state; and to store the quadruple in the experience replay buffer until the end of the round;

[0199] The target network q-value acquisition submodule 044 is used to sample small batches of experience from the experience replay buffer; and calculate the q-value of the target network using the small batches of experience.

[0200] The Critic network update submodule 045 is used to update the Critic network according to the loss function of the Critic network;

[0201] Actor network update submodule 046 is used to update the Actor network based on the policy gradient theorem;

[0202] The target network update submodule 047 is used to update the target network corresponding to the Actor network and the Critic network according to a soft update strategy.

[0203] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0204] Example 3

[0205] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code of a building HVAC adaptive coordinated control method based on deep reinforcement learning. The program code includes instructions for executing the building HVAC adaptive coordinated control method based on deep reinforcement learning of Embodiment 1 or any possible implementation thereof.

[0206] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives, SSDs).

[0207] Example 4

[0208] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0209] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the deep reinforcement learning-based adaptive coordinated control method for building HVAC in Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0210] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0211] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0212] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0213] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A building HVAC adaptive coordinated control method based on deep reinforcement learning, characterized in that, include: Construct an indoor temperature change model; describe the indoor temperature change using the indoor temperature change model; A building thermodynamic model is constructed based on the aforementioned indoor temperature change model; Based on the aforementioned building thermodynamics model, and combined with the EnergyPlus and Python co-simulation framework, a real-time dynamic control model for the building HVAC system is constructed. The real-time dynamic control model of the building HVAC system is transformed into a Markov decision process model. Based on the established expert rules, a deep reinforcement learning algorithm EB-TD3 is constructed. Based on the set optimization objective, the Markov decision process model is iteratively optimized using the deep reinforcement learning algorithm EB-TD3 to achieve adaptive real-time dynamic control of the building HVAC system. The expression for the indoor temperature change model is: ； In the formula, δt is the time increment; The temperature at the current time step; , and These represent the temperatures at the first three time steps; O represents the approximation order, i.e., the error term, indicating that when... When the value is very small, the approximate error of the derivative is approximately equal to... Same order; The expression for the building thermodynamic model is: ； ； In the formula, C h The heat capacity of indoor air; ρ air C is the density of air. ρ C is the specific heat capacity of air. T C is the volume of air. p Specific heat capacity at constant pressure; N l N represents the total number of hot zones. s N represents the total number of heat exchange surfaces. z h represents the total number of fluid branches. i A is the surface heat transfer coefficient; i T represents the surface area. si The room surface temperature; T h The room air temperature; T represents the mass flow rate of air entering from adjacent rooms. hi The temperature of the air in the adjacent room; The external air mass flow rate during infiltration; T ∞ The outside air temperature; It is the sum of external heat sources; This represents the sum of heat sources within the system. The core elements of the Markov decision process model include: state space, action space, state transition probability, reward, and discount factor. The expression for the state space is: ； In the formula, S represents the state space; s t The state of the agent at time t; The expression for the action space is: ； In the formula, A represents the action space; a n This indicates a specific control action; The state transition probability represents the likelihood of a transition between states, reflecting the agent's ability to move from state s t Transfer to s t+1 The process; The expression for the reward is: ； In the formula, R represents the reward; r t For an agent to be in state s at time t t And execute action a t The reward received at that time; The discount factor is used to balance the importance of immediate rewards and future rewards; In the process of iteratively optimizing the Markov decision process model using the deep reinforcement learning algorithm EB-TD3 according to the set optimization objective, the iterative optimization steps are as follows: Initialize the parameters of the Actor network and the Critic network; based on the same parameters, construct the target network corresponding to the Actor network and the Critic network; The current state is obtained through the Actor network, and a control action is determined based on the current state; noise is added when determining the control action to enhance the agent's exploration ability; After each control action is executed, the environment provides a reward to the agent and transitions to the next state; the quadruple is stored in the experience replay buffer until the end of the round; Mini-batch experiences are sampled from the experience replay buffer; the q-value of the target network is calculated using the mini-batch experiences; The Critic network is updated according to its loss function. The Actor network is updated based on the policy gradient theorem. According to the soft update strategy, the target networks corresponding to the Actor network and the Critic network are updated.

2. A building HVAC adaptive coordinated control device based on deep reinforcement learning, employing the building HVAC adaptive coordinated control method based on deep reinforcement learning as described in claim 1, characterized in that, include: The building thermodynamics model construction module is used to construct the indoor temperature change model as described in claim 1; and to describe the indoor temperature change through the indoor temperature change model. A building thermodynamic model is constructed based on the aforementioned indoor temperature change model; The module for constructing a real-time dynamic control model for a building HVAC system is used to construct a real-time dynamic control model for a building HVAC system based on the building thermodynamic model described in claim 1, combined with the EnergyPlus and Python co-simulation framework. A Markov decision process model conversion module is used to convert the real-time dynamic control model of the building HVAC system described in claim 1 into a Markov decision process model. The module for constructing and iteratively optimizing the deep reinforcement learning algorithm EB-TD3 is used to construct the deep reinforcement learning algorithm EB-TD3 as described in claim 1 based on set expert rules. Based on the set optimization objectives, the Markov decision process model is iteratively optimized using the deep reinforcement learning algorithm EB-TD3 to achieve adaptive real-time dynamic control of the building HVAC system.

Images