A commercial building HVAC control method and system based on evolutionary deep reinforcement learning
By transforming the HVAC control model into Markov game problem and applying the multi-agent attention evolution deep reinforcement learning algorithm, the problems of medium and low scalability and low performance of HVAC control in multi-regional commercial buildings are solved, and efficient energy management is achieved and comfort control.
Patent Information
- Application Number
- CN202210631141.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-12-13
- Filing Date
- 2022-06-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-06-06
AI Technical Summary
The existing multi-agent deep reinforcement learning methods have low scalability and low performance problems when applied to HVAC control of multi-regional commercial buildings, resulting in poor algorithm stability and poor optimization performance.
Using an evolutionary deep reinforcement learning method, the HVAC control model is transformed into Markov game problem, and the multi-agent attention evolution deep reinforcement learning algorithm is used to solve it in parallel, and the optimal control strategy for multiple groups of agent populations is trained, and the The deep neural network weight reorganization and screening can obtain the optimal control strategy of air valve agents and regional air supply agents for online deployment.
High scalability and significantly reduce HVAC energy costs while maintaining high thermal comfort and high air quality comfort.
Smart Images

Figure CN115017584B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the interdisciplinary field of artificial intelligence and building energy management, and in particular to a commercial building HVAC control method and system based on evolutionary deep reinforcement learning. Background Art
[0002] In China, building energy consumption has increased at an annual rate of about 10% over the past 20 years. According to official statistics, in 2018, the total energy consumption of buildings accounted for 46.5% of the total energy consumption in the country, and the total carbon emissions from buildings accounted for 51.3% of the total carbon emissions in the country. Among them, commercial buildings are large electricity users, and about 40% to 50% of the total electricity consumption is attributed to heating, ventilation, and air conditioning (HVAC) systems. This brings a great economic burden to commercial building operators.
[0003] Existing studies have proposed many methods to reduce HVAC energy costs, such as stochastic programming, Lyapunov optimization, model predictive control, and robust optimization. However, the above methods have two shortcomings: (1) they require prior information about uncertain parameters or prediction of uncertain parameters; (2) they require a clear building thermal dynamics model. However, since building thermal dynamics are related to many factors such as building materials, external weather, and indoor thermal disturbances, it is very challenging to establish a building thermal dynamics model that is both accurate and easy to control. With the development of Internet of Things technology and machine learning technology, data-driven energy optimization methods have become possible. Common methods include reinforcement learning, deep Q-network, deep deterministic policy gradient, asynchronous dominant actor-critic, etc. Although the above learning-based HVAC control methods have made some progress, none of them considers the regional coordination problem in multi-regional commercial buildings. In fact, there is heat conduction between different regions in multi-regional commercial buildings, and the costs associated with HVAC energy consumption lead to coupling between air supply rate decisions and damper position decisions between different regions. To solve the above problems, a possible solution is to use multi-agent deep reinforcement learning (MARL), such as the multi-agent deep deterministic policy gradient algorithm (MADDPG). However, as the number of agents increases, the problem becomes more complex and the variance of the policy gradient may grow exponentially, which in turn leads to poor algorithm stability and ultimately affects the optimization performance. Summary of the invention
[0004] The purpose of the present invention is to provide a commercial building HVAC control method and system based on evolutionary deep reinforcement learning, so as to solve the problems of low scalability and low performance when the existing HVAC control method using multi-agent deep reinforcement learning is applied to multi-area commercial buildings.
[0005] The present invention adopts the following technical solutions to achieve the above-mentioned invention object:
[0006] The present invention provides a commercial building HVAC control method based on evolutionary deep reinforcement learning, comprising:
[0007] Access pre-built multi-zone commercial building HVAC control models;
[0008] The HVAC control model is transformed into a Markov game problem involving the damper agent and the district air supply agent.
[0009] The multi-agent attention evolution deep reinforcement learning algorithm is used to solve the Markov game problem in parallel, and finally the optimal control strategy of multiple groups of intelligent agents is trained;
[0010] Use the optimal control strategies of multiple groups of intelligent agent populations obtained through training to perform performance tests and select several groups of intelligent agent populations for the test phase;
[0011] Reorganize the weights of the deep neural networks of several selected groups of intelligent agent populations;
[0012] The reorganized deep neural network weights are used as the initial weights of a larger multi-agent population model, and the larger Markov game problem is repeatedly solved until the size of the agent population is equal to the sum of the number of regions and the number of wind valves. A group of multi-agent populations with the highest fitness is screened out, and then the optimal control strategy for the wind valve agent and the regional wind supply agent is obtained.
[0013] The HVAC optimal control strategy for the air valve intelligent agent and the regional air supply intelligent agent is deployed online.
[0014] Further, the HVAC control model includes an objective function, decision variables and constraints;
[0015] The objective function expression is as follows:
[0016]
[0017] Where: Φ 1,t represents the energy cost associated with the blower, Φ 2,t represents the energy cost associated with the cooling coil;
[0018] Where: The energy cost associated with the blower is expressed as:
[0019]
[0020] Where: Approximately the energy consumption associated with the blower, μ is a constant coefficient, m i,t represents the air supply rate in time slot t in region i, pt represents the electricity price in time slot t, and τ represents the duration of a time slot;
[0021] The energy cost associated with the cooling coil is expressed as:
[0022] Φ 2,t =P t p t τ (3)
[0023] Where: P t is the energy consumption of the cooling coil in time slot t, expressed as:
[0024]
[0025] Where: C a represents the specific heat capacity of air, η is the efficiency coefficient of the cooling coil, COP is the performance coefficient associated with the cooler, is the temperature of the mixed air at time slot t, T s is the air supply temperature of the blower;
[0026] The decision variables include the air supply rate m for each zone i,t and the damper position of the air handling unit in HVAC t , the expression is as follows:
[0027] m i,t ∈{m 1 ,m 2 ,...,m M} (5)
[0028]
[0029] Where: m i,t Its value can be selected from M discrete levels, the total air supply rate To be less than the sum of the maximum air supply rates of all zones, σ t The value of can be selected from Z discrete levels:
[0030] σ t ∈{σ 1 ,σ 2 ,...,σ Z} (7)
[0031]
[0032] When σ t = 0, it means that there is only outdoor fresh air in the mixed air, and when σ t =1, it means that the mixed air comes entirely from the return air of each area;
[0033] The constraints include those related to indoor temperature and carbon dioxide concentration, and the expressions are as follows:
[0034] like represents the indoor temperature of area i at time slot t, then:
[0035]
[0036] Where: K i,t represents the number of people indoors in area i at time slot t, T i min and T i max They represent the minimum and maximum values of the comfortable temperature range of area i respectively;
[0037] cb i,t represents the carbon dioxide concentration in region i at time slot t, then:
[0038]
[0039] Where: Indicates the upper limit of the comfortable range of indoor carbon dioxide concentration in area i.
[0040] Furthermore, the Markov game problem includes state, action and reward functions:
[0041] The environmental state S t The expression is as follows:
[0042] S t =(o 1,t ,o 2,t ,...,o n,t ,o *,t ) (11)
[0043] In the formula: o 1,t ,o 2,t ,...,o n,t Middle i,t (1≤i≤n) represents the observed state of the agent related to the air supply rate decision in time slot t in region i, o *,t represents the observed state of the master control agent related to the air valve position decision at time slot t, o *,t =(cb 1,t ,...,cb i,t ,...,cb n,t ,K 1,t ,...,K i,t ,...,K n,t ), T t out is the outdoor temperature in area i at time slot t, Tt in is the indoor temperature in area i at time slot t, is the indoor temperature of the adjacent j area in time slot t of area i, N i represents the set of adjacent regions of region i, p t is the electricity price of time slot t, t' is the relative time of the current absolute time of time slot t within a day, cb i,t is the carbon dioxide concentration in region i at time slot t, K i,t is the number of occupants in time slot t in area i;
[0044] The action A t The expression is as follows:
[0045] A t =(m 1,t ,...,m i,t ,...,m n,t ,β t ) (12)
[0046] Where: m i,t is the air supply rate of HVAC zone i at time slot t, β t The air valve position of HVAC;
[0047] The reward function R t The expression is as follows:
[0048]
[0049]
[0050] Where: R 1,i,t is the reward obtained by the agent related to the air supply rate decision in area i and time slot t, R 2,i,t is the reward obtained by the master control agent related to the damper position decision at time slot t, α is the importance coefficient of the penalty cost caused by the HVAC energy cost to the comfortable temperature range deviation, β is the importance coefficient of the penalty cost caused by the indoor carbon dioxide concentration exceeding the upper limit to the penalty cost caused by the comfortable temperature range deviation, C 1,i,t (o i,t ) is the penalty cost caused by violating the indoor comfortable temperature range in time slot t in area i, is the penalty related to the energy cost of the fan in area i and time slot t, is the penalty associated with the cooling coil energy cost for time slot t in zone i, C 4,i,t (o i,t ) is the penalty cost caused by violating the upper limit of indoor carbon dioxide concentration in area i during time slot t, is the penalty associated with the cooling coil energy cost for all zones t time slots, C6,t (o *,t ) is the penalty cost caused by violating the upper limit of indoor carbon dioxide concentration in all areas during time slot t.
[0051] Furthermore, the network architecture used in the multi-agent attention evolution deep reinforcement learning algorithm includes an actor network and a critic network, and both the actor network and the critic network adopt an attention embedding module, and the output v of the attention embedding module i The expression is as follows:
[0052]
[0053] Where: f i (o j ,a j )(i≠j) represents the observed state o of the receiving environment j and action a of agent j j The output value of i,j Calculated by the following formula:
[0054]
[0055] Where: and W Φ is the parameter to be learned, β i,j The correlation between the embeddings of agent i and all other agents j is calculated by inner product, and then β is quantized by softmax function. i,j Normalize to get α i,j .
[0056] Furthermore, the optimal control strategies of the multiple groups of intelligent agent populations obtained through training are used to perform performance tests, and the method of selecting several groups of intelligent agent populations for the test phase includes:
[0057] The weight parameters of the deep neural network are superimposed with a Gaussian random variable with a mean of 0 and a variance of the weight, and loaded into the deep neural network; the parallel multi-agent population after adding disturbances to the weight parameters is screened;
[0058] On the premise of meeting the requirements of average temperature deviation and average carbon dioxide concentration deviation, fitness ranking and selection are carried out according to the size of energy cost.
[0059] Furthermore, the method for reorganizing the deep neural network weights of the selected groups of agent populations includes:
[0060] K groups {K1,...,K i ,...,K k}After the multi-agent population is screened;
[0061] Group C with high fitness The weight parameters of the deep neural network of the multi-agent population are copied and mixed two by two to form several new groups {C 1x1 ,C 1x2 ,...,C 1xc ,...,C ixi -1,...,C ixc ,...,C cxc}, at this time the number of agents in each group increases from n to 2n;
[0062] If the required number of agents is still not reached, K groups are randomly selected to continue parallel training.
[0063] Furthermore, the method for online deployment of the HVAC optimal control strategy of the air valve intelligent body and the regional air supply intelligent body includes:
[0064] Collect the observation status of each current time slot;
[0065] According to the observed state and HVAC optimal control strategy, the respective actions are selected, and then the decision actions of the air valve control position and the air supply rate of each area are obtained;
[0066] The obtained decision actions are deployed to the actual HVAC system for online control.
[0067] The present invention provides a commercial building HVAC control system based on evolutionary deep reinforcement learning, comprising:
[0068] Acquisition module: used to acquire pre-established multi-zone commercial building HVAC control models;
[0069] Design module: used to transform the HVAC control model into a Markov game problem involving damper agents and zone air supply agents;
[0070] Solution training module: used to solve Markov game problems in parallel using a multi-agent attention evolutionary deep reinforcement learning algorithm, and ultimately train the optimal control strategy for multiple groups of intelligent agent populations;
[0071] Screening module: used to perform performance testing using the optimal control strategies of multiple groups of intelligent agent populations obtained through training, and screen out several groups of intelligent agent populations for the test phase;
[0072] Reorganization module: used to reorganize the deep neural network weights of several selected groups of intelligent agent populations;
[0073] Solution screening module: The reorganized deep neural network weights are used as the initial weights of a larger multi-agent population model, and the larger Markov game problem is repeatedly solved until the size of the agent population is equal to the sum of the number of regions and the number of air valves. A group of multi-agent populations with the highest fitness is screened out, and then the optimal control strategy for the air valve agent and the regional wind supply agent is obtained;
[0074] Deployment module: used to deploy the HVAC optimal control strategy of the air valve intelligent agent and the regional air supply intelligent agent to the actual HVAC system for online control.
[0075] The beneficial effects of the present invention are as follows:
[0076] The method of the present invention adopts the attention mechanism and evolutionary learning ideas in the deep neural network training process, so its computational complexity will not increase sharply with the increase of the number of intelligent agents, and thus has high scalability;
[0077] Compared with the existing ON-OFF control strategy and the control strategy based on the existing deep reinforcement learning algorithm, the method of the present invention can significantly reduce the HVAC energy cost while maintaining high thermal comfort and high air quality comfort. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] Figure 1 It is a flow chart of a commercial building HVAC control method based on multi-agent attention evolutionary deep reinforcement learning provided by an embodiment of the present invention;
[0079] Figure 2 : is a comparison diagram of the total air supply rate regulation of the commercial building HVAC control method based on multi-agent attention evolution deep reinforcement learning provided by an embodiment of the present invention and other methods;
[0080] Figure 3 This is a comparison chart of various aspects of fitness tests of the commercial building HVAC control method based on multi-agent attention evolutionary deep reinforcement learning provided by an embodiment of the present invention and other methods. DETAILED DESCRIPTION
[0081] In order to further explain the technical means adopted by the present invention and the effects achieved, the technical scheme of the present invention is clearly and completely described below in conjunction with the accompanying drawings and preferred embodiments.
[0082] like Figure 1 As shown, the present invention provides a commercial building HVAC control method and system based on evolutionary deep reinforcement learning, and the specific steps are as follows:
[0083] Step 1: Obtain a pre-built multi-zone commercial building HVAC control model;
[0084] Step 2: Convert the HVAC control model into a Markov game problem involving the air valve agent and the zone air supply agent;
[0085] Step 3: Use the multi-agent attention evolutionary deep reinforcement learning algorithm to solve the Markov game problem in parallel, and finally train the optimal control strategy for multiple groups of intelligent agents;
[0086] Step 4: Use the optimal control strategies of multiple groups of intelligent agent populations obtained through training to perform performance tests and select several groups of intelligent agent populations for the test phase;
[0087] Step 5: Reorganize the weights of the deep neural network of several selected groups of intelligent agent populations;
[0088] Step 6: Use the reorganized deep neural network weights as the initial weights of a larger multi-agent population model, and go to step 3, repeating steps 3, 4, 5, and 6. When the size of the agent population is equal to the sum of the number of regions and the number of air valves, execute step 7;
[0089] Step 7: Filter out a group of multi-agent populations with the highest fitness, and then obtain the HVAC optimal control strategy of the air valve agent and the regional air supply agent;
[0090] Step 8: Deploy the obtained HVAC optimal control strategies of the air valve agent and the regional air supply agent to the actual HVAC system for online control.
[0091] In step 1 above, the HVAC control model of commercial buildings in four regions is constructed. For the HVAC energy cost minimization problem, the stochastic programming problem is constructed as follows:
[0092]
[0093] In the above optimization problem, the HVAC energy cost consists of two parts, which are related to the fan and the cooling coil. The energy cost related to the fan can be expressed as:
[0094]
[0095] in: Approximately the energy consumption associated with the blower, μ is a constant coefficient, m i,t represents the air supply rate in time slot t in region i, p t represents the electricity price in time slot t, and τ represents the duration of a time slot;
[0096] The energy cost associated with the cooling coil can be expressed as:
[0097] Φ 2,t =P t pt τ (3)
[0098] Where: P t is the energy consumption of the cooling coil in time slot t, which can be described as:
[0099]
[0100] In formula (4): C a represents the specific heat capacity of air, η is the efficiency coefficient of the cooling coil, COP is the performance coefficient associated with the cooler, is the temperature of the mixed air at time slot t, T s is the supply air temperature of the blower.
[0101] The constraints in formula (1) include:
[0102] (11) To solve the energy cost minimization problem, we must first maintain the thermal comfort and air quality comfort conditions in each area. represents the indoor temperature of area i at time slot t, so:
[0103]
[0104] Where: K i,t represents the number of people indoors in area i at time slot t, T i min and T i max They represent the minimum and maximum values of the comfortable temperature range of area i respectively;
[0105] (12) Air quality comfort can be expressed by carbon dioxide concentration. The maximum indoor carbon dioxide concentration is controlled within the comfortable range, so:
[0106]
[0107] Among them: cb i,t represents the carbon dioxide concentration in region i at time slot t, Indicates the upper limit of the comfortable range of indoor carbon dioxide concentration in area i.
[0108] (13) Since the air supply rate of each zone and the total air supply rate of HVAC have upper limits, we have:
[0109] m i,t ∈{m 1 ,m 2 ,...,m M} (7)
[0110]
[0111] Where: mi,t represents the air supply rate in time slot t in area i, and its value can be selected from M discrete levels. The total air supply rate It should be less than the sum of the maximum air supply rates of all zones.
[0112] (14) Similarly, σ t represents the damper position of the air handling unit in HVAC at time slot t, and its value can be selected from Z discrete levels:
[0113] σ t ∈{σ 1 ,σ 2 ,...,σ Z} (9)
[0114]
[0115] When σ t = 0, it means that there is only outdoor fresh air in the mixed air, and when σ t When =1, it means that the mixed air comes entirely from the return air of each area.
[0116] In step 2 above, the Markov game model related to the damper agent and the regional air supply agent is designed according to the established HVAC control model, and the state, action and reward functions are defined as follows:
[0117] (1) Environmental state S t The expression is as follows:
[0118] S t =(o 1,t ,o 2,t ,...,o n,t ,o *,t ) (11)
[0119] Among them: 1,t ,o 2,t ,...,o n,t That is o i,t (1≤i≤n) represents the observed state of the agent related to the air supply rate decision in time slot t in region i, o *,t represents the observed state of the master control agent related to the air valve position decision at time slot t. In order to simplify the experimental environment, only one master control agent related to the air valve position decision is set. *,t =(cb 1,t ,...,cb i,t ,...,cb n,t ,K 1,t ,...,K i,t ,...,K n,t ). In the above formula, Tt out is the outdoor temperature in area i at time slot t, T t in is the indoor temperature in area i at time slot t, is the indoor temperature of the adjacent j area in time slot t of area i, N i represents the set of adjacent regions of region i, p t is the electricity price of time slot t, t' is the relative time of the current absolute time of time slot t in a day. For example, when the time slot τ in the program is 15 minutes, there are 1440 minutes in a day, and there are 1440÷15=96 time slots in total, t'=mod(t,96). i,t is the carbon dioxide concentration in region i at time slot t, K i,t is the number of occupants in time slot t in area i;
[0120] (2) Action A t The expression is as follows:
[0121] A t =(m 1,t ,...,m i,t ,...,m n,t ,β t ) (12)
[0122] Where: m i,t is the air supply rate of HVAC zone i at time slot t, β t The air valve position of HVAC;
[0123] (3) Since our goal is to minimize the HVAC energy cost while maintaining the thermal comfort and air quality comfort of all zones, the reward function of all agents is related to four parts, namely, the penalty for the fan energy cost, the penalty for the cooling coil energy cost, the indoor temperature out-of-bounds penalty, and the indoor carbon dioxide concentration out-of-bounds penalty.
[0124] Fan energy cost penalty and Related, this is m i,t In order to achieve coordination and cooperation among the N regional agents, the energy cost of the fan should be punished. Since the behavior of the damper agent has no effect on the energy consumption of the fan, the reward of the damper agent in the first part is designed to be r N+1,1,t = 0. The first part of the regional agent's reward r i,1,t (1≤i≤N) is:
[0125]
[0126] The energy consumption of the cooling coil and the air supply rate m of area i at time slot t i,t(1≤i≤N) and the position of the air valve at time slot t β t So the energy cost penalty for the cooling coil should be equally shared among all N+1 agents. Therefore, the second part of the reward for the damper agent is N+1,2,t Can be designed as The reward for the second part of the regional agent is designed as:
[0127]
[0128] Since the indoor temperature of area i at time slot t The air supply rate m at time slot t-1 i,t-1 The indoor temperature out-of-bounds penalty should be considered in the reward of the regional agent, so the reward of the third part of the damper agent is r N+1,3,t =0, and the reward design of the third part of the regional agent is:
[0129] r i,3,t (s t )=-I i,t ([T i,t -T i max ] + +[T i min -T i,t ] + ) (15)
[0130] Among them: When T i min ≤T i,t ≤T i max And K i,t > 0, the third part of the regional agent r i,3,t (s t )=0; when T i,t ≥T i max And K i,t > 0, the third part of the regional agent r i,3,t (s t )=-(T i,t -T i max ); when T i,t <T i min And K i,t > 0, the third part of the regional agent r i,3,t (s t )=-(T i min -T i,t ). When Ki,t =0, I i,t =0, when K i,t >0, I i,t =1, that is, if no one occupies area i in time slot t, then r i,3,t (s t )=0.
[0131] The indoor carbon dioxide concentration depends on the air supply rate m of area i at time slot t i,t (1≤i≤N) and the position of the air valve β t , so the penalty for the indoor carbon dioxide concentration crossing the boundary should be equally distributed among all agents. The reward r of the fourth part of the damper agent N+1,4,t Can be designed as: The reward for the fourth part of the regional agent can be designed as:
[0132]
[0133] Taking into account the four parts of rewards for all agents in the design, the reward for the agent in region i is: α(r i,1,t (s t-1 ,a t-1 )+r i,2,t (s t-1 ,a t-1 ))+βr i,4,t (s t )+r i,3,t (s t ), α and β are positive weight coefficients in units of ℃ / $ and ℃ / ppm, and the reward of the damper agent is: α(r N+1,2,t (s t-1 ,a t-1 ))+βr N+1,4,t (s t ).
[0134] Because α and β represent the energy cost and the average indoor carbon dioxide concentration deviation ACD respectively (the average value of the cumulative value of the carbon dioxide concentration deviation comfort range in all areas in all time slots, that is, the average indoor carbon dioxide concentration deviation L represents the total number of time slots, The importance parameter of ATD relative to the average indoor temperature deviation, so the energy cost will decrease with the increase of α and the decrease of β. Similarly, when α is larger and β is smaller, the indoor carbon dioxide concentration deviation will increase. Similarly, when a larger α or a larger β is given, the average indoor temperature deviation increases. Therefore, this method can flexibly control between energy cost, indoor temperature comfort level and indoor carbon dioxide concentration comfort level. When actually controlling the HVAC system of a multi-zone commercial building, appropriate α and β can be selected according to different comfort and energy cost requirements.
[0135] In addition, the number of occupants in each area can be obtained through the electronic counting sensor at the door, the indoor and outdoor temperatures of each area can be obtained through the temperature sensor, the indoor carbon dioxide concentration of each area can be obtained through the carbon dioxide sensor, and the electricity price information can be obtained from the local power company's website.
[0136] In the above step 3, the agent attention deep neural network includes 5 actor networks and 5 critic networks when the number of regions is 4 and the number of dampers is 1, and each actor network and critic network has an attention embedding mechanism.
[0137] Attention Embedding Module v i Through each agent data collection module f i (o j ,a j ), f i (o j ,a j )(i≠j) represents the observed state o of the receiving environment j and action a of agent j j The output value, v i Represents the global attention embedding of all agents:
[0138]
[0139] Coefficient α i,j Calculated by the following formula:
[0140]
[0141] in and W Φ is the parameter to be learned, β i,j The correlation between the embeddings of agent i and all other agents j is calculated by inner product, and then β is quantized by softmax function. i,j Normalize to get α i,j By using the weighted average v in equation (15) iRepresenting the observed states and actions of other agents, it is possible to model the interactions between agent i and any number of other agents, which makes it easy to increase the number of agents in the parallel training population.
[0142] In the above step 4, the three groups of intelligent agent populations with the highest fitness are screened from the six parallel groups of multi-agent populations. The screening must first meet the conditions that the average temperature deviation is ≤1°C and the average carbon dioxide concentration deviation is ≤10ppm. Secondly, the fitness is sorted and selected according to the energy cost.
[0143] In step 5 above, the weight parameters of the deep neural network of the three groups of multi-agent populations with high fitness (such as {C1, C2, C3}) are copied and mixed to form the new six groups {C 1x1 ,C 1x2 ,C 1x3 ,C 2x2 ,C 2x3 ,C 3x3 To simplify the commercial building environment, there is only one damper agent, and the deep neural network weight parameters of the redundant damper agents generated by replication are discarded. At this time, the number of agents in each group is expanded from 5 to 9. After the above steps, the optimal control strategy of the multi-agent population with the required number of areas and the number of dampers is finally trained, and the optimal control strategy is deployed in a multi-area commercial building environment.
[0144] Figure 2 The difference between the total air supply rate regulation of the method of the present invention and other methods is demonstrated. Scheme 1 is a rule-based method (Baseline), Scheme 2 is a control method using the multi-agent deep deterministic policy gradient algorithm (MADDPG), and Scheme 3 is the proposed method of the present invention (Proposed). The test phase lasted for 31 days with a time interval of 15 minutes. This figure captures the test results from the 14th day to the 18th day. It can be seen that the total air supply rate regulation method of the method of the present invention is far superior to the rule-based method, and the air supply rate is lower than the MADDPG method in most time slots, thereby effectively reducing energy costs.
[0145] like Figure 3 As shown, it is a comparison chart of the adaptability of the method of the present invention and other methods in various aspects. Compared with the rule-based method, the method of the present invention is better in three performance indicators, among which: the energy cost can be reduced by 29.9%. Compared with the MADDPG method, the method of the present invention can reduce the energy cost by 13.6% under the premise of sacrificing a certain comfort. In fact, the comfort reduction is negligible (the average temperature deviation difference is less than 0.2 degrees, and the average carbon dioxide concentration deviation is less than 0.5ppm). It can be seen that the method of the present invention can significantly reduce the energy cost while maintaining high thermal comfort and high air quality comfort.
[0146] The present invention provides a commercial building HVAC control system based on multi-agent attention evolutionary deep reinforcement learning, comprising:
[0147] Acquisition module: used to acquire pre-established multi-zone commercial building HVAC control models;
[0148] Design module: used to transform the HVAC control model into a Markov game problem involving damper agents and zone air supply agents;
[0149] Solution training module: used to solve Markov game problems in parallel using a multi-agent attention evolutionary deep reinforcement learning algorithm, and ultimately train the optimal control strategy for multiple groups of intelligent agent populations;
[0150] Screening module: used to perform performance testing using the optimal control strategies of multiple groups of intelligent agent populations obtained through training, and screen out several groups of intelligent agent populations for the test phase;
[0151] Reorganization module: used to reorganize the deep neural network weights of several selected groups of intelligent agent populations;
[0152] Solution screening module: The reorganized deep neural network weights are used as the initial weights of a larger multi-agent population model, and the larger Markov game problem is repeatedly solved until the size of the agent population is equal to the sum of the number of regions and the number of air valves. A group of multi-agent populations with the highest fitness is screened out, and then the optimal control strategy for the air valve agent and the regional wind supply agent is obtained;
[0153] Deployment module: used to deploy the HVAC optimal control strategy of the air valve intelligent body and the regional air supply intelligent body online.
[0154] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A commercial building HVAC control method based on evolutionary deep reinforcement learning, characterized in that: The method comprises: Access pre-built multi-zone commercial building HVAC control models; The HVAC control model is transformed into a Markov game problem involving the damper agent and the district air supply agent. The multi-agent attention evolution deep reinforcement learning algorithm is used to solve the Markov game problem in parallel, and finally the optimal control strategy of multiple groups of intelligent agents is trained; Use the optimal control strategies of multiple groups of intelligent agent populations obtained through training to perform performance tests and select several groups of intelligent agent populations for the test phase; Reorganize the weights of the deep neural networks of several selected groups of intelligent agent populations; The reorganized deep neural network weights are used as the initial weights of a larger multi-agent population model, and the larger Markov game problem is repeatedly solved until the size of the agent population is equal to the sum of the number of regions and the number of wind valves. A group of multi-agent populations with the highest fitness is screened out, and then the optimal control strategy for the wind valve agent and the regional wind supply agent is obtained. The HVAC optimal control strategy of the air valve intelligent agent and the regional air supply intelligent agent is deployed to the actual HVAC system for online control; The HVAC control model includes an objective function, decision variables and constraints; The objective function expression is as follows: Where: Φ 1,t represents the energy cost associated with the blower, Φ 2,t represents the energy cost associated with the cooling coil; Where: The energy cost associated with the blower is expressed as: Where: Approximately the energy consumption associated with the blower, μ is a constant coefficient, m i,t represents the air supply rate in time slot t in region i, p t represents the electricity price in time slot t, τ represents the duration of a time slot; the energy cost associated with the cooling coil is expressed as: F 2,t =P t p t t (3) Where: P t is the energy consumption of the cooling coil in time slot t, expressed as: Where: C a represents the specific heat capacity of air, η is the efficiency coefficient of the cooling coil, COP is the performance coefficient associated with the cooler, is the temperature of the mixed air at time slot t, T s is the air supply temperature of the blower; The decision variables include the air supply rate m for each zone i,t and the damper position of the air handling unit in HVAC t , the expression is as follows: m i,t ∈{m 1 ,m 2 ,...,m M } (5) Where: m i,t Its value can be selected from M discrete levels, the total air supply rate To be less than the sum of the maximum air supply rates of all zones, σ t The value of is chosen from Z discrete levels: s t ∈{σ 1 ,s 2 ,...,s Z } (7) When σ t = 0, it means that there is only outdoor fresh air in the mixed air, and when σ t =1, it means that the mixed air comes entirely from the return air of each area; The constraints include those related to indoor temperature and carbon dioxide concentration, and the expressions are as follows: like represents the indoor temperature of area i at time slot t, then: Where: K i,t represents the number of people indoors in area i at time slot t, T i min and T i max They represent the minimum and maximum values of the comfortable temperature range of area i respectively; cb i,t represents the carbon dioxide concentration in region i at time slot t, then: Where: It represents the upper limit of the comfortable range of indoor carbon dioxide concentration in area i; The Markov game problem includes state, action and reward function: Environmental status S t The expression is as follows: S t =(the 1,t ,the 2,t ,...,the n,t ,the *,t ) (11) Where: 1,t ,o 2,t ,...,o n,t Middle i,t represents the observed state of the agent related to the air supply rate decision in time slot t in area i, 1≤i≤n, o *,t represents the observed state of the master control agent related to the air valve position decision at time slot t, o *,t =(cb 1,t ,...,cb i,t ,...,cb n,t ,K 1,t ,...,K i,t ,...,K n,t ), T t out is the outdoor temperature in area i at time slot t, T t in is the indoor temperature in area i at time slot t, is the indoor temperature of the adjacent j area in time slot t of area i, N i represents the set of adjacent regions of region i, p t is the electricity price of time slot t, t' is the relative time of the current absolute time of time slot t within a day, cb i,t is the carbon dioxide concentration in region i at time slot t, K i,t is the number of occupants in time slot t in area i; The action A t The expression is as follows: A t =(m 1,t ,...,m i,t ,...,m n,t ,β t ) (12) Where: m i,t is the air supply rate of HVAC zone i at time slot t, β t The air valve position of HVAC; The reward function R t The expression is as follows: Where: R 1,i,t is the reward obtained by the agent related to the air supply rate decision in area i and time slot t, R 2,i,t is the reward obtained by the master control agent related to the damper position decision at time slot t, α is the importance coefficient of the penalty cost caused by the HVAC energy cost to the comfortable temperature range deviation, β is the importance coefficient of the penalty cost caused by the indoor carbon dioxide concentration exceeding the upper limit to the penalty cost caused by the comfortable temperature range deviation, C 1,i,t (o i,t ) is the penalty cost caused by violating the indoor comfortable temperature range in time slot t in area i, is the penalty related to the energy cost of the fan in area i and time slot t, is the penalty associated with the cooling coil energy cost for time slot t in zone i, C 4,i,t (o i,t ) is the penalty cost caused by violating the upper limit of indoor carbon dioxide concentration in area i during time slot t, is the penalty associated with the cooling coil energy cost for all zones t in time slot, C 6,t (o *,t ) is the penalty cost caused by violating the upper limit of indoor carbon dioxide concentration in all areas during time slot t.
2. A commercial building HVAC control method based on evolutionary deep reinforcement learning according to claim 1, characterized in that: The network architecture used in the multi-agent attention evolution deep reinforcement learning algorithm includes an actor network and a critic network. Both the actor network and the critic network adopt an attention embedding module. The output v of the attention embedding module i The expression is as follows: Where: f i (o j ,a j ) represents the observed state of the receiving environment o j and action a of agent j j The output value, i≠j, coefficient α i,j Calculated by the following formula: Where: and W Φ is the parameter to be learned, β i,j The correlation between the embeddings of agent i and all other agents j is calculated by inner product, and then β is quantized by softmax function. i,j Normalize to get α i,j .
3. A commercial building HVAC control method based on evolutionary deep reinforcement learning according to claim 1, characterized in that: Methods for performing performance tests using the optimal control strategies of multiple groups of intelligent agent populations obtained through training and selecting several groups of intelligent agent populations for the test phase include: A Gaussian random variable with a mean of 0 and a variance of the weight is superimposed on the weight parameters of the deep neural network and loaded into the deep neural network; the parallel multi-agent population after adding disturbance to the weight parameters is screened; and fitness is sorted and selected according to the energy cost while meeting the requirements of average temperature deviation and average carbon dioxide concentration deviation.
4. A commercial building HVAC control method based on evolutionary deep reinforcement learning according to claim 3, characterized in that: The method of reorganizing the deep neural network weights of the selected groups of agent populations includes: K groups {K1,...,K i ,...,K k }After the multi-agent population is screened; Group C with high fitness The weight parameters of the deep neural network of the multi-agent population are copied and mixed two by two to form several new groups {C 1x1 ,C 1x2 ,...,C 1xc ,...,C ixi-1 ,...,C ixc ,...,C cxc }, at this time the number of agents in each group is expanded from n to 2n; if the required number of agents is still not reached, K groups are randomly selected to continue parallel training.
5. A commercial building HVAC control method based on evolutionary deep reinforcement learning according to claim 1, characterized in that: The method for online deployment of the HVAC optimal control strategy of the air valve intelligent body and the regional air supply intelligent body includes: Collect the observation status of each current time slot; According to the observed state and HVAC optimal control strategy, the respective actions are selected, and then the decision actions of the air valve control position and the air supply rate of each area are obtained; The obtained decision actions are deployed to the actual HVAC system for online control.
6. A commercial building HVAC control system based on evolutionary deep reinforcement learning, characterized in that include: Acquisition module: used to acquire pre-established multi-zone commercial building HVAC control models; Design module: used to transform the HVAC control model into a Markov game problem involving damper agents and zone air supply agents; Solution training module: used to solve Markov game problems in parallel using a multi-agent attention evolutionary deep reinforcement learning algorithm, and ultimately train the optimal control strategy for multiple groups of intelligent agent populations; Screening module: used to perform performance testing using the optimal control strategies of multiple groups of intelligent agent populations obtained through training, and screen out several groups of intelligent agent populations for the test phase; Reorganization module: used to reorganize the deep neural network weights of several groups of selected agent populations; Solution screening module: uses the reorganized deep neural network weights as the initial weights of a larger multi-agent population model, repeatedly solves the larger Markov game problem until the size of the agent population is equal to the sum of the number of regions and the number of air valves, and selects a group of multi-agent populations with the highest fitness, thereby obtaining the optimal control strategy for the air valve agent and the regional wind supply agent; Deployment module: used to deploy the HVAC optimal control strategy of the air valve intelligent body and the regional air supply intelligent body to the actual HVAC system for online control; The HVAC control model includes an objective function, decision variables and constraints; The objective function expression is as follows: Where: Φ 1,t represents the energy cost associated with the blower, Φ 2,t represents the energy cost associated with the cooling coil; Where: The energy cost associated with the blower is expressed as: Where: Approximately the energy consumption associated with the blower, μ is a constant coefficient, m i,t represents the air supply rate in time slot t in region i, p t represents the electricity price in time slot t, τ represents the duration of a time slot; the energy cost associated with the cooling coil is expressed as: F 2,t =P t p t t (3) Where: P t is the energy consumption of the cooling coil in time slot t, expressed as: Where: C a represents the specific heat capacity of air, η is the efficiency coefficient of the cooling coil, COP is the performance coefficient associated with the cooler, is the temperature of the mixed air at time slot t, T s is the air supply temperature of the blower; The decision variables include the air supply rate m for each zone i,t and the damper position of the air handling unit in HVAC t , the expression is as follows: m i,t ∈{m 1 ,m 2 ,...,m M } (5) Where: m i,t Its value can be selected from M discrete levels, the total air supply rate To be less than the sum of the maximum air supply rates of all zones, σ t The value of is chosen from Z discrete levels: s t ∈{σ 1 ,s 2 ,...,s Z } (7) When σ t = 0, it means that there is only outdoor fresh air in the mixed air, and when σ t =1, it means that the mixed air comes entirely from the return air of each area; The constraints include those related to indoor temperature and carbon dioxide concentration, and the expressions are as follows: like represents the indoor temperature of area i at time slot t, then: Where: K i,t represents the number of people indoors in area i at time slot t, T i min and T i max They represent the minimum and maximum values of the comfortable temperature range of area i respectively; cb i,t represents the carbon dioxide concentration in region i at time slot t, then: Where: It represents the upper limit of the comfortable range of indoor carbon dioxide concentration in area i; The Markov game problem includes state, action and reward function: Environmental status S t The expression is as follows: S t =(the 1,t ,the 2,t ,...,the n,t ,the *,t ) (11) Where: 1,t ,o 2,t ,...,o n,t Middle i,t represents the observed state of the agent related to the air supply rate decision in time slot t in area i, 1≤i≤n, o *,t represents the observed state of the master control agent related to the air valve position decision at time slot t, o *,t =(cb 1,t ,...,cb i,t ,...,cb n,t ,K 1,t ,...,K i,t ,...,K n,t ), T t out is the outdoor temperature in area i at time slot t, T t in is the indoor temperature in area i at time slot t, is the indoor temperature of the adjacent j area in time slot t of area i, N i represents the set of adjacent regions of region i, p t is the electricity price of time slot t, t' is the relative time of the current absolute time of time slot t within a day, cb i,t is the carbon dioxide concentration in region i at time slot t, K i,t is the number of occupants in time slot t in area i; The action A t The expression is as follows: A t =(m 1,t ,...,m i,t ,...,m n,t ,β t ) (12) Where: m i,t is the air supply rate of HVAC zone i at time slot t, β t is the air valve position of HVAC; the reward function R t The expression is as follows: Where: R 1,i,t is the reward obtained by the agent related to the air supply rate decision in area i and time slot t, R 2,i,t is the reward obtained by the master control agent related to the damper position decision at time slot t, α is the importance coefficient of the penalty cost caused by the HVAC energy cost to the comfortable temperature range deviation, β is the importance coefficient of the penalty cost caused by the indoor carbon dioxide concentration exceeding the upper limit to the penalty cost caused by the comfortable temperature range deviation, C 1,i,t (o i,t ) is the penalty cost caused by violating the indoor comfortable temperature range in time slot t in area i, is the penalty related to the energy cost of the fan in area i and time slot t, is the penalty associated with the cooling coil energy cost for time slot t in zone i, C 4,i,t (o i,t ) is the penalty cost caused by violating the upper limit of indoor carbon dioxide concentration in area i during time slot t, is the penalty associated with the cooling coil energy cost for all zones t in time slot, C 6,t (o *,t ) is the penalty cost caused by violating the upper limit of indoor carbon dioxide concentration in all areas during time slot t.
