HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning

Abstract

The invention provides an HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning. Through an entropy-based data screening method, a strategy evaluation module and a design reward function, HVAC system temperature control and energy consumption optimization are realized under external condition interference. According to the entropy-based data screening method, the value of data for agent training is calculated through an error formula, high-value data is repeatedly used for agent training, and the training effect is enhanced; the strategy evaluation module is used for further solving the problem of overestimation of a Q value on a double-Q structure, correcting the output of the Q value, reducing the influence of a random Q value, and adding regularization of a strategy and an actual action into a loss function updating process, so that the influence of random noise on the Q value is eliminated, and the accuracy of strategy output is ensured; a reward function for controlling the temperature and optimizing the energy consumption is designed, so that the intelligent agent can balance the contradiction among multiple tasks, and the optimal solution of multi-target task intersection is searched, so that the HVAC system can control the temperature to be always kept in the temperature range, and meanwhile, unnecessary energy consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a temperature control and energy consumption optimization method for an HVAC system based on deep reinforcement learning. Background Art

[0002] Heating, Ventilation and Air Conditioning (HVAC) is an integral part of modern buildings, and is designed to provide a comfortable indoor environment. HVAC systems mainly refer to heating, ventilation and air conditioning systems in buildings, which can control indoor temperature and humidity to improve the comfort of indoor users. Usually, the control system only controls the temperature of a single control variable, but with the implementation of the "double carbon" plan, it is necessary to consider reducing the energy consumed by the operation of the HVAC system. Since the operation of the HVAC system accounts for more than one-third of the energy consumption of the building, there is huge potential for reducing the energy consumption of the system.

[0003] With the development of intelligent technology, HVAC systems are equipped with a variety of sensors to detect the real-time status of equipment in the controlled environment, allowing temperature control and energy management. Many advanced intelligent algorithms can be used to control the real-time temperature of system equipment, but the control performance is insufficient. At the same time, there is a certain contradiction between temperature control and energy consumption optimization, and it is difficult for existing algorithms to balance different control purposes. In current applications, predictive models are usually used to predict the state of a period of time in the future, and control is performed in advance based on the predicted state. However, there are many external interferences that will seriously affect the prediction results, resulting in the subsequent control performance not meeting the requirements, and in order to achieve multiple required goals at the same time, simple predictive control cannot be achieved. Therefore, in order to reduce the impact of external interference and improve control performance, training an algorithm with stability and high performance is the most important task for temperature control and energy consumption optimization. Summary of the invention

[0004] An embodiment of the present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning, which is used to solve the technical problems existing in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme.

[0006] HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning, including:

[0007] S1 calculates the reward function of the control model based on the acquired environmental parameters and HVAC control status data according to the requirements of temperature control and energy consumption optimization;

[0008] The control model includes a first branch, a second branch and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer sequentially arranged along the data flow direction, and the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer sequentially arranged along the data flow direction, the actor-critic network includes an actor network and a critic network sequentially arranged in the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks;

[0009] The critic network includes a critic first Q network and a critic second Q network which are arranged in parallel with each other; the critic first Q network and the critic second Q network respectively include two sub-networks, and the two sub-networks of the critic first Q network are used to calculate and obtain Q1 value and Q2 value respectively; the sub-networks of the critic second Q network are used to calculate and obtain minQ' value respectively, so that the actor critic network can calculate and obtain the value for reverse updating the critic first Q network and the critic second Q network through Q1 value, Q2 value and minQ' value;

[0010] The reward function includes:

[0011] Pass-through

[0012] R=R 能耗 +βR 温度 (1) Construct the overall reward function; where the energy consumption reward value R 能耗 and temperature reward value R 温度 Separately through

[0013] R 能耗 =W 平均 / W 当前平均 (2)

[0014] R 温度 =-(T 当前 -T 设定均值 ) 2 / (T 设定上限 -T 设定均值 ) 2 (3) is calculated; β is the ratio of the adjustment temperature reward to the total reward; in formula (2), W 平均 It refers to the average energy consumption of the system under the control of the control algorithm, W 当前平均 It refers to the average energy consumption from the start of the algorithm to the current state; in formula (3), T 当前 Refers to the current temperature, T 设定均值 Refers to the average temperature of the temperature range to be maintained, T 设定上限 It refers to the highest temperature in the temperature range;

[0015] S2 inputs the actual environment data in the building to which the HVAC belongs and the HVAC control status data into the reward function, calculates the reward value r of the control system, and screens the actual environment data corresponding to the calculated reward value r through the entropy-based data screening layer to obtain high-value data;

[0016] S3 initializes the four Q networks of the critic network, initializes the actor network, and returns the initialized data to the experience pool;

[0017] S4 obtains the state s of the agent in the current environment at each time step t , in the current state s t Select and execute action a, and change the current state s t And action a is input into the reward function to obtain the reward value r, so that the agent is transferred to the new state s after executing action a t+1 ;

[0018] S5 evaluates the execution effect of step S4, and if the execution effect is lower than the first preset threshold, re-executes step S4, otherwise, stores the execution result in the experience pool;

[0019] S6 repeatedly executes steps S4 and S5 multiple times, and stores the execution results in the experience pool s i ,a,r,s i+1 t∈i; determine whether the action strategy in the experience pool is greater than the second preset threshold, if so, execute step S6, otherwise, return to execute step S3;

[0020] S7 passes through the policy evaluation layer and uses

[0021] Q(s,a)←Q(s,a)+a[r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )](4) Calculate the experience pool s i ,a,r,s i+1 The target Q value of the action strategy in t∈i; calculated by the gradient descent method

[0022] δ=|r+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (5)

[0023] δ=|t+αH(s,a)+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (6)

[0024] H(s,a)=- logπ(s,a) (7) The difference between the target Q value obtained by calculation and the target Q value minimized by the gradient descent method; In formula (4), s is the state representing the dimensional data that affects the control target; a is the control action calculated by the agent through the strategy in the current state; r is the reward obtained by taking action in the current state; Q(s t ,a t ) is the cumulative target Q value that the agent is expected to obtain in the future under a certain state and action; γ is a discount factor used to measure the importance of future rewards; in formula (7), π(s,a) represents the probability of taking action a in state s;

[0025] S8 Action a of the critic Q network based on updating weights t+1 , combined with the strategy evaluation of the action π(s) selected by the actor network, through the loss function

[0026]

[0027] Calculate the error of the execution results of steps S4 to S8, and then use the error to reversely update the critic's second Q network; in formula (8), π(s) represents the action selected by the strategy under the agent state s; y is calculated by the formula y=r+γmax Q(s t+1 ,a t+1 ; θ′) is calculated, where γ is the discount factor;

[0028] S9 dynamically adjusts the β value of the reward function according to the control results to improve the control accuracy of the agent in achieving specific goals;

[0029] S10 returns to execute steps S4 to S9, trains the agent multiple times, calculates the temperature offset and average energy consumption after the training is completed, and saves the network parameters and the optimal control model weights;

[0030] S11 controls the HVAC using the optimal control model obtained after executing step S11.

[0031] Preferably, step S2 comprises:

[0032] S21 inputs each piece of actual environment data and HVAC control state data in the building to which the HVAC belongs into the reward function, and calculates and obtains a reward value r corresponding to each piece of actual environment data and HVAC control state data;

[0033] S22 sorts all reward values ​​r, and obtains high-value data by comparing all reward values ​​r with a third preset threshold.

[0034] Preferably, step S1 also includes a process of cleaning and normalizing the acquired environmental parameters and HVAC control status data.

[0035] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning. The entropy-based data screening method, the strategy evaluation module and the design reward function realize the temperature control and energy consumption optimization of the HVAC system under the interference of external conditions. The entropy-based data screening method calculates the value of data for the training of the intelligent agent through the error formula, and repeatedly uses high-value data for the training of the intelligent agent to enhance the training effect. The strategy evaluation module further solves the problem of overestimation of the Q value on the dual Q structure and corrects the output of the Q value. Not only does the dual Q network reduce the influence of the random Q value, but the strategy evaluation also adds the regularization of the strategy and the actual action to the loss function update process, eliminating the influence of random noise on the Q value and ensuring the accuracy of the strategy output. Designing a reward function that can control temperature and optimize energy consumption allows the intelligent agent to balance the contradictions between multiple tasks and find the optimal solution for the intersection of multiple objective tasks. The new reward function enables the HVAC system to control the temperature within the temperature range while reducing unnecessary energy consumption.

[0036] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0038] Figure 1 A processing flow chart of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0039] Figure 2 A structural diagram of an energy consumption optimization model for the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0040] Figure 3 A schematic diagram of a critic-actor network for a HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0041] Figure 4 A schematic diagram of the operation process of the entropy-based data screening method of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0042] Figure 5A schematic diagram of the strategy evaluation and updating process of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0043] Figure 6 A schematic diagram of state and action modeling of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0044] Figure 7 A schematic diagram of a preferred embodiment of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0045] Figure 8 A schematic diagram of the control effect of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0046] Fig. 9 This is a comparison chart of energy consumption between the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention and other methods in the prior art. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0048] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0049] The present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning, which is used to solve the following technical problems existing in the prior art:

[0050] In a building's HVAC system, the main goal is to ensure indoor temperature while minimizing energy consumption. Existing HVAC system control methods mainly include traditional proportional-integral-differential (PID) control and model predictive control (MPC).

[0051] PID control is a simple and widely used control method. The PID controller reduces the control error, that is, the difference between the set value and the actual value, by controlling the output of the device. The advantages of the PID controller are simple structure, easy design and implementation. However, the PID controller has the following limitations: Long adjustment time: PID control requires a lot of debugging time to obtain the best control parameters. Poor adaptability: PID control has poor adaptability to changes in system parameters and external interference, which can easily lead to a decrease in control performance. Difficulty in controlling nonlinear systems: For complex nonlinear systems, PID control is not effective and it is difficult to meet the high performance requirements of the system.

[0052] Model predictive control is an advanced model-based control method that optimizes control inputs by predicting system behavior over a period of time in the future. The advantages of MPC include:

[0053] a. Optimize control performance. MPC can achieve better control performance by optimizing the objective function.

[0054] b. Multivariable control. MPC can handle multiple-input multiple-output (MIMO) systems and is suitable for complex control tasks.

[0055] c. Consider constraints. MPC can consider various constraints of the system during the optimization process, such as physical limitations of input and output.

[0056] Although MPC has the above advantages, there are also some problems in its application:

[0057] a. High computational complexity. MPC needs to perform optimization calculations in each control cycle, which requires a large amount of calculations and high real-time requirements.

[0058] b. Strong model dependence. The performance of MPC depends on the accuracy of the system model. For systems with inaccurate models or large environmental changes, the control effect will be affected.

[0059] c. Complex implementation. The implementation of MPC requires high computing power and algorithm complexity, which increases the development and maintenance costs of the system.

[0060] In order to overcome the limitations of traditional control methods, deep reinforcement learning (DRL) has been widely used in HVAC systems as an emerging control strategy in recent years. DRL can achieve efficient control in complex environments by learning the state-action relationship of the environment. The advantages of the DRL method include:

[0061] a. Strong adaptability. DRL can adapt to changes in the environment through continuous learning and improve control performance.

[0062] b. Processing high-dimensional data. DRL can handle high-dimensional state and action spaces and is suitable for complex control tasks.

[0063] c. Learning nonlinear control strategies. DRL can learn complex nonlinear control strategies and improve the control accuracy of the system.

[0064] However, DRL also faces some challenges in practical applications: Long training time: The training process of DRL requires a lot of computing resources and time, resulting in a long system development cycle. Poor stability: DRL is prone to instability and convergence difficulties during training, which affects control performance. Large data requirements: DRL requires a large amount of training data and has a high dependence on data, which increases the development cost of the system.

[0065] The above prior art has the following disadvantages:

[0066] 1. Because there are too many factors to consider in accurately controlling the HVAC system, a large amount of interactive training data is needed to learn effective strategies. It is necessary to reduce the model's dependence on the amount of data and improve the utilization of existing training data.

[0067] 2. HVAC systems usually have complex nonlinear characteristics. Existing methods find it very difficult to establish accurate mathematical models in the face of these nonlinear characteristics and cannot fully adapt to the dynamic changes of the system.

[0068] 3. Existing control methods are all single-objective control methods. When faced with multi-task objectives, the model needs to be adjusted, and it is difficult to meet multi-task objectives, which affects the control effect.

[0069] In view of this, the present invention is to solve the following problems:

[0070] 1. In order to reduce the model's dependence on the amount of data, the model's utilization of the explored training data is improved. The existing data is analyzed for value, and data with greater training value is screened out. At the same time, data with higher training value is actively extracted and learned.

[0071] 2. Achieve precise control of complex systems to adapt to dynamic changes, and use deep reinforcement learning models to establish dynamic relationships between inputs and outputs. The control strategy adopted by the model needs to be more robust, and a strategy evaluation module needs to be designed to evaluate the effectiveness of the strategy, while further optimizing the effective strategy.

[0072] 3. Conventional reward functions lack the design of multi-objective tasks, making it difficult for the control system to meet multiple objective tasks at the same time. The reward function designed by this method considers the optimal solution between the balance of various objectives, so that the intelligent agent can complete the requirements of different task objectives.

[0073] See also Figure 1 The present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning, comprising the following steps:

[0074] S1 calculates the reward function based on the acquired environmental parameters and HVAC control status data according to the requirements of temperature control and energy consumption optimization;

[0075] like Figure 2 As shown, the method provided by the present invention designs an improved control model, including a first branch, a second branch and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer sequentially arranged along the data flow direction, and the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer sequentially arranged along the data flow direction, the actor-critic network includes an actor network and a critic network sequentially arranged in the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks;

[0076] S2 inputs the actual environmental data in the building to which the HVAC belongs into the reward function, calculates the reward value r of the control system, and screens the actual environmental data corresponding to the calculated reward value r through the entropy-based data screening layer to obtain high-value data;

[0077] S3 initializes the four Q networks of the critic network, initializes the actor network, and returns the initialized data to the experience pool;

[0078] S4 obtains the state s of the agent in the current environment at each time step t , in the current state s t Select and execute action a, and change the current state s t And action a is input into the reward function to obtain the reward value r, so that the agent is transferred to the new state s after executing action a t+1 ;

[0079] S5 evaluates the execution effect of step S4, and if the execution effect is lower than the first preset threshold, re-executes step S4, otherwise, stores the execution result in the experience pool;

[0080] S6 repeatedly executes steps S4 and S5 and stores the execution results in the experience pool (s i ,a,r,s i+1t∈i); determine whether the action strategy in the experience pool is greater than the second preset threshold, if so, execute step S6, otherwise, return to execute step S3;

[0081] S7 calculates the experience pool (s i ,a,r,s i+1 The target Q value of the action strategy in t∈i) is minimized by the gradient descent method with the target Q value and the Q value of the critic network, and then the weight of the critic network is calculated and updated;

[0082] S8 Action a of the critic Q network based on updating weights t+1 , combined with the policy evaluation of the action π(s) selected by the actor network, the actor network is updated by calculation;

[0083] S9 dynamically adjusts the parameter value of the reward function according to the control results to improve the control accuracy of the intelligent agent in achieving specific goals;

[0084] S10 returns to execute steps S4 to S9, trains the agent multiple times, calculates the temperature offset and average energy consumption after the training is completed, and saves the network parameters and the optimal control model weights;

[0085] S11 uses the optimal control model obtained after executing step S11 to control the HVAC, so that the energy consumption of the HVAC reaches the optimal state under the premise that the indoor temperature meets the requirements.

[0086] In the preferred embodiment provided by the present invention, the specific implementation process of each key step is as follows:

[0087] 1. Design reward function

[0088] The task goal is to comprehensively consider the energy consumption and temperature requirements of the system. By optimizing the objective function and adjusting the control strategy, the comprehensive optimization of energy consumption and temperature is achieved to ensure that the system energy consumption is minimized while meeting the user's temperature requirements [21,24]. To realize the design of the reward function, the state and action must be modeled first.

[0089] (1) State and action modeling

[0090] The status records the equipment parameters of the building HVAC system to achieve real-time monitoring of the system. The present invention mainly solves the tasks of temperature control and energy consumption optimization of the HVAC system, and needs to record the status feature points related to the task. The status records the HVAC power consumption, fan coil power consumption, gas boiler power consumption, indoor area temperature, outdoor temperature, outdoor air pressure, and outdoor humidity. The action is the control measure executed according to the status, including the switch and set value of the heat pump, the switch and set value of the fan coil, the set temperature of the mixer, and the set temperature value of the area.

[0091] The premise for achieving the energy consumption optimization task is to control the indoor area temperature within the set range [21,24]. The external variables that affect temperature control are outdoor temperature, outdoor air pressure and outdoor humidity. Under the influence of external variables, the system switches the heat pump and temperature setting and the fan coil switch and temperature setting, and inputs the two into the mixer to achieve the set temperature of the mixer and input it into the indoor area to control the temperature. If the indoor area temperature is lower than the set range [21,24], according to the external influences related to outdoor temperature, air pressure and humidity, these states are input into the intelligent agent, and its strategy network outputs actions such as heat pump, fan coil and mixer set values ​​to control the temperature so that the indoor temperature returns to the set range. The specific process is as follows: Figure 4 shown.

[0092] HVAC power consumption mainly comes from fan coil unit power consumption and gas boiler power consumption. In order to save energy, the switch and set value of the heat pump and the switch and set value of the fan coil unit should be reasonably controlled to accurately achieve the mixer set temperature value. When the equipment control temperature is within the temperature setting range, it is necessary to optimize the equipment switching frequency and reasonable equipment control value. By controlling each device in the system, the temperature control setting range close to the indoor temperature (the control system is close to 24 degrees in summer and close to 21 degrees in winter to reduce energy consumption), reduce unnecessary heating or cooling capacity, and reduce the total energy consumption of the system.

[0093] (2) Reward Function Modeling

[0094] In deep reinforcement learning, the design of the reward function is usually adjusted to enable the agent to complete different task goals. A positive reward value encourages the agent's current behavior; a negative reward value punishes the agent's actions. The overall reward function design of the present invention consists of energy consumption optimization and temperature control, as shown in formula (1). The energy consumption reward value and the temperature reward value are shown in formula (2) and formula (3).

[0095] R=R 能耗 +βR 温度 (1) By adjusting the proportion of temperature reward to total reward by β, the target reward function can be dynamically adjusted according to the training effect;

[0096] R 能耗 =W 平均 / W 当前平均 (2)

[0097] Where R 能耗 The value of is usually greater than 1; W 平均 It refers to the average energy consumption of the system under the control of the control algorithm compared; W 当前平均 It refers to the average energy consumption from the start of the algorithm to the present;

[0098] R 温度 =-(T当前 -T 设定均值 ) 2 / (T 设定上限 -T 设定均值 ) 2 (3)

[0099] Where R 温度 The value of is less than 0; T 当前 Refers to the current temperature; T 设定均值 Refers to the average temperature of the temperature range to be maintained; T 设定上限 It refers to the highest temperature in the temperature range.

[0100] The temperature function can only be kept within the temperature range, and its value range is between [-1,0]. If the temperature exceeds the set range, the temperature reward value will be a large negative number. Combining the energy consumption and temperature functions, the agent can only get a positive reward if it keeps the system within the temperature range. The lower the energy consumption is, the greater the reward value will be.

[0101] The model structure of the present invention obtains reward values ​​according to the state and action of the building environment operation data by designing a new reward function to achieve multiple task goals, and then inputs the calculated data into the entropy-based data screening method to screen out high-value data (environmental data). The high-value training data is evaluated through two Q networks to optimize the Q value, and then the minimum value is taken to reduce the error generated by the prediction, and the optimal Q value is used to update the intelligent agent. The structure of the entire model is as follows Figure 2 As shown, it includes a first branch, a second branch and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer arranged in sequence along the data flow direction, the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer arranged in sequence along the data flow direction, the actor-critic network includes an actor network and a critic network arranged in sequence in the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks.

[0102] Among them, the critic network of the actor-critic network includes a critic first Q network and a critic second Q network set in parallel with each other, and the critic second Q network is the target Q network. The critic first Q network and the critic second Q network each include two sub-networks, and the two sub-networks of the critic first Q network are used to calculate and obtain the Q1 value and the Q2 value respectively. The sub-networks of the critic second Q network are used to calculate and obtain the minQ' value respectively, so that the actor-critic network can calculate the error for reverse updating the first Q network and the critic second Q network through the Q1 value, the Q2 value and the minQ' value. Through the processing of the above-mentioned actor-critic network, the random relationship between the state and the action is maintained as much as possible, and the error caused by the strong connection between the state and the action is reduced.

[0103] Data screening method based on entropy

[0104] The data screening method based on entropy is used to screen high-value data and give priority to the input of intelligent training. The data is the actual data inside the building, which refers to the four-tuple data (s) of the current state, action, reward value and next state. i ,a,r,s i+1 t∈i). The state refers to the power consumption of the HVAC system, the power consumption of the fan coil, the power consumption of the gas boiler, the regional temperature, the outdoor temperature, the outdoor air pressure, and the outdoor humidity in the system. The action is the control quantity selected by the agent through the state and control strategy, including the switch and set value of the heat pump, the switch and set value of the fan coil, the set temperature of the mixer, and the set temperature value of the region. The building will automatically generate data for the next state when it has a control action. The reward is calculated by the reward function through the state and action. The value function is calculated through the above data to evaluate the potential contribution of each data to the strategy optimization.

[0105] Entropy is an important factor that affects the judgment of high-value data. The higher the entropy value, the higher the value, which means that the action uncertainty of the strategy prediction is higher. High-value data with high uncertainty enables the model to actively explore unknown environments and explore the global optimal solution. High value means that the existing state or action taken is data that has not been explored by the current control strategy, has a large deviation from the current strategy, is more random, and has more learning value. High-value data points are more likely to optimize the strategy.

[0106] Entropy is used to measure the uncertainty of the model's prediction of samples, so that the strategy learned by the agent is the optimal solution after exploring the global situation, and prevents the agent from converging to a suboptimal strategy prematurely. By giving priority to these high-entropy samples for labeling, the model can actively learn and improve. Adding the uncertainty of entropy to the error formula allows the model to actively identify and filter data with higher value, accelerate the priority processing of high-value data, reduce the number of low-value data training times, and effectively improve the learning efficiency and performance of the control algorithm. The high value judgment is determined by the size of the Q value error δ and the entropy value. The Q value calculation formula is shown in formula (4).

[0107] Q(s,a)←Q(s,a)+α[r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )] (4)

[0108] s is the state representing the various dimensions of data that affect the control target; a is the control action calculated by the agent through the strategy in the current state; r is the reward obtained by taking action in the current state; Q(s t ,a t ) is the cumulative reward that the agent is expected to obtain in the future under a certain state and action; γ is a discount factor that measures the importance of future rewards;

[0109] The original error formula is shown in the following formula (5), which calculates the difference between the Q value generated by the actual action and the Q value generated by the predicted strategy. In this module, the difference δ is used to mark the value of the data. The value data generated by formula (5) will cause the model to fall into a suboptimal solution. The main reason is that the agent using the original error function does not fully explore the environment. Therefore, the entropy of uncertainty is added to the original error formula to give a higher value to the state data that the agent has not explored. Encourage the agent to explore the environment comprehensively until it finds the global optimum. The original error formula is shown in formula (5), the error formula with entropy added is shown in formula (6), and the entropy formula is shown in formula (7):

[0110] δ=|r+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (5)

[0111] δ=|r+αH(s,a)+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (6)

[0112] Among them, δ is the difference in entropy including uncertainty between the existing strategy and the explored strategy, and its size represents the value of the data; H(s,a) is the entropy value in the current state; α ranges from [0,1] and is used to dynamically adjust the ratio of entropy value to error, and reduce the ratio of entropy value in the later stage of training;

[0113] H(s,a)=-logπ(s,a) (7)

[0114] π(s,a) represents the probability of taking action a in state s

[0115] The uncertainty of actions chosen by the model in exploring the environment will decrease. In order to prevent the data from running in the building environment from directly entering the data repository without being extracted, the data from real-time operation is regarded as high-value data and input into the agent for training together with other calculated high-value data. While training, the real-time operation data is also entered into the entropy-based data screening method to calculate the value, and the accumulated data is entered into the data repository. The agent updates the strategy through training with high-value data, and also updates the entropy-based data screening method, recalculates the value of the data in the data repository and uses it for training. The overall process is as follows: Figure 3 shown.

[0116] Prioritize the processing of identified high-value data, prioritize updating the policy network and value network, and ensure that key data in the learning process is fully utilized. Through the high-value data priority replay mechanism, high-value data is stored in the experience replay pool first, and sampled and used multiple times during the training process to improve the efficiency of strategy optimization.

[0117] The role of strategy evaluation is to optimize the model and reduce the control error. From formula (4), we can see that when maxQ(s t+1 ,a t+1 )When maximizing the Q value, since the Q value itself is an inaccurate estimate and may contain random noise, the maximization operation tends to choose a Q value that contains positive noise. This positive bias leads to the overestimation problem, that is, the overestimation of the expected value. Since the Q value is used in each update, the overestimation bias will gradually accumulate, resulting in a Q value that is much higher than the actual value. This bias is further amplified during the continuous update process, especially in the case of high noise. The overestimation problem is mainly manifested in the agent's overestimation of the value of certain state-action pairs during training, resulting in the strategy selecting actions that are too optimistic, thus affecting the effectiveness of the strategy.

[0118] In order to solve the overestimation problem in the existing DRL method, there is an existing technology that uses two independent Q functions to reduce the overestimation problem. Specifically, the two Q functions are updated independently and the maximization operation is used alternately to reduce the deviation. However, this method only uses randomness to reduce the overestimation, and does not fundamentally remove the accumulation of noise. It only reduces the overestimation problem of the Q value.

[0119] For example, in some traditional Q-value function estimation algorithms, the update formula is shown in equations (i) and (ii):

[0120] y=r+γmaxQ(s t+1 ,a t+1 ;θ′) (i)

[0121]

[0122] Where γ is the discount factor, which is used to attenuate the impact of future rewards and ensure that the agent considers both immediate rewards and future rewards when making decisions; θ′ is the parameter of the target network; θ is the parameter of the current Q network; L(θ) is the loss function, which represents the loss value of the current Q network parameters; Indicates expected value.

[0123] Since the maximization operation maxQ(s t+1 ,a t+1 ; θ′) will tend to choose an overestimated Q value, which will cause the problem of a high y value in the formula. The result of the impact value function is closely related to the action selected by the strategy, and the accumulation of errors will also gradually affect the action selected by the strategy. Therefore, regularizing the action selected by the strategy and the action selected by the action network as a reference to the variable affecting the result in the loss function update process can greatly reduce the overestimation of the Q value. In an embodiment of the present invention, a strategy evaluation is added to the dual Q function structure, which greatly reduces the overestimation of the Q value and improves the accuracy of the control strategy. The specific process is as follows Figure 3 As shown, the strategy evaluation update process is as shown in formula (8).

[0124]

[0125] In the formula, π(s) represents the action selected by the strategy in the agent state s, and y is Figure 3 minQ' in.

[0126] Figure 4The error in includes value error and noise error. The previous loss function only considers the error of the Q-value function. The update method of policy evaluation takes into account the role of action difference in eliminating noise errors. The update process of policy evaluation can eliminate the influence of random noise on the overestimation of Q-value results. Introducing the policy evaluation method into the model structure of the dual Q function can effectively eliminate the accumulation of deviations based on random noise overestimation.

[0127] The present invention also provides an embodiment for showing the preferred process and beneficial effects of the present invention.

[0128] like Figure 5 As shown, it includes the following processes:

[0129] Step 1: Collect various environmental parameters and control status data including indoor and outdoor temperature, humidity, equipment operating status, energy consumption data, etc. through sensors and controllers.

[0130] Step 2: Data preprocessing. Clean the collected data and remove outliers. Normalize the data so that the data is on the same scale for easy model training.

[0131] Step 3: Design a reward function based on the requirements of temperature control and energy consumption optimization. Input the actual operation results of the building into the reward function and calculate the reward value of the system control. The actual environmental data corresponding to the calculated reward value r is filtered and processed through the entropy-based data screening layer to obtain high-value data.

[0132] Step 4: Initialize the two Q networks of the critic network and the two target Q networks, with parameters θ1, θ2 and θ1′, θ2′ respectively. The initial parameters θ and θ′ are the same. Initialize the actor network parameters and the data playback experience pool D.

[0133] Step 5: At each time step t, the agent obtains the current state s from the environment t , select action a, obtain reward r after executing the action, and transfer to the new state s t+1 . Evaluate the experiences explored in the third step and eliminate those with poor results. i ,a,r,s i+1 t∈i) is stored in the experience replay pool D. The fourth step should be repeated several times until there are at least 50 experiences in the experience replay pool D.

[0134] Step 6: Identify high-value data, calculate the value of each piece of data, and sort them according to their actual learning value. When extracting from the experience pool, data with more learning value can be extracted first (this can be done by setting a third preset threshold). Sample small batches of experience for the agent to learn. For the experience sampled in step 4 (si ,a,r,s i+1 t∈i) calculates the Q value of the target, and compares the Q value calculated by the Q network with the target Q value, uses the gradient descent method to minimize the loss function, and updates the weights θ1, θ2 of the critic Q network through formula (7).

[0135] Step 7: Update the critic target Q network through the updated critic Q network in step 6. Update the actor policy network through the policy evaluation of the action selection a of the Q network updated in step 6 and the policy selection action π(s) of the policy network through formula (4). Use the policy evaluation result to correct the output of the Q network to reduce the overestimation bias caused by random noise.

[0136] Step 8: Dynamically adjust the β value of the reward function according to the control results to improve the control accuracy of the intelligent agent in achieving specific goals.

[0137] Step 9: Return to step 5 and train the agent 1200 times. Calculate the average temperature offset and average energy consumption. Save the network parameters and model weights.

[0138] Step 10: End the process.

[0139] In the process of constructing the model, the learning process of the control strategy is greatly accelerated by giving priority to high-value data. The priority processing mechanism of high-value data ensures that the system optimizes the strategy under the guidance of key data, avoiding the interference of inefficient data, and through the priority high-value data playback mechanism, high-value data is used for training multiple times, which further improves the learning efficiency. The use of the strategy evaluation method greatly reduces the overestimation problem of the Q-value function, and combines the dual Q function to optimize the output of the Q network, improves the stability of the strategy in complex environments, and enhances the robustness of the system. More accurate control strategies enable the HVAC system to be fine-tuned according to actual needs, reducing unnecessary energy consumption. The present invention has achieved remarkable beneficial effects in improving the temperature control and energy consumption optimization of the HVAC system. The experimental results are as follows. Figure 7 and Figure 8 The method of the present invention uses a precise control strategy so that the system can better maintain the set temperature range, and can effectively reduce the energy consumption of the system and improve the energy utilization efficiency.

[0140] In summary, the present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning. The temperature control and energy consumption optimization of the HVAC system under the interference of external conditions are realized through an entropy-based data screening method, a strategy evaluation module and a designed reward function. The entropy-based data screening method calculates the value of data for intelligent agent training through an error formula, and repeatedly uses high-value data for intelligent agent training to enhance the training effect. The strategy evaluation module further solves the problem of overestimation of Q value on the dual Q structure and corrects the output of Q value. Not only does the dual Q network reduce the influence of random Q value, but the strategy evaluation also adds the regularization of strategy and actual action to the loss function update process, eliminates the influence of random noise on Q value, and ensures the accuracy of strategy output. Designing a reward function that can control temperature and optimize energy consumption allows the intelligent agent to balance the contradictions between multiple tasks and find the optimal solution for the intersection of multiple objective tasks. The new reward function enables the HVAC system to control the temperature within the temperature range while reducing unnecessary energy consumption.

[0141] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0142] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for temperature control and energy consumption optimization of HVAC system based on deep reinforcement learning, characterized in that: include: S1 calculates the reward function of the control model based on the acquired environmental parameters and HVAC control status data according to the requirements of temperature control and energy consumption optimization; The control model includes a first branch, a second branch and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer arranged in sequence along the data flow direction, the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer arranged in sequence along the data flow direction, the actor-critic network includes an actor network and a critic network arranged in sequence in the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks; The critic network includes a critic first Q network and a critic second Q network which are arranged in parallel with each other; the critic first Q network and the critic second Q network each include two sub-networks, and the two sub-networks of the critic first Q network are used to calculate and obtain the Q1 value and the Q2 value respectively; The sub-networks of the critic second Q network are respectively used to calculate the minQ' value, so that the actor critic network can calculate the Q1 value, Q2 value and minQ' value to obtain the value for reverse updating the critic first Q network and the critic second Q network; The reward function includes: Pass-through R=R 能耗 +βR 温度 (1) Construct the overall reward function; where the energy consumption reward value R 能耗 and temperature reward value R 温度 Separately through R 能耗 =W 平均 / W 当前平均 (2) R 温度 =-(T 当前 -T 设定均值 ) 2 / (T 设定上限 -T 设定均值 ) 2 (3) Calculated; β is the ratio of the adjustment temperature reward to the total reward; In formula (2), W 平均 It refers to the average energy consumption of the system under the control of the control algorithm, W 当前平均 It refers to the average energy consumption from the start of the algorithm to the current state; in formula (3), T 当前 Refers to the current temperature, T 设定均值 Refers to the average temperature of the temperature range to be maintained, T 设定上限 It refers to the highest temperature in the temperature range; S2 inputs the actual environment data in the building to which the HVAC belongs and the HVAC control state data into the reward function, calculates the reward value r of the control system, and screens the actual environment data corresponding to the calculated reward value r through the entropy-based data screening layer to obtain high-value data; S3 initializes the four Q networks of the critic network, initializes the actor network, and returns the initialized data to the experience pool; S4 obtains the state s of the agent in the current environment at each time step t , in the current state s t Select and execute action a, and change the current state s t and action a are input into the reward function to obtain the reward value r, so that the agent is transferred to the new state s after executing action a t+1 ; S5 evaluates the execution effect of step S4, and if the execution effect is lower than the first preset threshold, re-executes step S4, otherwise, stores the execution result in the experience pool; S6 repeatedly executes steps S4 and S5 multiple times, and stores the execution results in the experience pool s i ,a,r,s i+1 t∈i; determine whether the action strategy in the experience pool is greater than the second preset threshold, if so, execute step S6, otherwise, return to execute step S3; S7 passes through the strategy evaluation layer and uses the formula Q(s,a)←Q(s,a)+α[r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )] (4) Calculate the experience pool s i ,a,r,s i+1 The target Q value of the action strategy in t∈i; calculated by the gradient descent method δ=|r+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (5) δ=|r+αH(s,a)+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (6) H(s,a)=- logπ(s,a) (7) The difference between the target Q value obtained by calculation and the target Q value minimized by the gradient descent method; in formula (4), s is the state representing the dimensional data affecting the control target; a is the control action calculated by the agent through the strategy in the current state; r is the reward obtained by taking action in the current state; Q(s t ,a t ) is the cumulative target Q value that the agent is expected to obtain in the future under a certain state and action; γ is a discount factor used to measure the importance of future rewards; in formula (7), π(s,a) represents the probability of taking action a in state s; S8 Action a of the critic Q network based on updating weights t+1 , combined with the strategy evaluation of the action π(s) selected by the actor network, through the loss function Calculate the error of the execution results of steps S4 to S8, and then use the error to reversely update the critic second Q network; in formula (8), π(s) represents the action selected by the strategy under the agent state s; y is calculated by the formula y=r+γmax Q(s t+1 ,a t+1 ; θ′) is calculated, where γ is the discount factor; S9 dynamically adjusts the β value of the reward function according to the control result to improve the control accuracy of the intelligent agent in achieving the specific goal; S10 returns to execute steps S4 to S9, trains the agent multiple times, calculates the temperature offset and average energy consumption after the training is completed, and saves the network parameters and the optimal control model weights; S11 controls the HVAC using the optimal control model obtained after executing step S11.

2. The method according to claim 1, characterized in that Step S2 includes: S21 inputs each piece of actual environment data and HVAC control state data in the building to which the HVAC belongs into the reward function, and calculates and obtains a reward value r corresponding to each piece of actual environment data and HVAC control state data; S22 sorts all reward values ​​r, and obtains the high-value data by comparing all reward values ​​r with a third preset threshold.

3. The method according to claim 1, characterized in that Step S1 also includes a process of cleaning and normalizing the acquired environmental parameters and HVAC control status data.

Images