Temperature control and energy consumption optimization method of HVAC system based on deep reinforcement learning

Abstract

The present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning. Through an entropy-based data screening method, a policy evaluation module, and a designed reward function, the temperature control and energy consumption optimization of the HVAC system are achieved under external interference. The entropy-based data screening method calculates the value of data for intelligent agent training through an error formula, repeatedly uses high-value data for intelligent agent training, and enhances the training effect. The policy evaluation module further addresses the overestimation of Q values ​​based on the dual-Q structure, corrects the output of Q values, and reduces the impact of random Q values. Regularization of the policy and actual action is also added to the loss function update process, eliminating the impact of random noise on Q values ​​and ensuring the accuracy of the policy output. The reward function designed for controlling temperature and optimizing energy consumption enables the intelligent agent to balance the contradictions between multiple tasks, find the optimal solution for the intersection of multiple target tasks, and enable the HVAC system to control the temperature to remain within the temperature range while reducing unnecessary energy consumption.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning. Background Art

[0002] Heating, ventilation, and air conditioning (HVAC) systems are an integral part of modern buildings, designed to provide a comfortable indoor environment. HVAC systems primarily refer to a building's heating, ventilation, and air conditioning systems, controlling indoor temperature and humidity to enhance user comfort. Conventional control systems typically focus solely on temperature, but with the ongoing implementation of the "dual carbon" initiative, reducing the energy consumption of HVAC systems is crucial. Since HVAC system operation accounts for over one-third of a building's energy consumption, there is significant potential for reducing this energy consumption.

[0003] With the development of intelligent technology, HVAC systems are equipped with a variety of sensors to monitor the real-time status of equipment in the controlled environment, allowing for temperature control and energy management. Many advanced intelligent algorithms can be used to control the real-time temperature of system equipment, but the control performance is insufficient. Furthermore, there is a certain contradiction between temperature control and energy optimization, and existing algorithms struggle to balance these different control objectives. In current applications, predictive models are often used to predict future conditions and use these predicted conditions to implement control measures in advance. However, many external interferences can severely impact the prediction results, resulting in substandard control performance. Furthermore, achieving multiple objectives simultaneously requires more than just predictive control. Therefore, to mitigate the impact of external interference and improve control performance, training a stable and high-performance algorithm is crucial for temperature control and energy optimization. Summary of the Invention

[0004] An embodiment of the present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning, which is used to solve the technical problems existing in the prior art.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning, including:

[0007] S1 calculates the reward function of the control model based on the acquired environmental parameters and HVAC control status data according to the requirements of temperature control and energy consumption optimization;

[0008] The control model includes a first branch, a second branch, and an intelligent agent. The first branch includes a reward function and an entropy-based data screening layer sequentially arranged along the data flow direction. The entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch. The second branch includes an actor-critic network and a policy evaluation layer sequentially arranged along the data flow direction. The actor-critic network includes an actor network and a critic network sequentially arranged along the data flow direction. The actor network has an action policy network, and the critic network has four Q networks.

[0009] The critic network includes a critic first Q network and a critic second Q network set in parallel with each other; the critic first Q network and the critic second Q network each include two sub-networks, and the two sub-networks of the critic first Q network are used to calculate the Q1 value and the Q2 value respectively; the sub-networks of the critic second Q network are used to calculate the minQ' value, so that the actor-critic network can calculate the Q1 value, Q2 value and minQ' value to be used for reverse updating of the critic first Q network and the critic second Q network;

[0010] The reward function includes:

[0011] Pass-through

[0012] R=R 能耗 +βR 温度 (1) Construct the overall reward function; where the energy consumption reward value R 能耗 and temperature reward value R 温度 Separately through

[0013] R 能耗 =W 平均 / W 当前平均 (2)

[0014] R 温度 =-(T 当前 -T 设定均值 ) 2 / (T 设定上限 -T 设定均值 ) 2 (3) is calculated; β is the ratio of the adjustment temperature reward to the total reward; in formula (2), W 平均 Refers to the average energy consumption of the system under the control of the control algorithm, W 当前平均 It refers to the average energy consumption from the start of the algorithm to the current state; in formula (3), T 当前 Refers to the current temperature, T 设定均值 Refers to the average temperature of the temperature range to be maintained, T 设定上限 It refers to the highest temperature in the temperature range;

[0015] S2 inputs the actual environmental data of the HVAC building and the HVAC control status data into the reward function, calculates the reward value r of the control system, and filters the actual environmental data corresponding to the calculated reward value r through the entropy-based data screening layer to obtain high-value data;

[0016] S3 initializes the four Q networks of the critic network, initializes the actor network, and returns the initialized data to the experience pool;

[0017] S4 obtains the state s of the agent in the current environment at each time step t , in the current state s t Select and execute action a, and change the current state s t And action a is input into the reward function to obtain the reward value r, so that the agent is transferred to the new state s after performing action a t+1 ;

[0018] S5 evaluates the execution effect of step S4. If the execution effect is lower than the first preset threshold, step S4 is re-executed. Otherwise, the execution result is stored in the experience pool.

[0019] S6 repeats steps S4 and S5 multiple times and stores the execution results in the experience pool s i ,a,r,s i+1 t∈i; determine whether the action strategy in the experience pool is greater than the second preset threshold, if so, execute step S6, otherwise return to execute step S3;

[0020] S7 passes through the policy evaluation layer and uses

[0021] Q(s,a)←Q(s,a)+a[r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )](4) Calculate the experience pool s i ,a,r,s i+1 The target Q value of the action strategy in t∈i; calculated by the gradient descent method

[0022] δ=|r+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (5)

[0023] δ=|t+αH(s,a)+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (6)

[0024] H(s,a)=- logπ(s,a) (7) The difference between the target Q value obtained by calculation and the target Q value minimized by the gradient descent method; In formula (4), s is the state representing the various dimensional data that affect the control target; a is the control action calculated by the agent through the strategy in the current state; r is the reward obtained by taking action in the current state; Q(s t ,a t ) is the cumulative target Q value that the agent is expected to obtain in the future under a certain state and action; γ is a discount factor used to measure the importance of future rewards; in formula (7), π(s,a) represents the probability of taking action a in state s;

[0025] S8 Action a of the critic Q network based on updated weights t+1 , combined with the strategy evaluation of the action π(s) selected by the actor network, through the loss function

[0026]

[0027] Calculate the error of the execution results of steps S4 to S8, and then use the error to reversely update the critic's second Q network; in formula (8), π(s) represents the action selected by the strategy under the agent state s; y is calculated by the formula y = r + γmax Q(s t+1 ,a t+1 ; θ′) is calculated, where γ is the discount factor;

[0028] S9 dynamically adjusts the β value of the reward function based on the control results to improve the control accuracy of the intelligent agent in achieving specific goals;

[0029] S10 returns to executing steps S4 to S9, trains the agent multiple times, calculates the temperature offset and average energy consumption after the training is completed, and saves the network parameters and the optimal control model weights;

[0030] S11 controls the HVAC using the optimal control model obtained after executing step S11.

[0031] Preferably, step S2 includes:

[0032] S21 inputs each piece of actual environmental data and HVAC control status data in the building to which the HVAC belongs into the reward function, and calculates a reward value r corresponding to each piece of actual environmental data and HVAC control status data;

[0033] S22 sorts all reward values ​​r and obtains high-value data by comparing all reward values ​​r with a third preset threshold.

[0034] Preferably, step S1 further includes a process of performing data cleaning and normalization processing on the acquired environmental parameters and HVAC control status data.

[0035] As can be seen from the technical solutions provided by the above-described embodiments of the present invention, the present invention provides a method for temperature control and energy optimization of an HVAC system based on deep reinforcement learning. Through an entropy-based data screening method, a policy evaluation module, and a designed reward function, the method achieves temperature control and energy optimization of the HVAC system under external interference. The entropy-based data screening method calculates the value of data for agent training using an error formula, repeatedly reusing high-value data for agent training to enhance training effectiveness. The policy evaluation module further addresses the overestimation of Q values ​​based on the dual-Q architecture, correcting the output of Q values. Not only does the dual-Q network reduce the impact of random Q values, but the policy evaluation also incorporates regularization of the policy and actual action into the loss function update process, eliminating the influence of random noise on Q values ​​and ensuring the accuracy of the policy output. Designing a reward function that can both control temperature and optimize energy consumption allows the agent to balance conflicts between multiple tasks and find the optimal solution for the intersection of multiple objective tasks. This new reward function enables the HVAC system to maintain temperature within a specified range while reducing unnecessary energy consumption.

[0036] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A process flow chart of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0039] Figure 2 This is a diagram of the energy consumption optimization model structure of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0040] Figure 3 A schematic diagram of the critic-actor network for the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0041] Figure 4 A schematic diagram illustrating the operation process of the entropy-based data screening method of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0042] Figure 5Schematic diagram of the strategy evaluation and update process of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0043] Figure 6 A schematic diagram of the state and action modeling of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0044] Figure 7 A schematic diagram of a preferred embodiment of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0045] Figure 8 A schematic diagram of the control effect of the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention;

[0046] Figure 9 A comparison chart of energy consumption between the HVAC system temperature control and energy consumption optimization method based on deep reinforcement learning provided by the present invention and other methods in the prior art. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0048] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0049] The present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning, which is used to solve the following technical problems existing in the prior art:

[0050] In a building's HVAC system, the primary goal is to maintain a constant indoor temperature while minimizing energy consumption. Existing HVAC system control methods primarily include traditional Proportional Integral Differential (PID) control and Model Predictive Control (MPC).

[0051] PID control is a simple and widely used control method. A PID controller reduces control error—the difference between a setpoint and its actual value—by controlling the output of a device. The advantages of a PID controller are its simple structure and ease of design and implementation. However, PID controllers have the following limitations: Long adjustment time: PID control requires a significant amount of debugging time to obtain optimal control parameters. Poor adaptability: PID control has poor adaptability to changes in system parameters and external disturbances, which can easily lead to decreased control performance. Difficulty controlling nonlinear systems: PID control is ineffective for complex nonlinear systems and struggles to meet the high-performance requirements of the system.

[0052] Model predictive control is an advanced model-based control method that optimizes control inputs by predicting system behavior over a period of time. The advantages of MPC include:

[0053] a. Optimize control performance. MPC can achieve better control performance by optimizing the objective function.

[0054] b. Multivariable control. MPC can handle multiple-input multiple-output (MIMO) systems and is suitable for complex control tasks.

[0055] c. Considering constraints: MPC can consider various system constraints during the optimization process, such as physical limitations of inputs and outputs.

[0056] Although MPC has the above advantages, its application also has some problems:

[0057] a. High computational complexity. MPC requires optimization calculations within each control cycle, which requires a large amount of computation and places high demands on real-time performance.

[0058] b. Strong model dependence. The performance of MPC depends on the accuracy of the system model. For systems with inaccurate models or large environmental changes, the control effect will be affected.

[0059] c. Complex implementation. The implementation of MPC requires high computing power and algorithm complexity, which increases the development and maintenance costs of the system.

[0060] To overcome the limitations of traditional control methods, Deep Reinforcement Learning (DRL) has been widely used in HVAC systems in recent years as an emerging control strategy. DRL can achieve efficient control in complex environments by learning the state-action relationship of the environment. The advantages of DRL methods include:

[0061] a. Strong adaptability. DRL can adapt to changes in the environment through continuous learning and improve control performance.

[0062] b. Processing high-dimensional data. DRL can handle high-dimensional state and action spaces and is suitable for complex control tasks.

[0063] c. Learning nonlinear control strategies. DRL can learn complex nonlinear control strategies and improve the control accuracy of the system.

[0064] However, DRL also faces some challenges in practical applications: Long training time: The DRL training process requires a large amount of computing resources and time, resulting in a long system development cycle. Poor stability: DRL is prone to instability and convergence difficulties during training, affecting control performance. High data requirements: DRL requires a large amount of training data, resulting in a high dependence on data and increasing system development costs.

[0065] The above-mentioned prior art has the following disadvantages:

[0066] 1. Because precise HVAC system control requires numerous factors, learning effective strategies requires a large amount of interactive training data. This requires reducing the model's reliance on data volume and improving the utilization of existing training data.

[0067] 2. HVAC systems typically have complex nonlinear characteristics. Existing methods find it very difficult to establish accurate mathematical models in the face of these nonlinear characteristics and are unable to fully adapt to the dynamic changes of the system.

[0068] 3. Existing control methods are all single-objective control methods. When faced with multi-task objectives, the model needs to be adjusted, which makes it difficult to meet multi-task objectives and affects the control effect.

[0069] In view of this, the present invention aims to solve the following problems:

[0070] 1. To reduce the model's reliance on data volume, the model improves its utilization of previously explored training data. This involves performing a value analysis on existing data, screening out data with greater training value, and actively extracting and learning from data with even greater training value.

[0071] 2. Achieve precise control of complex systems that adapt to dynamic changes, using deep reinforcement learning models to establish dynamic relationships between inputs and outputs. This requires making the control strategies adopted by the model more robust, designing a strategy evaluation module to assess strategy effectiveness, and further optimizing effective strategies.

[0072] 3. Conventional reward functions lack the ability to design multi-objective tasks, making it difficult for control systems to simultaneously meet multiple objectives. The reward function designed by this method considers the optimal solution between the various objectives, enabling the agent to complete the requirements of different task objectives.

[0073] See also Figure 1 The present invention provides a method for temperature control and energy consumption optimization of an HVAC system based on deep reinforcement learning, comprising the following steps:

[0074] S1 calculates the reward function based on the obtained environmental parameters and HVAC control status data according to the requirements of temperature control and energy consumption optimization;

[0075] like Figure 2 As shown, the method provided by the present invention designs an improved control model, including a first branch, a second branch and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer arranged in sequence along the data flow direction, the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer arranged in sequence along the data flow direction, the actor-critic network includes an actor network and a critic network arranged in sequence along the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks;

[0076] S2 inputs the actual environmental data of the HVAC building into the reward function, calculates the reward value r of the control system, and filters the actual environmental data corresponding to the calculated reward value r through the entropy-based data screening layer to obtain high-value data;

[0077] S3 initializes the four Q networks of the critic network, initializes the actor network, and returns the initialized data to the experience pool;

[0078] S4 obtains the state s of the agent in the current environment at each time step t , in the current state s t Select and execute action a, and change the current state s t And action a is input into the reward function to obtain the reward value r, so that the agent transfers to the new state s after performing action a t+1 ;

[0079] S5 evaluates the execution effect of step S4. If the execution effect is lower than the first preset threshold, step S4 is re-executed. Otherwise, the execution result is stored in the experience pool.

[0080] S6 repeatedly executes steps S4 and S5 and stores the execution results in the experience pool (s i ,a,r,s i+1t∈i); determine whether the action strategy in the experience pool is greater than the second preset threshold, if so, execute step S6, otherwise return to execute step S3;

[0081] S7 calculates the experience pool (s i ,a,r,s i+1 The target Q value of the action strategy in t∈i) is minimized by gradient descent with the target Q value and the Q value of the critic network, and then the weight of the critic network is calculated and updated;

[0082] S8 Action a of the critic Q network based on updated weights t+1 , combined with the policy evaluation of the action π(s) selected by the actor network, the actor network is updated by calculation;

[0083] S9 dynamically adjusts the parameter value of the reward function according to the control results to improve the control accuracy of the intelligent agent in achieving specific goals;

[0084] S10 returns to executing steps S4 to S9, trains the agent multiple times, calculates the temperature offset and average energy consumption after the training is completed, and saves the network parameters and the optimal control model weights;

[0085] S11 uses the optimal control model obtained after executing step S11 to control the HVAC, so that the energy consumption of the HVAC reaches the optimal state under the premise that the indoor temperature meets the requirements.

[0086] In the preferred embodiment provided by the present invention, the specific implementation process of each key step is as follows:

[0087] 1. Designing the Reward Function

[0088] The task objective comprehensively considers the system's energy consumption and temperature requirements. By optimizing the objective function and adjusting the control strategy, a comprehensive optimization of energy consumption and temperature is achieved, ensuring that the system energy consumption is minimized while meeting the user's temperature requirements [21,24]. To implement the reward function design, the state and action must first be modeled.

[0089] (1) State and action modeling

[0090] The status function records the equipment parameters of the building's HVAC system, enabling real-time monitoring of the system. This invention primarily addresses the tasks of temperature control and energy optimization in HVAC systems, requiring the recording of task-related status features. The status function records HVAC power consumption, fan coil unit power consumption, gas boiler power consumption, indoor zone temperature, outdoor temperature, outdoor air pressure, and outdoor humidity. Actions are control measures executed based on the status, including the on / off and setpoint values ​​of the heat pump, fan coil unit power consumption, mixer setpoints, and zone setpoints.

[0091] The premise for achieving the energy consumption optimization task is to control the indoor area temperature within the set range [21,24]. The external variables that affect temperature control are outdoor temperature, outdoor air pressure and outdoor humidity. Under the influence of external variables, the system switches the heat pump and temperature setting and the fan coil switch and temperature setting, inputs the two into the mixer, and reaches the set temperature of the mixer and inputs it into the indoor area to control the temperature. If the indoor area temperature is lower than the set range [21,24], according to the external influences related to outdoor temperature, air pressure and humidity, these states are input into the intelligent agent, and its strategy network outputs actions such as heat pump, fan coil and mixer set values ​​to control the temperature so that the indoor temperature returns to the set range. The specific process is as follows: Figure 4 shown.

[0092] HVAC power consumption primarily comes from fan coil units and gas boilers. To conserve energy and accurately achieve the mixer's setpoint temperature, it's necessary to properly control the heat pump's on / off and fan coil unit's on / off settings. When the device's controlled temperature is within the setpoint range, it's important to optimize the device's on / off frequency and control values. By controlling each device within the system to maintain a temperature control range close to the indoor temperature (close to 24°C in summer and 21°C in winter), energy consumption is reduced, and unnecessary heating or cooling capacity is minimized, ultimately lowering the system's overall energy consumption.

[0093] (2) Reward Function Modeling

[0094] In deep reinforcement learning, the reward function is often adjusted to ensure that the agent achieves different task objectives. Positive rewards encourage the agent's current behavior, while negative rewards penalize the agent's actions. The overall reward function design of this invention consists of energy optimization and temperature control, as shown in Equation (1). The energy and temperature rewards are shown in Equations (2) and (3).

[0095] R=R 能耗 +βR 温度 (1) By adjusting the proportion of temperature reward to total reward by β, the target reward function can be dynamically adjusted according to the training effect;

[0096] R 能耗 =W 平均 / W 当前平均 (2)

[0097] where R 能耗 The value of is usually greater than 1; 平均 Refers to the average energy consumption of the system under the control of the control algorithm; W 当前平均 Refers to the average energy consumption from the start of the algorithm to the present;

[0098] R 温度 =-(T当前 -T 设定均值 ) 2 / (T 设定上限 -T 设定均值 ) 2 (3)

[0099] where R 温度 The value of is less than 0; T 当前 Refers to the current temperature; T 设定均值 Refers to the average temperature of the temperature range to be maintained; T 设定上限 It refers to the highest temperature in the temperature range.

[0100] The temperature function only works if the system stays within the temperature range (-1, 0). If the temperature exceeds the set range, the temperature reward will be a large negative number. Combining the energy consumption and temperature functions, the agent will only receive a positive reward if it keeps the system within the temperature range. The lower the energy consumption, the greater the reward.

[0101] The model structure of the present invention obtains reward values ​​based on the state and action of the building environment operation data by designing a new reward function to achieve multiple task goals, and then inputs the calculated data into the entropy-based data screening method to screen out high-value data (environmental data). The high-value training data is evaluated through two Q networks to optimize the Q value, and then the minimum value is taken to reduce the error generated by the prediction. The optimal Q value is used to update the intelligent agent. The structure of the entire model is as follows Figure 2 As shown, it includes a first branch, a second branch and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer arranged in sequence along the data flow direction, and the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer arranged in sequence along the data flow direction, the actor-critic network includes an actor network and a critic network arranged in sequence in the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks.

[0102] The critic network of the actor-critic network includes a first Q-network and a second Q-network, which are set up in parallel. The second Q-network is the target Q-network. The first Q-network and the second Q-network each include two subnetworks. The two subnetworks of the first Q-network are used to calculate the Q1 value and the Q2 value, respectively. The subnetworks of the second Q-network are used to calculate the minQ' value, so that the actor-critic network can calculate the error for reverse updating the first Q-network and the second Q-network using the Q1 value, Q2 value, and minQ' value. Through the processing of the above actor-critic network, the random relationship between state and action is maintained as much as possible, reducing the error caused by the strong connection between state and action.

[0103] Data screening method based on entropy

[0104] The data screening method based on entropy is used to screen high-value data and give priority to the input of intelligent training. Data is the actual data inside the building, which refers to the four-tuple data (s) of current state, action, reward value and next state. i ,a,r,s i+1 t∈i). The state refers to the HVAC system power consumption, fan coil unit power consumption, gas boiler power consumption, zone temperature, outdoor temperature, outdoor air pressure, and outdoor humidity in the system. The action is the control variable selected by the agent based on the state and control strategy. This includes the on / off switch and setpoint of the heat pump, the on / off switch and setpoint of the fan coil, the setpoint temperature of the mixer, and the setpoint temperature of the zone. When the building takes a control action, it automatically generates data for the next state. The reward is calculated using the reward function based on the state and action. This data is used to calculate the value function, evaluating each piece of data's potential contribution to policy optimization.

[0105] Entropy is a key factor in determining high-value data. Higher entropy values ​​indicate higher value, indicating greater uncertainty in the strategy's predicted actions. High-value data with high uncertainty encourages the model to proactively explore unknown environments and search for the global optimal solution. High-value data refers to states or actions that are not previously explored by the current control strategy. These data deviate significantly from the current strategy, are more random, and have greater learning value. High-value data points are therefore more likely to optimize the strategy.

[0106] Entropy is used to measure the uncertainty of the model's sample predictions, ensuring that the strategy learned by the agent is the optimal solution after exploring the global situation, preventing the agent from prematurely converging to a suboptimal strategy. By prioritizing these high-entropy samples for labeling, the model can actively learn and improve. Incorporating the uncertainty of entropy into the error formula allows the model to actively identify and filter data with higher value, accelerate the priority processing of high-value data, reduce the number of low-value data training times, and effectively improve the learning efficiency and performance of the control algorithm. High value is determined by the size of the Q-value error δ and the entropy value. The Q-value calculation formula is shown in Equation (4).

[0107] Q(s,a)←Q(s,a)+α[r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )] (4)

[0108] s is the state representing the various dimensional data that affect the control target; a is the control action calculated by the agent through the strategy in the current state; r is the reward obtained by taking action in the current state; Q(s t ,a t ) is the cumulative reward that the agent is expected to receive in the future under a certain state and action; γ is a discount factor that measures the importance of future rewards;

[0109] The original error formula is shown in the following formula (5), which calculates the difference between the Q value generated by the actual action and the Q value generated by the predicted strategy. In this module, the difference δ is used to mark the value of the data. The value data generated by formula (5) will cause the model to fall into a suboptimal solution. The main reason is that the agent using the original error function does not fully explore the environment. Therefore, the entropy of uncertainty is added to the original error formula to give a higher value to the state data that the agent has not explored. Encourage the agent to fully explore the environment until it finds the global optimum. The original error formula is shown in formula (5), the error formula with entropy is shown in formula (6), and the entropy formula is shown in formula (7):

[0110] δ=|r+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (5)

[0111] δ=|r+αH(s,a)+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (6)

[0112] Where δ is the entropy difference between the existing strategy and the explored strategy, including uncertainty, and its size represents the value of the data; H(s,a) is the entropy value in the current state; α ranges from [0,1] and is used to dynamically adjust the ratio of entropy to error, reducing the ratio of entropy in the later stages of training.

[0113] H(s,a)=-logπ(s,a) (7)

[0114] π(s,a) represents the probability of taking action a in state s

[0115] The uncertainty of the actions selected by the strategy adopted by the model exploration environment will decrease. In order to prevent the data generated by the building environment from entering the data repository directly without being extracted, the data generated by the real-time operation is regarded as high-value data and input into the intelligent agent for training together with other calculated high-value data. During training, the real-time operation data is also entered into the entropy-based data screening method to calculate the value, and the accumulated data is entered into the data repository. The intelligent agent updates the strategy through training with high-value data, and also updates the entropy-based data screening method, and recalculates the value of the data in the data repository for training. The overall process is as follows: Figure 3 shown.

[0116] Prioritize the processing of identified high-value data and prioritize updating the policy network and value network to ensure that key data is fully utilized during the learning process. Through the high-value data priority replay mechanism, high-value data is stored first in the experience replay pool and sampled and reused multiple times during the training process, improving the efficiency of policy optimization.

[0117] The role of strategy evaluation is to optimize the model and reduce the control error. According to formula (4), when maxQ(s t+1 ,a t+1 When maximizing the Q-value, since the Q-value itself is an inaccurate estimate and may contain random noise, the maximization operation tends to choose a Q-value that contains positive noise. This positive bias leads to the overestimation problem, that is, an overestimation of the expected value. Since the Q-value is used in every update, the overestimation bias gradually accumulates, causing the Q-value to be much higher than the actual value. This bias is further amplified during the continuous update process, especially in the presence of high noise. The overestimation problem mainly manifests itself in the agent overestimating the value of certain state-action pairs during training, resulting in overly optimistic actions selected by the policy, which in turn affects the effectiveness of the policy.

[0118] To address the overestimation problem in existing DRL methods, existing techniques have been developed to mitigate this problem by using two independent Q functions. Specifically, the two Q functions are updated independently and alternately maximized to reduce bias. However, this approach only uses randomness to reduce overestimation and does not fundamentally eliminate noise accumulation. It only mitigates the overestimation problem of Q values.

[0119] For example, in some traditional Q-value function estimation algorithms, the update formulas are shown in equations (i) and (ii):

[0120] y=r+γmaxQ(s t+1 ,a t+1 ;θ′) (i)

[0121]

[0122] Where γ is the discount factor, which is used to attenuate the impact of future rewards and ensure that the agent considers both immediate rewards and future rewards when making decisions; θ′ is the parameter of the target network; θ is the parameter of the current Q network; L(θ) is the loss function, which represents the loss value of the current Q network parameters; Indicates expected value.

[0123] Since the maximization operation maxQ(s t+1 ,a t+1 ; θ′) will tend to choose an overestimated Q value, which will cause the problem of a high y value in the formula. The result of the influence value function is closely related to the action selected by the strategy, and the accumulation of errors will also affect the action selected by the strategy step by step. Therefore, the overestimation of the Q value can be greatly reduced by regularizing the action selected by the strategy and the action selected by the action network in the loss function update process as a variable reference affecting the result. In the embodiment of the present invention, a method is proposed that adds a strategy evaluation on the basis of the dual Q function structure, which greatly reduces the overestimation of the Q value and improves the accuracy of the control strategy. The specific process is as follows Figure 3 As shown, the strategy evaluation update process is as shown in formula (8).

[0124]

[0125] In the formula, π(s) represents the action selected by the strategy in the agent state s, and y is Figure 3 minQ' in.

[0126] Figure 4The error in the Q-value function includes both value error and noise error. Previous loss functions only considered the error in the Q-value function. The policy evaluation update method takes into account the role of action differences in eliminating noise error. The policy evaluation update process can eliminate the influence of random noise on Q-value overestimation. Incorporating the policy evaluation method into the dual Q-function model structure effectively eliminates the accumulation of bias caused by random noise overestimation.

[0127] The present invention also provides an embodiment for demonstrating the preferred process and beneficial effects of the present invention.

[0128] like Figure 5 As shown, it includes the following processes:

[0129] Step 1: Collect various environmental parameters and control status data including indoor and outdoor temperature, humidity, equipment operating status, energy consumption data, etc. through sensors and controllers.

[0130] Step 2: Data preprocessing. Clean the collected data to remove outliers. Normalize the data so that it is on the same scale for easier model training.

[0131] Step 3: Design a reward function based on the requirements of temperature control and energy optimization. Input the building's actual operating results into the reward function to calculate the reward value for system control. The actual environmental data corresponding to the calculated reward value r is filtered through an entropy-based data filtering layer to obtain high-value data.

[0132] Step 4: Initialize the two Q networks of the critic network and the two target Q networks with parameters θ1, θ2 and θ1′, θ2′ respectively. The initial parameters θ and θ′ are the same. Initialize the parameters of the actor network and the data replay experience pool D.

[0133] Step 5: At each time step t, the agent obtains the current state s from the environment t , select action a, obtain reward r after executing the action, and transfer to the new state s t+1 . Evaluate the experiences found in the third step and eliminate those with poor results. i ,a,r,s i+1 t∈i) is stored in the experience replay pool D. The fourth step should be repeated several times until there are at least 50 experiences in the experience replay pool D.

[0134] Step 6: Identify high-value data, calculate the value of each piece of data, and sort them according to their actual learning value. When extracting from the experience pool, data with greater learning value can be extracted first (this can be done by setting a third preset threshold). Sample small batches of experience for the agent to learn. For the experience sampled in step 4 (si ,a,r,s i+1 t∈i) calculates the Q value of the target, and compares the Q value calculated by the Q network with the target Q value, uses the gradient descent method to minimize the loss function, and updates the weights θ1, θ2 of the critic Q network through formula (7).

[0135] Step 7: Update the critic target Q network using the updated critic Q network from step 6. Update the actor policy network using formula (4) by evaluating the action a selected by the updated Q network from step 6 and the policy action π(s) selected by the policy network. Use the policy evaluation results to correct the output of the Q network to reduce the overestimation bias caused by random noise.

[0136] Step 8: Dynamically adjust the β value of the reward function based on the control results to improve the control accuracy of the intelligent agent in achieving specific goals.

[0137] Step 9: Return to step 5 and train the agent 1200 times. Calculate the average temperature offset and average energy consumption. Save the network parameters and model weights.

[0138] Step 10: End the process.

[0139] In the process of constructing the model, the learning process of the control strategy is greatly accelerated by giving priority to high-value data. The priority processing mechanism of high-value data ensures that the system optimizes the strategy under the guidance of key data, avoiding the interference of inefficient data, and through the priority high-value data playback mechanism, high-value data is used for training multiple times, which further improves the learning efficiency. The use of the strategy evaluation method greatly reduces the overestimation problem of the Q-value function, and combines the dual Q function to optimize the output of the Q network, improves the stability of the strategy in complex environments, and enhances the robustness of the system. More accurate control strategies enable the HVAC system to be fine-tuned according to actual needs, reducing unnecessary energy consumption. The present invention has achieved significant beneficial effects in improving the temperature control and energy consumption optimization of the HVAC system. The experimental results are as follows. Figure 7 and Figure 8 The method of the present invention uses a precise control strategy to enable the system to better maintain the set temperature range, while effectively reducing the energy consumption of the system and improving energy utilization efficiency.

[0140] In summary, the present invention provides a method for temperature control and energy optimization of an HVAC system based on deep reinforcement learning. Through an entropy-based data screening method, a policy evaluation module, and a designed reward function, this method achieves temperature control and energy optimization of the HVAC system under external interference. The entropy-based data screening method calculates the value of data for agent training using an error formula, repeatedly reusing high-value data for agent training to enhance training effectiveness. The policy evaluation module further addresses the overestimation of Q values ​​based on the dual-Q architecture, correcting the output of Q values. Not only does the dual-Q network reduce the impact of random Q values, but the policy evaluation also incorporates regularization of the policy and actual actions into the loss function update process, eliminating the influence of random noise on Q values ​​and ensuring the accuracy of the policy output. Designing a reward function that can both control temperature and optimize energy consumption allows the agent to balance conflicts between multiple tasks and find the optimal solution for multi-task intersections. This new reward function enables the HVAC system to maintain temperature within a specified range while reducing unnecessary energy consumption.

[0141] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0142] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for temperature control and energy consumption optimization of HVAC systems based on deep reinforcement learning, characterized by: include: S1 calculates the reward function of the control model based on the obtained environmental parameters and HVAC control status data according to the requirements of temperature control and energy consumption optimization; The control model includes a first branch, a second branch, and an intelligent agent; the first branch includes a reward function and an entropy-based data screening layer arranged in sequence along the data flow direction, the entropy-based data screening layer is used to obtain high-value data and transmit it to the second branch; the second branch includes an actor-critic network and a strategy evaluation layer arranged in sequence along the data flow direction, the actor-critic network includes an actor network and a critic network arranged in sequence along the data flow direction, the actor network has an action strategy network, and the critic network has four Q networks; The critic network includes a critic first Q network and a critic second Q network set in parallel with each other; the critic first Q network and the critic second Q network each include two sub-networks, and the two sub-networks of the critic first Q network are used to calculate and obtain Q1 value and Q2 value respectively; The sub-networks of the critic's second Q-network are respectively used to calculate the minQ' value, so that the actor-critic network can be used to calculate the Q1 value, Q2 value and minQ' value for reverse updating the critic's first Q-network and the critic's second Q-network; The reward function includes: Pass-through R=R 能耗 +βR 温度 (1) Construct the overall reward function; where the energy consumption reward value R 能耗 and temperature reward value R 温度 Separately through R 能耗 =W 平均 / W 当前平均 (2) R 温度 =-(T 当前 -T 设定均值 ) 2 / (T 设定上限 -T 设定均值 ) 2 (3) Calculated; β is the ratio of the adjustment temperature reward to the total reward; In formula (2), W 平均 Refers to the average energy consumption of the system under the control of the control algorithm, W 当前平均 It refers to the average energy consumption from the start of the algorithm to the current state; in formula (3), T 当前 Refers to the current temperature, T 设定均值 Refers to the average temperature of the temperature range to be maintained, T 设定上限 It refers to the highest temperature in the temperature range; S2 inputs the actual environmental data of the building to which the HVAC system belongs and the HVAC control status data into the reward function, calculates a reward value r of the control system, and filters the actual environmental data corresponding to the calculated reward value r through the entropy-based data screening layer to obtain high-value data; S3 initializes the four Q networks of the critic network, initializes the actor network, and returns the initialized data to the experience pool; S4 obtains the state s of the agent in the current environment at each time step t , in the current state s t Select and execute action a, and change the current state s t And action a is input into the reward function to obtain the reward value r, so that the agent is transferred to the new state s after performing action a. t+1 ; S5 evaluates the execution effect of step S4. If the execution effect is lower than the first preset threshold, step S4 is re-executed. Otherwise, the execution result is stored in the experience pool. S6 repeats steps S4 and S5 multiple times and stores the execution results in the experience pool s i ,a,r,s i+1 t∈i; determine whether the action strategy in the experience pool is greater than the second preset threshold, if so, execute step S6, otherwise return to execute step S3; S7 passes the strategy evaluation layer and uses the formula Q(s,a)←Q(s,a)+α[r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )] (4) Calculate the experience pool s i ,a,r,s i+1 The target Q value of the action strategy in t∈i; calculated by the gradient descent method δ=|r+αH(s,a)+Q(s t+1 ,a t+1 )-Q(s t ,a t )| (6) The difference between the target Q value obtained by calculation and the target Q value minimized by the gradient descent method; in formula (4), s is the state representing the dimensional data that affects the control target; a is the control action calculated by the agent through the strategy in the current state; r is the reward obtained by taking action in the current state; Q(s t ,a t ) is the cumulative target Q value that the agent is expected to obtain in the future under a certain state and action; γ is a discount factor used to measure the importance of future rewards. In formula (6), H(s,a) represents the entropy value of the current state, which is calculated by formula H(s,a) = -logπ(s,a) (7). In formula (7), π(s,a) is the probability of taking action a in state s. S8 Action a of the critic Q network based on updated weights t+1 , combined with the strategy evaluation of the action π(s) selected by the actor network, through the loss function Calculate the error of the execution results of steps S4 to S8, and then use the error to reversely update the critic's second Q network; in formula (8), π(s) represents the action selected by the strategy under the agent state s; y is calculated by the formula y=r+γmax Q(s t+1 ,a t+1 ; θ′) is calculated, where γ is the discount factor; S9 dynamically adjusts the β value of the reward function according to the control result to improve the control accuracy of the intelligent agent in achieving the specific goal; S10 returns to executing steps S4 to S9, trains the agent multiple times, calculates the temperature offset and average energy consumption after the training is completed, and saves the network parameters and the optimal control model weights; S11 controls the HVAC using the optimal control model obtained after executing step S11.

2. The method according to claim 1, characterized in that Step S2 includes: S21 inputs each piece of actual environment data and HVAC control status data in the building to which the HVAC belongs into the reward function, and calculates and obtains a reward value r corresponding to each piece of actual environment data and HVAC control status data; S22 sorts all the reward values ​​r and obtains the high-value data by comparing all the reward values ​​r with a third preset threshold.

3. The method according to claim 1, characterized in that Step S1 also includes a process of performing data cleaning and normalization processing on the acquired environmental parameters and HVAC control status data.

Images