Heating ventilation air conditioner energy conservation and emission reduction method and system based on reinforcement learning and attention mechanism

By applying reinforcement learning and attention mechanisms in HVAC control, combined with Transformer network and multiple punishment reward functions, the problem of traditional control systems being difficult to adapt to environmental changes and unreasonable reward function design is solved, and more accurate HVAC control is achieved and long-term energy efficiency optimization.

CN118938666APending Publication Date: 2024-11-12NINGBO CAOCHUAN TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410987821.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional HVAC control systems are difficult to adapt to environmental changes, cannot achieve long-term energy efficiency and comfort optimization, and the unreasonable design of the reward function leads to poor application of reinforcement learning in HVAC control.

Method used

Using a method based on reinforcement learning and attention mechanism, a reinforcement learning model is constructed using the Transformer network. Through historical data training models, a reward function including temperature and humidity punishment, energy consumption punishment and temperature and humidity standard deviation punishment is designed to achieve optimization decisions for action sequences. .

Benefits of technology

It achieves more accurate HVAC control, adapts to system changes, reduces energy consumption, improves system stability, extends equipment life, meets long-term optimization goals, and reduces energy waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118938666A_ABST
    Figure CN118938666A_ABST
Patent Text Reader

Abstract

The invention discloses a heating ventilation air conditioner energy conservation and emission reduction method and system based on reinforcement learning and an attention mechanism. The reasoning stage comprises the steps that B1, a trained reinforcement learning model predicts a plurality of control combinations according to the current state; b2, selecting the first K control action combinations with the highest rewards by using a reward function; b3, predicting a new state after the action is executed for each selected control action combination; b4, predicting a new control action combination for each new state, and selecting K control action combinations with the highest accumulated rewards; b5, repeating the steps B3 to B4 until an optimal action sequence is found; and B6, taking the action sequence with the highest accumulated reward as a final decision. According to the scheme, reinforcement learning is used for controlling energy conservation and emission reduction of the heating ventilation air conditioner, more accurate heating ventilation control can be achieved, and the environment learning capacity of reinforcement learning is used for enabling the control system to automatically adjust the control strategy according to real-time feedback so as to adapt to system changes and uncertainty of external conditions; therefore, the control capability of the system fitting the reality is continuously improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of HVAC control, and in particular relates to a HVAC energy-saving and emission-reduction method and system based on reinforcement learning and attention mechanism. Background Art

[0002] Large-scale industrial HVAC systems consume a lot of energy and cover a wide area, with the total energy bill for a year generally ranging from millions to tens of millions.

[0003] Traditional HVAC control systems mostly use PID (proportional-integral-differential) controllers, which are simple and mature control methods that are widely used in industrial process control. However, they also have many limitations, such as single-factor calculation control, extensive debugging based on experience, static rule setting, oscillation debugging control, etc. With the development of artificial intelligence technology, especially the application of machine learning algorithms, many innovative methods have emerged in the field of HVAC control. For example, Ren Chunsheng and Wu Rihan proposed "The Latest Application and Prospect of Artificial Intelligence in HVAC", and Midea's "Green Digital Future" both mentioned the use of machine learning algorithms to achieve energy-saving and emission reduction control of HVAC.

[0004] However, in HVAC control, environmental conditions are constantly changing, such as indoor and outdoor temperature and humidity, occupancy rate, etc. General machine learning algorithms are difficult to adapt to the changing environmental conditions, resulting in poor control effects. Moreover, in the field of HVAC control, we hope to optimize long-term energy efficiency and comfort through automatic control. General machine learning algorithms can only meet short-term control effects and cannot meet the long-term optimization target requirements of HVAC control. To this end, some studies have proposed to apply reinforcement learning to HVAC control, such as "Q-learning-based air conditioning system cooling side reinforcement learning algorithm". This study specifically proposed a model-free optimization method based on reinforcement learning and used it for the optimization of air conditioning cooling water system, where the state variables are wet bulb temperature and system cooling load, the action variables are the frequency of fans and water pumps, and the optimization target is system efficiency COP. However, this method only focuses on the system efficiency COP, without considering human comfort, and is not very practical in HVAC control. Moreover, the success of long-term optimization depends largely on the design of the reward function. In the above scheme, there is an unreasonable reward function setting problem, which makes it impossible to reflect the advantages of reinforcement learning in HVAC control. Summary of the invention

[0005] The purpose of the present invention is to address the above problems and propose a HVAC energy-saving and emission-reduction method and system based on reinforcement learning and attention mechanism, which uses reinforcement learning to control the HVAC to achieve the purpose of energy saving and emission reduction while ensuring human comfort.

[0006] A method for energy saving and emission reduction of HVAC based on reinforcement learning and attention mechanism, comprising:

[0007] Training phase

[0008] Build a reinforcement learning model based on the Transformer network;

[0009] Construct sa sequence combination based on historical data, s represents the state of HVAC environmental feedback, and a represents the control action on HVAC;

[0010] Use sa sequence combination and cross loss function to train the reinforcement learning model;

[0011] Reasoning Phase

[0012] B1. The trained reinforcement learning model predicts multiple control combinations based on the current state;

[0013] B2. Use the reward function to select the top K highest reward control action combinations;

[0014] B3. For each selected control action combination, predict the new state after executing the action;

[0015] B4. Predict a new control action combination for each new state and select the K control action combinations with the highest cumulative rewards;

[0016] B5. Repeat B3-B4 until the optimal action sequence is found;

[0017] B6. Take the action sequence with the highest cumulative reward as the final decision.

[0018] In the above-mentioned HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, when the control action combination predicted according to the state includes a control action combination without action, it is considered that the optimal action sequence has been found;

[0019] Alternatively, when the control action combination with the highest cumulative reward among the control action combinations predicted according to the state is no action, it is considered that the optimal action sequence has been found;

[0020] Alternatively, the optimal sequence of actions is considered to have been found after reaching a predetermined number of decision steps.

[0021] In the above HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, in the reasoning stage, before the last action of the final decision is completed, steps B1-B5 are re-executed to make the final decision of the next stage;

[0022] In the process of executing the decision, the real state information is received in real time. When the difference between the real state information and the predicted state exceeds the set state threshold, the reasoning execution steps B1-B5 are restarted;

[0023] In step B1, the reinforcement learning model is based on the observed state s t Predict one or more control action combinations a with an action probability higher than a set threshold t1、…… a tn .

[0024] In the above-mentioned HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, the historical data is a continuous sa sequence combination, which contains state transition information from state to action and then to a new state, so that the reinforcement learning model can learn the ability to predict actions according to states during training, and at the same time learn the ability to predict new actions according to old states and executed actions;

[0025] In step B3, the reinforcement learning model predicts the new state s after executing the action t+1 .

[0026] In the above-mentioned HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, the state s includes the average indoor temperature, the average indoor humidity, the supply air temperature, the supply air humidity, and the current frequency of the blower;

[0027] The control action a includes the opening of the surface cooler valve, the opening of the heater valve, and the opening of the humidifier valve.

[0028] In the above HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, the reward function includes temperature and humidity penalty, energy consumption penalty and temperature and humidity standard deviation penalty;

[0029] The energy consumption penalty includes: no penalty when the surface cooler valve, heater valve, and humidifier valve are closed, and the energy consumption penalty increases linearly from zero to one hundred valve openings.

[0030] In the above-mentioned HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, the K is 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10.

[0031] In the above-mentioned HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism, in step B4, the K control action combinations with the highest cumulative rewards and the previous control action combination constitute a K action sequence starting from the current state to the current step;

[0032] Training phase: Use the reward function to calculate the reward value corresponding to each row of training data in the SA sequence offline to construct a SAR sequence combination. Use the SAR sequence combination to train the reinforcement learning model using the cross loss function.

[0033] In the above-mentioned HVAC energy conservation and emission reduction method based on reinforcement learning and attention mechanism, K>2, and before step S5, it also includes obtaining the cumulative rewards of K action sequences and sorting them. When the difference between the cumulative rewards of the action sequences ranked later and the action sequences ranked first is greater than the set difference, these action sequences are deleted, and K is set to Kn, where n represents the number of deleted action sequences, until K≤2.

[0034] A HVAC energy-saving and emission-reduction system based on reinforcement learning and attention mechanism performs HVAC energy-saving and emission-reduction control by executing the above method.

[0035] The advantages of the present invention are:

[0036] 1. This solution uses reinforcement learning to control HVAC energy conservation and emission reduction, which can achieve more accurate and detailed HVAC control. In addition, the environmental learning ability of reinforcement learning enables the control system to automatically adjust the control strategy according to real-time feedback to adapt to system changes and external conditions. Uncertainty, thereby continuously improving the system's control ability to fit reality;

[0037] 2. The reward function designed for reinforcement learning in this scheme not only involves energy consumption and temperature and humidity, but also takes into account the standard deviation of temperature and humidity. By controlling the fluctuation range of temperature and humidity as small as possible, the valve can be avoided from opening and closing too much to improve the stability of the system, reduce noise and vibration, and extend the service life of the equipment. In addition, energy waste can be reduced from the perspective of reducing the large amount of energy consumed by the system in a short period of time to reach the set temperature and humidity level;

[0038] At the same time, the energy consumption penalty of the valve opening of the surface cooler, heater, and humidifier is taken into consideration, so that the valve opening can be kept as small as possible and unnecessary valves can be closed to avoid the cold and heat offset caused by the simultaneous opening of the surface cooling / heating valves, resulting in energy waste;

[0039] 3. This solution uses the Transfromer network for reinforcement learning of HVAC energy conservation and emission reduction control. By using the fast learning ability of Transfromer and using 2 years of historical data to train the network, a reinforcement learning model with different weights for different state variables is constructed, so that the model can learn more accurate action predictions, so that in practical applications, the prediction results given will receive higher rewards;

[0040] 4. In the reasoning stage of this scheme, the reward function is used to select the top K control action combinations with the highest rewards in each step, so that the model learns to select the optimal action sequence in a series of states to achieve the goal of maximizing long-term accumulated rewards, and whether to continue to make decisions is determined by whether to continue the action, which is in line with the application scenario of HVAC control;

[0041] 5. In the decision-making process, according to the size of the cumulative reward function value, the action sequence with a smaller reward value is eliminated and the K value is updated to reduce the computational burden of the system; for example, the system can select a larger K value to ensure the correctness of the final choice through more sequence actions, and continuously simplify the sequence through specific conditions to offset the computational burden brought by using a larger K value. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a training process of a HVAC energy-saving and emission-reduction method based on reinforcement learning and attention mechanism in Embodiment 1 of the present invention;

[0043] Figure 2 is a flow chart of the reasoning process in the HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism in the first embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of the reasoning process with K being 3 in the first embodiment of the present invention;

[0045] Figure 4 This is a comparison chart of the December month after this solution was applied to a listed pharmaceutical company in Changzhou, Jiangsu Province, compared with the traditional PID + manual control.

[0046] Figure 5 This is a November year-on-year comparison chart of this solution applied to a listed pharmaceutical company in Changzhou, Jiangsu Province, compared with the traditional PID + manual control;

[0047] Figure 6 This is the first phase of alternating comparison between this solution and traditional PID+manual control after it was applied to a listed pharmaceutical company in Changzhou, Jiangsu.

[0048] Figure 7 This is the second phase of alternating comparison between this solution and traditional PID+manual control after it was applied to a listed pharmaceutical company in Changzhou, Jiangsu;

[0049] Figure 8 Flow chart of the reasoning process in the HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism in Example 2; DETAILED DESCRIPTION

[0050] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] The present solution provides a HVAC energy-saving and emission-reduction method based on reinforcement learning and attention mechanism, and a system for executing the method. The system may include a computer and a memory, the memory is used to store the reinforcement learning model deployed in the system, and the computer is used to run the reinforcement learning model and execute the HVAC energy-saving and emission-reduction method based on reinforcement learning and attention mechanism.

[0052] Embodiment 1

[0053] This embodiment takes the industrial manufacturing workshop as an example. First, the state variables involved in reinforcement learning include collecting sensor data of the environment in the industrial manufacturing workshop, constructing state variables s as input parameters of the model, including the average temperature and average humidity in the workshop, the air supply temperature and humidity, the current frequency of the blower, etc. At the same time, the opening data of each valve in the workshop are collected, and the control variable a is constructed as the output parameter of the model, including the opening of the cooler, heater, and humidifier valves. This embodiment collects a total of 2 years of historical data as a model training data set.

[0054] The control purpose of this scheme is to achieve three goals by controlling the opening of the cooling / heating / humidifying valves: keep the temperature and humidity in the workshop within the process index range, reduce energy consumption, and reduce the standard deviation (reduce frequent fluctuations in temperature and humidity in the workshop). The supply air temperature and humidity represent the dry bulb temperature and relative humidity of the air sent into the workshop from the supply air duct. They are objective observation variables, so they are included in the state variables. The blower current is also an observation variable (one of the model inputs). The air conditioner is turned on and off manually by the workers. When the air conditioner is turned off (the blower current is 0), the model predicts that its valve opening is 0; when the air conditioner is turned on (the blower current is greater than 0), the model will respond quickly according to the current environmental conditions and give it reasonable thresholds for each valve to achieve the three purposes described above.

[0055] The reward function designed in this embodiment takes three aspects into consideration at the same time:

[0056] The degree of temperature and humidity penalty is determined by how far the indoor temperature and humidity measurements deviate from the set values. When the temperature and humidity are close to the set values, no penalty is given. When the temperature and humidity exceed a certain range, the further they deviate from the set values, the greater the penalty:

[0057] r th = -(α·{1·[|t setpoint -t avg |>1]}·|t setpoint -t avg |+β·{1·[|h setpoint -h avg |>1]}·|h setpoint -h avg |)

[0058] α, β are the penalty weights for temperature and humidity, t setpoint ,h setpoint is the required temperature and humidity setting value, t avg ,h avg is the average temperature and humidity in the workshop.

[0059] In this example, if the temperature and humidity deviate from the set value within ±1, there is no penalty. If it exceeds ±1, the greater the deviation, the greater the penalty.

[0060] Energy consumption penalty: No penalty when the cooling / heating / humidifying valve is closed. When the valve opening is from 0 to 100, the energy consumption penalty increases linearly:

[0061] r power =-(θ·val cool +γ·vαl heat +η·val hum )

[0062] val cool ,val heai ,val hum are the valve openings of the industrial air conditioner surface cooler, heater, and humidifier respectively. θ, γ, and η are the penalty weights of the surface cooling / heating / humidifying valve openings respectively.

[0063] Temperature and humidity standard deviation penalty: Statistics are collected over a period of time, such as the standard deviation of temperature and humidity changes within 3 hours (recorded every 5 minutes). If the temperature standard deviation in the past 3 hours is less than or equal to 0.1, no penalty will be imposed; if the humidity standard deviation is less than or equal to 0.5, no penalty will be imposed (indicating that the control in the past 3 hours was very stable). The higher the standard deviation, the greater the penalty:

[0064] r std =-(μ·(100-val t_std )+v·(100-val h_std ))

[0065]

[0066] Among them, val t_std Indicates the temperature standard deviation score (the higher the score, the better, the full score is 100, which means the temperature standard deviation x in the past 3 hours is less than or equal to 0.1, that is, the temperature change is very small and the control is very stable). The higher the score, the smaller the reward temperature standard deviation penalty;

[0067] val h_std Represents the humidity standard deviation score. The higher the score, the smaller the reward humidity standard deviation penalty.

[0068] val t_std The x in the x formula represents the standard deviation of the temperature over the past 3 hours;

[0069] val h_std The x in the formula represents the standard deviation of humidity over the past 3 hours;

[0071] The sum of the three rewards is used as the final reward function value:

[0072] r = r(s, a) = r th +r power+r std

[0073] The ultimate control goal achieved by the model is to strictly limit the temperature and humidity within the process index range and keep the valve opening as small as possible, and close the valves that do not need to be opened as much as possible, so as to avoid the surface cooling / heating valves opening at the same time, resulting in the cancellation of cold and heat and energy waste, while keeping the temperature and humidity fluctuations as small as possible (for example, avoiding large opening and closing of valves).

[0074] A typical industrial air conditioning control unit generally consists of one fresh air air conditioner and multiple purification air conditioners. This solution takes a global perspective, taking into account multiple air conditioners and multi-dimensional data, and considering the energy consumption of all HVACs turned on in the venue, the distance between the temperature and humidity of each room and the set value, and the standard deviation. The final model will predict a set of optimal valve control strategies to maximize the overall reward function.

[0075] Specifically, this scheme attempts to apply the Transfromer network, which has significant effects in the field of natural language processing, to the field of HVAC control and combine it with reinforcement learning for reasoning, such as Figure 1 As shown, the training phase process is as follows:

[0076] Construct a reinforcement learning model based on the Transformer network; the main part of the model is GPT composed of the Transformer network;

[0077] Construct sa sequence combination based on historical data, s represents the state of HVAC environmental feedback, and a represents the control action on HVAC;

[0078] A tokenizer is pre-constructed for historical data, using sa sequence combination and cross loss function for training.

[0079] Through the aforementioned training methods, the model learns to predict actions more accurately, so that in practical applications, the prediction results given will receive higher rewards. That is, by using Transformer as the basic network and using historical data and cross loss functions to train the model, more accurate action predictions are used to obtain more desirable environmental effects, and the value of the reward function is increased by reducing the prediction error, so that the model makes decisions based on several sets of control combinations with high reward function values ​​during the inference stage, thereby indirectly improving the decision-making ability of the model.

[0080] Furthermore, the historical data is a continuous combination of sa sequences, which contains state transition information from state to action and then to the new state, so that the reinforcement learning model can learn the ability to predict actions based on states during training, and at the same time learn the ability to predict new actions based on old states and executed actions, that is, learn the probability of state transition, such as predicting action a1 based on state s1, and then further predicting the new state s2 with the highest probability of transitioning to after executing the action based on action a1.

[0081] The definition of the reward function is pre-set. During the training phase, the corresponding reward value will be calculated based on each collected time point in the historical data sa sequence for the model to learn. In other words, during the training phase, the reward result corresponding to each row of training data is calculated offline using the reward function, that is, the reward value corresponding to the a action is used in the s state to construct the sar sequence, and then the sar sequence is used to train the model. The model recursively predicts the action and reward values ​​through observation, and calculates the cross entropy loss function with the actual action and reward results to implement model training.

[0082] like Figure 2 As shown, the trained reinforcement learning model is used for reasoning and decision making:

[0083] B1. Reinforcement learning model based on the observed state s t Predict one or more control action combinations a with an action probability higher than a set threshold t1、…… a tn The action probability threshold is set by those skilled in the art based on experience and the prediction effect of the trained reinforcement learning model.

[0084] B2. Use the reward function to select the top K highest reward control action combinations a t 1 、…… a t k;

[0085] B3. For each selected control action combination a t i, predict the new state after executing the action;

[0086] B4. Predict a new control action combination for each new state, and select the K control action combinations with the highest cumulative rewards to enter the next round;

[0087] B5. Repeat B3-B4 until the optimal action sequence is found;

[0088] B6. Take the action sequence with the highest cumulative reward as the final decision.

[0089] like Figure 3As shown, the cumulative reward in step B4 refers to the cumulative reward of the action sequence composed of the current step and the previous action combination. The K control action combinations with the highest cumulative rewards and the previous control action combination constitute the K action sequence from the current state to the current step. If this cycle is in the second step, K is 3, and the original state s t Multiple control combinations will be predicted, and the one with the highest reward will be taken t 1 、 a t 2 、 a t 3 Three action combinations, each action combination may contain control objects of the HVAC system such as the surface cooler valve and the humidifier valve. Then for each action combination a t 1 、 a t 2 、 a t 3 Predict the new state s after executing the action t+1 1.s t+1 2,s t+1 3. For each new state s t+1 1.s t+1 2,s t+1 3 Predict new control action combinations, s t+1 1→a t+ 111 、 a t+1 12 、 a t+1 13,s t+1 2→a t+1 twenty one 、 a t+1 twenty two 、 a t+1 23,s t+1 3→a t+1 31 、 a t+1 32 、 a t+1 33. Then, a t+1 The cumulative reward of 11 is r(a t+1 11)+r(a t 1), a t+1 The cumulative reward of 12 is r(a t+1 12)+r(a t 1), a t+1 The cumulative reward of 13 is r(a t+1 13)+r(a t 1), a t+1 The cumulative reward of 21 is r(a t+1 21)+r(a t 2) ... The rest of the action combinations are similar and will not be described in detail. Here, it is assumed that a is taken in the second step.t+1 11.a t+1 12. a t+1 31 three groups, namely Figure 3 The three groups filled with grayscale.

[0090] The cumulative reward in step B6 also refers to the cumulative reward of the action sequence composed of the current step and the previous action. However, the decision is completed when step B6 is executed. The system will select the action sequence with the highest cumulative reward from the K sequences as the final decision. Figure 3 In , assuming that the third step is the last step, the final K sequences of the system are:

[0091] a t 1——a t+1 11——a t+2 111

[0092] a t 2——a t+1 21——a t+2 211

[0093] a t 2——a t+1 21——a t+2 212

[0094] The system will select the action sequence with the highest cumulative reward from these three sequences as the final decision.

[0095] Furthermore, in the reasoning stage, before the last action of the final decision is executed, steps B1-B5 can be re-executed to make the final decision of the next stage while the last action is executed, or at a certain time point before the last action is executed. In other words, a continuous decision is made for one stage each time according to the setting, and then a continuous decision is made for the next stage before the execution of the decision of one stage is completed, so as to perform continuous actions.

[0096] At the same time, this solution preferably receives the real state information in real time during the decision-making process. When the difference between the real state information and the predicted state exceeds the set state threshold, the reasoning execution steps B1-B5 are restarted. Figure 3 The final action sequence selected in is a t 1——a t+1 11——a t+2 111, when action combination a is executed t 1, it is found that the actual state information is different from the predicted s t+1 1 is too large to exceed the set state threshold, then the inference will be restarted with the real s t+1 1 is the original state s tRe-execute B1-B6 to obtain a new action sequence, and the action combinations that were not executed in the original action sequence will be replaced by the new action sequence. The setting of the state threshold is determined by those skilled in the art based on experience and is not limited here. The difference comparison can use any existing technology that can compare the similarity between two groups of actions, such as the cosine similarity algorithm, the Manhattan distance algorithm, etc.

[0097] Specifically, in step B5, when the control action combination predicted according to the state includes a control action combination without action, it is considered that the optimal action sequence has been found.

[0098] Alternatively, when the control action combination with the highest reward among the control action combinations predicted according to the state is no action, it is considered that the optimal action sequence has been found.

[0099] The above two methods start from the current moment and carry out a series of action planning until the optimal action sequence is found when no action control is needed. For example, at the next moment (this is called the first moment), the opening of the air cooler valve is adjusted to 32, and the rest are not acted upon. At the second moment, the opening of the humidifier valve is adjusted to 42, and the rest are not acted upon. At the third moment, there is a control action combination that does not require adjustment of any control object, or a control action combination with the highest cumulative reward that does not require adjustment of any control object. Then the optimal action sequence is determined to be to first adjust the opening of the air cooler valve to 32, and after entering the new state, adjust the opening of the humidifier valve to 42.

[0100] Alternatively, the optimal sequence of actions is considered to have been found after reaching a predetermined number of decision steps.

[0101] The application case below this embodiment uses the second method.

[0102] Figure 4 and Figure 5 This is the result of the on-site deployment and application of this solution in a listed pharmaceutical company in Changzhou, Jiangsu:

[0103] like Figure 4 As shown, the algorithm actually controls the HVAC from December 1 to December 25, 2023. Compared with 2022 (PID + manual control), the year-on-year energy saving rate reaches 43%. The use of chilled water is reduced by 43%, the use of hot steam is reduced by 24%, and the use of humidification steam is reduced by 46%.

[0104] like Figure 5 As shown, the algorithm actually controls the HVAC from November 1 to December 1, 2023. Compared with 2022 (PID + manual control), the year-on-year energy saving rate reaches 35%. Among them, the use of chilled water is reduced by 46%, the use of hot steam is increased by 67%, and the use of humidification steam is reduced by 94%.

[0105] There is a big difference in the amount of hot steam used in the two sets of data. After analysis, it is believed that the reason is:

[0106] 1) The two experiments were conducted for only one month each, which resulted in a certain degree of random deviation;

[0107] 2) The factory production capacity / external environment temperature and humidity will change in different months, resulting in fluctuations in the final results.

[0108] However, both sets of data reflect a relatively high energy saving rate overall.

[0109] In order to further ensure the reliability of the experiment, this embodiment also conducts two phases of alternating experiments on the above deployment. Alternating experiments can reduce the impact of random factors such as external temperature and humidity changes and production capacity fluctuations, and the experimental results will be more reliable and objective.

[0110] The first phase of the experiment is an alternating experiment lasting 32 days (00:00 on March 14, 2024 to 00:00 on April 16, 2024), in which the 1st, 3rd, ..., 31st days are controlled by PID, and the 2nd, 4th, ..., 32nd day are controlled by the artificial intelligence algorithm proposed in this scheme. Figure 6 shown.

[0111] The second phase of the experiment is an alternating experiment lasting 70 days (00:00 on May 6, 2024 to 00:00 on July 16, 2024), in which the 1st, 3rd, ..., 69th day are controlled by PID, and the 2nd, 4th, ..., 70th day are controlled by artificial intelligence algorithm. Figure 7 shown.

[0112] Figure 6 and Figure 7 In the figure, Price-total represents the total energy bill. Under the premise that both the model and PID achieve stable control, the model saves 29.62% / 29.48% of the overall energy consumption compared with PID.

[0113] The total energy consumption is composed of the energy consumption of cooling, heating and humidification valves (including electricity, steam and chilled water), corresponding to total_cold, total_heat and total_humidity respectively. Looking at the energy consumption of the humidification valve alone, the model consumes 55.37% / 78.26% more energy than PID respectively. However, from the perspective of cooling / heating valves, the energy bills of these two items are higher and the model is more energy-efficient than PID, and the total energy consumption is still lower than the model. It can be seen that compared with the traditional PID+manual control solution, this solution has outstanding advantages in energy saving. Theoretically, this solution also has more advantages, such as more stable and precise control capabilities, which can reduce the waste caused by cold and hot offset, and reduce human labor.

[0114] For intervention, it can realize rapid response of single-point status and real-time control of multiple working conditions in minutes.

[0115] Automatic control, etc.

[0116] Embodiment 2

[0117] This embodiment is similar to the first embodiment, except that in this embodiment, K>2 is preferably selected, such as 3, 4, 5, 6, 7, 8, 9, 10, etc. And, Figure 8 As shown, before step S5, it also includes that after obtaining the K control action combinations with the highest cumulative rewards in each step, a cumulative reward sorting is performed. When the difference between the action sequences ranked later and the action sequences ranked first is greater than the set difference, these action sequences are deleted, and K is set to Kn, n represents the number of action sequences deleted, until K≤2, n can be equal to 0. If after one deletion, the action sequence is left with only the first one, the second action sequence is retained, that is, it is guaranteed that the final action sequence has at least two. The set difference is set by those skilled in the art based on experience, and it is necessary to balance the amount of calculation and the decision accuracy. Generally, 10%-20% of the cumulative reward of the first action sequence is taken. Through this scheme, according to the size of the cumulative reward function value, the action sequence with a smaller reward value is eliminated and the K value is updated, which can reduce the system calculation burden. The system can select a larger K value to ensure the correctness of the final selection through more sequence actions, and continuously simplify the sequence through specific conditions to offset the calculation burden brought by using a larger K value, thereby greatly improving the decision accuracy while ensuring that the amount of calculation is not significantly increased.

[0118] The specific embodiments described in this embodiment are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.

Claims

1. A method for HVAC energy saving and emission reduction based on reinforcement learning and attention mechanism, It is characterized in that include: Training phase Build a reinforcement learning model based on the Transformer network; Construct sa sequence combination based on historical data, s represents the state of HVAC environmental feedback, and a represents the control action on HVAC; Use sa sequence combination and cross loss function to train the reinforcement learning model; Reasoning Phase B1. The trained reinforcement learning model predicts multiple control combinations based on the current state; B2. Use the reward function to select the top K highest reward control action combinations; B3. For each selected control action combination, predict the new state after executing the action; B4. Predict a new control action combination for each new state and select the K control action combinations with the highest cumulative rewards; B5. Repeat B3-B4 until the optimal action sequence is found; B6. Take the action sequence with the highest cumulative reward as the final decision.

2. According to the HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 1, It is characterized in that When the control action combination predicted according to the state includes a control action combination without action, it is considered that the optimal action sequence has been found; Alternatively, when the control action combination with the highest cumulative reward among the control action combinations predicted according to the state is no action, it is considered that the optimal action sequence has been found; Alternatively, the optimal sequence of actions is considered to have been found after reaching a predetermined number of decision steps.

3. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 2, It is characterized in that In the reasoning stage, before the last action of the final decision is completed, steps B1-B5 are re-executed to make the final decision of the next stage; In the process of executing the decision, the real state information is received in real time. When the difference between the real state information and the predicted state exceeds the set state threshold, the reasoning execution steps B1-B5 are restarted; In step B1, the reinforcement learning model is based on the observed state s t Predict one or more control action combinations a with an action probability higher than a set threshold t1、…… a tn .

4. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 1, It is characterized in that The historical data is a continuous combination of sa sequences, which contains state transition information from state to action and then to a new state, so that the reinforcement learning model can learn the ability to predict actions based on states during training, and learn the ability to predict new actions based on old states and executed actions; In step B3, the reinforcement learning model predicts the new state s after executing the action t+1 .

5. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 1, It is characterized in that The state s includes the average indoor temperature, the average indoor humidity, the supply air temperature, the supply air humidity, and the current frequency of the blower; The control action a includes the opening of the surface cooler valve, the opening of the heater valve, and the opening of the humidifier valve.

6. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 5, It is characterized in that The reward function includes temperature and humidity penalty, energy consumption penalty, and temperature and humidity standard deviation penalty; The energy consumption penalty includes: no penalty when the surface cooler valve, heater valve, and humidifier valve are closed, and the energy consumption penalty increases linearly from zero to one hundred valve openings.

7. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 1, It is characterized in that The K is 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10.

8. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to any one of claims 1 to 7, It is characterized in that In step B4, the K control action combinations with the highest cumulative rewards and the previous control action combination constitute a sequence of K actions from the current state to the current step; Training phase: Use the reward function to calculate the reward value corresponding to each row of training data in the SA sequence offline to construct a SAR sequence combination. Use the SAR sequence combination to train the reinforcement learning model using the cross loss function.

9. The HVAC energy saving and emission reduction method based on reinforcement learning and attention mechanism according to claim 8, It is characterized in that K>2, and before step S5, the process also includes obtaining the cumulative rewards of K action sequences and sorting them. When the difference between the cumulative rewards of the action sequences ranked later and the action sequences ranked first is greater than the set difference, these action sequences are deleted, and K is set to Kn, where n represents the number of deleted action sequences, until K≤2.

10. A HVAC energy saving and emission reduction system based on reinforcement learning and attention mechanism, It is characterized in that Energy saving and emission reduction control of HVAC is performed by executing any one of the methods of claims 1-9.

Citation Information

Cited By

  • Safety management system and method for intelligent money box

    CN119940939A

  • CIP cleaning system control method, device, equipment, medium and program product

    CN121008469A

  • Human indoor behavior intention reasoning method and system based on deep inverse reinforcement learning, electronic equipment and storage medium

    CN122020424A