Heating ventilation air conditioner energy-saving optimization control method and system based on offline reinforcement learning

Abstract

The invention relates to a heating ventilation air conditioner energy-saving optimization control method and system based on offline reinforcement learning, and the method comprises the following steps: carrying out the modeling of a heating ventilation air conditioner system through a neural network, and enabling the modeling to serve as a simulator, predicting control target data of a next moment by acquiring environmental data of a previous moment, environmental data of a current moment, control target data and valve opening data; simulation data of the heating ventilation air-conditioning system are generated through the simulator, and a training set is constructed; establishing a reinforcement learning model by taking the minimum valve opening and the minimum valve opening variation at adjacent moments on the premise that the control target meets the requirement as an optimization target; using the training set to train the reinforcement learning model offline; and control target data and environment data of the heating ventilation air conditioning system at the previous moment and the current moment are obtained in real time, and the trained reinforcement learning model is used for outputting an optimal valve opening control scheme. Compared with the prior art, the method has the advantages that the generated control scheme is low in energy consumption, good in robustness and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of HVAC control, and in particular to a HVAC energy-saving optimization control method and system based on offline reinforcement learning. Background Art

[0002] There are many control methods in the heating, ventilation, and air conditioning (HVAC) system. The most widely used traditional control methods are proportional-integral-derivative (PID) and model predictive control (MPC).

[0003] PID controller is simple to implement, easy to understand and debug; it performs well in many simple control tasks, such as temperature control, speed control, etc.; it has a fast response speed and is suitable for applications that require high real-time performance. However, for complex nonlinear systems or systems with large changes, PID controller may not provide good control performance; it requires manual parameter adjustment, is not sensitive to system changes, and requires experienced engineers to adjust.

[0004] MPC can handle complex nonlinear systems and works better for systems with multiple inputs and outputs. It can consider the impact of the controller on the future system state and optimize the control process. It has strong robustness and adaptability and responds well to system changes and disturbances. However, its debugging is relatively complex. It is very difficult to establish an accurate and easy-to-control thermodynamic model for a more complex industrial scenario, and its versatility is very limited.

[0005] In recent years, more and more researchers have begun to use reinforcement learning, deep Q networks, etc. for research. Although the above content has made some progress, there are also shortcomings. First, in the process of using reinforcement learning, the environment is not modeled. Only real data and rules in the algorithm are used to approach the real environment, and existing data are not used to increase data diversity; second, when determining the state space, the first-order Markov process is strictly followed, and the control process often has delays; third, the control hardware is a mechanical structure, and the operation changes between adjacent controls cannot be too large. Current research only considers energy-saving related factors in the design of reward functions, and does not consider the actual situation of mechanical hardware; fourth, offline reinforcement learning requires a large amount of data. Under normal circumstances, the amount of factory data is difficult to meet training needs. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a HVAC energy-saving optimization control method and system based on offline reinforcement learning. By controlling the valve opening of the recooling coil and the reheating coil of the fresh air unit (Make-up Air Unit, MAU) in the HVAC system, while ensuring that the outlet air temperature and humidity meet the requirements, the valve opening is reduced as much as possible and the valve opening changes smoothly, thereby reducing the water flow and reducing the energy consumption of the refrigerator.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, a heating, ventilation and air conditioning energy-saving optimization control method based on offline reinforcement learning is provided, the method comprising the following steps:

[0009] S1, using neural network to model the HVAC system as a simulator, and predicting the control target data at the next moment by obtaining the environmental data at the previous moment and the environmental data, control target data and valve opening data at the current moment;

[0010] S2, generates simulation data of HVAC system through simulator to build training set;

[0011] S3, taking the minimum valve opening and the minimum change of valve opening at adjacent moments as the optimization goal under the premise of achieving the control target, a reinforcement learning model is established;

[0012] S4, offline training of the reinforcement learning model using the training set;

[0013] S5, real-time acquisition of the control target data and environmental data of the HVAC system at the previous moment and the current moment, and output of the optimal valve opening control scheme using the trained reinforcement learning model.

[0014] As an optimal technical solution, the control target data includes outlet air temperature, humidity and dew point temperature, and the environmental data includes recooling coil water supply temperature and pressure, reheating coil water supply temperature and pressure, number of humidification water pumps, humidification water temperature and supply air temperature.

[0015] As a preferred technical solution, the valve opening data is subjected to white noise processing before being input into the neural network for prediction.

[0016] As a preferred technical solution, there are multiple neural network models, and the control target data at the current moment are predicted respectively by multiple neural network models, and the median of all predicted control target data is used as the final prediction result.

[0017] As a preferred technical solution, considering the delay behavior in the control process, the environmental state of the reinforcement learning model is set to the control target data and environmental data at the previous moment and the current moment.

[0018] As a preferred technical solution, the action of the reinforcement learning model is set to the valve opening, including the recooling valve opening and the reheating valve opening of the fresh air unit.

[0019] As a preferred technical solution, the reward setting of the reinforcement learning model is:

[0020] R=c1*R SF +c2*R O +c3*R DIF_O

[0021] In the above formula, R is the total reward, R SF is the reward of the fresh air unit's air supply temperature and humidity, R O is the valve opening reward, R O_DIF is the reward for the change in valve opening, c1, c2, c3 are the weight coefficients of each reward; where,

[0022]

[0023] Among them, SFT real 、SFT standard They are the actual value and standard value of supply air temperature, SFH real , SFH standard They are the actual value and standard value of humidity, SFLD real , SFLD standard are the actual value and standard value of dew point temperature, r1, r2, r3 are importance adjustment coefficients;

[0024]

[0025] CWO max 、CWO real are the maximum and actual opening values ​​of the recooling valve, HWO max 、HWO real are the maximum and actual values ​​of the reheat valve opening, respectively, and a1 and a2 are importance adjustment coefficients;

[0026]

[0027] DIFF_CWO max , DIFF_CWO are the maximum and actual values ​​of the recooling valve opening change, DIFF_HWO max , DIFF_HWO are the maximum and actual values ​​of the hot and cold valve opening changes respectively, b1 and b2 are the importance adjustment coefficients.

[0028] As a preferred technical solution, the offline training of the reinforcement learning model using the training set specifically includes the following steps:

[0029] S401, randomly selecting a time point from the historical moments as the current moment, and obtaining the input data required by the neural network and the environment state and action data of the reinforcement learning model corresponding to the time point from the historical data of the training set;

[0030] S402, inputting the acquired input data required by the neural network corresponding to the time point into multiple simulators respectively, taking the average of the outputs of the multiple simulators to obtain the prediction control target data at the next moment, and calculating the variance of the outputs of the multiple simulators, combining the prediction control target data at the next moment and the historical data in the training set, to obtain the environmental state at the next moment;

[0031] S403, calculating the reward of this time step using the action and the environment state at the next moment;

[0032] S404, defining a gap variable, which is a Boolean variable, and taking a value of true or false; determining whether the variance is greater than a preset threshold, if so, assigning the gap variable a value of true, otherwise, assigning the gap variable a value of false;

[0033] S405, storing the current environment state, action, reward, next environment state and gap variable as experience data in the experience pool;

[0034] S406, randomly sampling a batch of experience data from the experience pool as training samples, wherein the training samples include data of multiple time steps;

[0035] S407, updating the parameters of the reinforcement learning model using the training samples;

[0036] S408, continuously repeating steps S401-S407 until the model converges or reaches a preset number of training times.

[0037] As a preferred technical solution, in step S405, the first-order Markov process and the second-order Markov process are mixed according to a preset ratio to obtain experience data stored in the experience pool, wherein the experience data corresponding to the first-order Markov process is obtained by executing steps S401-S404 once, and the experience data corresponding to the second-order Markov process is obtained by executing steps S401-S404 once, and then executing steps S402-S404 again based on the obtained environmental state at the next moment and the action at the next moment to obtain data at the next moment as the experience data.

[0038] According to a second aspect of the present invention, a HVAC energy-saving optimization control system based on offline reinforcement learning is provided, which is used to implement the above method, and the system comprises:

[0039] Simulation module: Use neural network to model the HVAC system as a simulator, and predict the control target data at the next moment by obtaining the environmental data at the previous moment and the environmental data, control target data and valve opening data at the current moment;

[0040] Training set construction module: Generates simulation data of HVAC system through simulator to construct training set;

[0041] Reinforcement learning module construction and training module: Establish a reinforcement learning model with the optimization goal of minimizing the valve opening and minimizing the change in valve opening at adjacent moments under the premise that the control target is met; use the training set to train the reinforcement learning model offline;

[0042] Optimization control module: obtains the control target data and environmental data of the HVAC system at the previous and current moments in real time, and uses the trained reinforcement learning model to output the optimal valve opening control plan.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention uses an integrated multiple neural network model to replace the traditional HVAC thermodynamic model, which greatly reduces the complexity and labor cost. At the same time, it uses an integrated model and takes sensor noise into consideration to improve the model robustness.

[0045] (2) By combing the factory HVAC air circulation process, the present invention adopts a mixed method of first-order and second-order Markov processes, making the experience pool data of offline reinforcement learning richer and more realistic, and ultimately significantly improving the learning effect, and the resulting control scheme is more reasonable and effective.

[0046] (3) In the reward design of the offline reinforcement learning model, the present invention fully combines the energy-saving needs of the factory and the actual hardware control principle. First, the valve opening is minimized while ensuring the factory production requirements. Second, the change in valve opening in adjacent time periods is strictly controlled to avoid hardware damage. The optimal control solution finally obtained can not only meet the energy-saving needs well, but also avoid hardware damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 The figure is a comparison curve diagram of the total valve opening using the method of the present invention and the PID control method in an embodiment. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] This embodiment provides a HVAC energy-saving optimization control method based on offline reinforcement learning. Figure 1 As shown, the method comprises the following steps:

[0053] S1, using neural network to model the HVAC system as a simulator, predicting the control target data at the next moment by obtaining the environmental data at the previous moment and the environmental data, control target data and valve opening data at the current moment.

[0054] In reinforcement learning, a large number of samples are needed to enable the agent to learn a better strategy, but in general, it is difficult to obtain the amount of data that can meet the requirements of reinforcement learning training. Therefore, many studies currently obtain data through modeling and simulation, but it is very difficult to construct an accurate thermodynamic model of HVAC in a factory through traditional methods. In order to solve this problem, this embodiment constructs a HVAC model of the factory through a neural network.

[0055] During this process, the following questions were considered:

[0056] 1. Current reinforcement learning is often limited to the standard Markov process, that is, time t is only affected by time t-1, but in the HVAC system, air circulates and the impact has a time delay problem;

[0057] 2. The historical data of valve opening are all obtained through sensors, and valves are mechanical components. The environmental conditions in factories are generally poor. As the service life of sensors increases, the accuracy will decrease, and the accuracy of sensors and mechanical components needs to be verified. Considering the above factors, although the historical data of valve opening is within a reasonable range, there may be noise;

[0058] 3. The prediction robustness based on a single neural network model is poor.

[0059] To solve the above problems, the neural network modeling process is as follows:

[0060] S11, determine the neural network input and output

[0061] The HVAC system is modeled using a neural network as a simulator, which uses the environmental data at time t-1 and the environmental data at time t, the control target data and the valve opening data to predict the control target data at time t+1.

[0062] In this embodiment, the control target data includes the outlet air temperature, humidity and dew point temperature, and the environmental data includes the recooling coil water supply temperature and pressure, the reheating coil water supply temperature and pressure, the number of humidification water pumps, the humidification water temperature and the supply air temperature. That is, the input and output of the neural network can be expressed as:

[0063] enter:

[0064] 1) Environmental data at time t-1: CWTO, CWPO, HWTO, HWPO, JSO, JSTO, TsensorO;

[0065] 2) Environmental data and control target data at time t: CWT, CWP, HWT, HWP, JS, JST, Tsensor, SFT, SFH, SFLD;

[0066] 3) Valve opening at time t: CWO, HWO;

[0067] Output:

[0068] The control target data at time t+1 includes supply air temperature, humidity, and dew point temperature.

[0069] Among them, SFT, SFH, SFLD, CWT, CWP, HWT, HWP, JS, JST, and Tsensor are the outlet air temperature, humidity, dew point temperature, re-cooling coil water supply temperature, re-cooling coil water supply pressure, reheat coil water supply temperature, reheat coil water supply pressure, number of humidification water pumps, humidification water temperature, and supply air temperature at time t, respectively. Similarly, SFTO, SFHO, SFLDO, CWTO, CWPO, HWTO, HWPO, JSO, JSTO, and TsensorO are the outlet air temperature, humidity, dew point temperature, re-cooling coil water supply temperature, re-cooling coil water supply pressure, reheat coil water supply temperature, reheat coil water supply pressure, number of humidification water pumps, humidification water temperature, and supply air temperature at time t-1, respectively.

[0070] The overall energy-saving principle is to control the opening of the valve to ultimately make the MAU air supply temperature and humidity meet the requirements. In order to eliminate the errors of the mechanical components and sensors that control the valve opening, white noise is added to CWO and HWO when inputting the neural network to simulate real noise.

[0071] Based on the experience of traditional machine learning, the results of the integrated learning model are generally better than the prediction results of a single model. Therefore, this embodiment uses 4-16 neural networks to construct the HVAC model. Finally, the median output of each model is taken as the final prediction result.

[0072] S2, generates simulated data of the HVAC system through the simulator and constructs a training set.

[0073] S3, with the minimum valve opening and the minimum change of valve opening at adjacent moments as the optimization goal under the premise of achieving the control target, a reinforcement learning model is established.

[0074] While ensuring that the air supply temperature and humidity of the MAU in the HVAC system are within the range of production requirements, the factory HVAC energy consumption minimization is modeled as a Markov process, and the corresponding environmental state, action and reward functions are designed.

[0075] S31, set the environment status

[0076] Taking into account the delay behavior in the control process, the environmental state of the reinforcement learning model is set to the control target data and environmental data of the previous moment and the current moment, that is, state = [SFT, SFH, SFLD, CWT, CWP, HWT, HWP, JS, JST, Tsensor, SFTO, SFHO, SFLDO, C WTO, CWPO, HWTO, HWPO, JSO, JSTO, TsensorO].

[0077] S32, set action

[0078] The action of the reinforcement learning model is set as the valve opening, including the recooling valve opening and reheating valve opening of the fresh air unit, that is, a = [CWO, HWO], where CWO is the recooling valve opening in the MAU, and HWO is the reheating valve opening.

[0079] S33, Set rewards

[0080] According to this technical solution, the ultimate goal is to ensure that the valve opening is as small as possible while ensuring that the MAU air supply temperature and humidity meet the requirements, and the adjacent valve opening setting values ​​and the air outlet temperature and humidity are as stable as possible. Therefore, the reward expression is set as follows:

[0081] R=c1*R SF +c2*R O +c3*R DIF_O

[0082] In the above formula, R is the total reward, R SF is the reward of the fresh air unit's air supply temperature and humidity, R O is the valve opening reward, R O_DIF is the reward for the change in valve opening, and c1, c2, c3 are the weight coefficients of each reward.

[0083] R SF The difference between the supply air temperature, humidity and dew point temperature and the standard value is mainly considered. The reward range of each single item is scaled to [0,1]. The importance of each single item is adjusted by r1, r2, and r3. The expression is:

[0084]

[0085] Among them, SFT real 、SFT standard They are the actual value and standard value of supply air temperature, SFH real , SFH standard They are the actual value and standard value of humidity, SFLD real , SFLD standard are the actual value and standard value of dew point temperature respectively, r1, r2, r3 are importance adjustment coefficients.

[0086] R O The main consideration is the difference between the opening of the recooling and reheating valves and the maximum opening. The larger the actual valve opening, the lower the reward. The reward range of each single item is scaled to [0,1]. The importance of each single item is adjusted by a1, a2, and its expression is:

[0087]

[0088] Among them, CWO max 、CWO real are the maximum and actual opening values ​​of the recooling valve, HWO max 、HWO real are the maximum and actual values ​​of the reheat valve opening respectively, and a1 and a2 are the importance adjustment coefficients.

[0089] R DIF_O The main consideration is the change in the opening of the recooling and reheating valves. The greater the change, the smaller the reward. max and DIFF_CWO max is the maximum value of the allowed valve change, the reward range of each single item is scaled to [0,1], and the importance of each single item is adjusted by b1, b2, and its expression is:

[0090]

[0091] Among them, DIFF_CWO is the actual value of the change in the opening of the re-cooling valve, DIFF_HWO is the actual value of the change in the opening of the hot and cold valves, and b1 and b2 are importance adjustment coefficients.

[0092] S4, use the training set to train the reinforcement learning model offline.

[0093] The specific steps include:

[0094] S401, randomly select a time point from the historical moments as the current moment t, and obtain the input data input required by the neural network corresponding to the time point t from the historical data of the training set, as well as the environmental state state and corresponding action action of the reinforcement learning model.

[0095] S402, input the input data input required by the neural network corresponding to the acquired time point into multiple simulators respectively, take the average of the outputs of multiple simulators to obtain the predicted control target data output at the next moment, output is the supply air temperature, humidity, and dew point temperature at time t+1, and calculate the variance output_std of the outputs of multiple simulators, combine the predicted control target data output at the next moment and the historical data in the training set, and obtain the environmental state next_state at the next moment.

[0096] S403, using the action action and the environment state next_state at the next moment to calculate the reward reward of this time step.

[0097] S404, define the gap variable halt, which is a Boolean variable with a value of true or false; determine whether the variance output_std is greater than the preset threshold threshold. If so, it means that the effect difference between the sample point in different simulators is large, and the gap variable halt is assigned to true, that is, halt = True; if the variance output_std is less than the preset threshold threshold, it means that the effect difference between the sample point in different simulators is small, and the gap variable halt is assigned to false, that is, halt = False.

[0098] S405, storing the current environment state, action, reward, next environment state and gap variable as experience data into the experience pool.

[0099] In offline reinforcement learning, an experience pool (Experience Replay Buffer) is required to store historical experience. At present, since reinforcement learning uses a first-order Markov process, the experience pool will store the current state, action, reward, next_state, and halt parameters obtained by executing steps S401-S404 once in the experience pool. And the second-order Markov process is considered to be optimal. In this embodiment, the first-order and second-order Markov processes are mixed in a certain proportion. After the second-order Markov process obtains the state, action, reward, next_state, and halt parameters in the first-order Markov process, steps S402-S404 are executed again, and finally next_state, next_action, next_reward, next_state_2, and halt_2 parameters are obtained. In short, the first-order Markov process uses the state and action at time t to obtain the state at time t+1, and the second-order Markov process uses the results obtained in the first order, and uses the state and action at time t+1 to obtain the state at time t+2.

[0100] S406, randomly sampling a batch of experience data from the experience pool as training samples, wherein the training samples include data of multiple time steps.

[0101] S407, using the training samples to update the parameters of the reinforcement learning model, this process uses the actor-critic model.

[0102] S408, continuously repeating steps S401-S407 until the model converges or reaches a preset number of training times.

[0103] After training, the model will learn the optimal action strategy to be taken under given environmental conditions and the corresponding reward function.

[0104] S5, real-time acquisition of the control target data and environmental data of the HVAC system at the previous moment and the current moment, and output of the optimal valve opening control scheme using the trained reinforcement learning model.

[0105] After the model training is completed, the model can be used to make online decisions, that is, the real-time status of the system is fed back through the database or PLC, and input into the offline reinforcement learning model to select the optimal action according to the current environmental status to achieve the optimization goal of the system. At the same time, the experience pool can continue to be used to further optimize the performance and stability of the model.

[0106] Through actual deployment in the factory, we can get Figure 2 The results shown are under the AI ​​control of the offline reinforcement learning method of the present invention (corresponding to Figure 2 solid line), compared with the original PID control (corresponding Figure 2 The dotted line shows that the total opening of the two valves is lower, resulting in a lower water flow and lower chiller energy consumption.

[0107] The above is an introduction to the method embodiment. The following is a further explanation of the solution of the present invention through a system embodiment.

[0108] A heating, ventilation and air conditioning energy-saving optimization control system based on offline reinforcement learning, used to implement the above method, the system comprising:

[0109] Simulation module: Use neural network to model the HVAC system as a simulator, and predict the control target data at the next moment by obtaining the environmental data at the previous moment and the environmental data, control target data and valve opening data at the current moment;

[0110] Training set construction module: Generates simulation data of HVAC system through simulator to construct training set;

[0111] Reinforcement learning module construction and training module: Establish a reinforcement learning model with the optimization goal of minimizing the valve opening and minimizing the change in valve opening at adjacent moments under the premise that the control target is met; use the training set to train the reinforcement learning model offline;

[0112] Optimization control module: obtains the control target data and environmental data of the HVAC system at the previous and current moments in real time, and uses the trained reinforcement learning model to output the optimal valve opening control plan.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A HVAC energy-saving optimization control method based on offline reinforcement learning, characterized in that: The method comprises the following steps: S1, using neural network to model the HVAC system as a simulator, and predicting the control target data at the next moment by obtaining the environmental data at the previous moment and the environmental data, control target data and valve opening data at the current moment; S2, generates simulation data of HVAC system through simulator to build training set; S3, taking the minimum valve opening and the minimum change of valve opening at adjacent moments as the optimization goal under the premise of achieving the control target, a reinforcement learning model is established; S4, offline training of the reinforcement learning model using the training set; S5, real-time acquisition of the control target data and environmental data of the HVAC system at the previous moment and the current moment, and output of the optimal valve opening control scheme using the trained reinforcement learning model.

2. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: The control target data include outlet air temperature, humidity and dew point temperature, and the environmental data include recooling coil water supply temperature and pressure, reheating coil water supply temperature and pressure, number of humidification water pumps, humidification water temperature and supply air temperature.

3. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: The valve opening data is processed by adding white noise before being input into the neural network for prediction.

4. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: There are multiple neural network models, and the control target data at the current moment are predicted respectively by the multiple neural network models, and the median of all predicted control target data is taken as the final prediction result.

5. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: Taking into account the delay behavior in the control process, the environmental state of the reinforcement learning model is set to the control target data and environmental data of the previous moment and the current moment.

6. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: The action of the reinforcement learning model is set as the valve opening, including the recooling valve opening and the reheating valve opening of the fresh air unit.

7. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: The reward setting of the reinforcement learning model is: R=c1*R SF +c2*R O +c3*R DIF_O In the above formula, R is the total reward, R SF is the reward of the fresh air unit's air supply temperature and humidity, R O is the valve opening reward, R O_DIF is the reward for the change in valve opening, c1, c2, c3 are the weight coefficients of each reward; where, Among them, SFT real 、SFT standard They are the actual value and standard value of supply air temperature, SFH real , SFH standard They are the actual value and standard value of humidity, SFLD real , SFLD standard are the actual value and standard value of dew point temperature, r1, r2, r3 are importance adjustment coefficients; CWO max 、CWO real are the maximum and actual opening values ​​of the recooling valve, HWO max 、HWO real are the maximum and actual values ​​of the reheat valve opening, respectively; a1 and a2 are importance adjustment coefficients; DIFF_CWO max , DIFF_CWO are the maximum and actual values ​​of the recooling valve opening change, DIFF_HWO max , DIFF_HWO are the maximum and actual values ​​of the hot and cold valve opening changes respectively, b1 and b2 are the importance adjustment coefficients.

8. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 1 is characterized in that: The offline training of the reinforcement learning model using the training set specifically includes the following steps: S401, randomly selecting a time point from the historical moments as the current moment, and obtaining the input data required by the neural network and the environment state and action data of the reinforcement learning model corresponding to the time point from the historical data of the training set; S402, inputting the acquired input data required by the neural network corresponding to the time point into multiple simulators respectively, taking the average of the outputs of the multiple simulators to obtain the prediction control target data at the next moment, and calculating the variance of the outputs of the multiple simulators, combining the prediction control target data at the next moment and the historical data in the training set, to obtain the environmental state at the next moment; S403, calculating the reward of this time step using the action and the environment state at the next moment; S404, defining a gap variable, which is a Boolean variable, and taking a value of true or false; determining whether the variance is greater than a preset threshold, if so, assigning the gap variable a value of true, otherwise, assigning the gap variable a value of false; S405, storing the current environment state, action, reward, next environment state and gap variable as experience data in the experience pool; S406, randomly sampling a batch of experience data from the experience pool as training samples, wherein the training samples include data of multiple time steps; S407, updating the parameters of the reinforcement learning model using the training samples; S408, continuously repeating steps S401-S407 until the model converges or reaches a preset number of training times.

9. The HVAC energy-saving optimization control method based on offline reinforcement learning according to claim 8, characterized in that: In the step S405, the first-order Markov process and the second-order Markov process are mixed according to a preset ratio to obtain experience data stored in the experience pool, wherein the experience data corresponding to the first-order Markov process is obtained by executing steps S401-S404 once, and the experience data corresponding to the second-order Markov process is obtained by executing steps S401-S404 once, and then executing steps S402-S404 again based on the obtained environmental state at the next moment and the action at the next moment to obtain data at the next moment as the experience data.

10. A HVAC energy-saving optimization control system based on offline reinforcement learning, characterized in that: For implementing the method according to any one of claims 1 to 9, the system comprises: Simulation module: Use neural network to model the HVAC system as a simulator, and predict the control target data at the next moment by obtaining the environmental data at the previous moment and the environmental data, control target data and valve opening data at the current moment; Training set construction module: Generates simulation data of HVAC system through simulator to construct training set; Reinforcement learning module construction and training module: Establish a reinforcement learning model with the optimization goal of minimizing the valve opening and minimizing the change in valve opening at adjacent moments under the premise that the control target is met; use the training set to train the reinforcement learning model offline; Optimization control module: obtains the control target data and environmental data of the HVAC system at the previous and current moments in real time, and uses the trained reinforcement learning model to output the optimal valve opening control plan.

Images