Machine learning control strategy optimization method for all fresh air air conditioning system

Abstract

The invention relates to the technical field of heating ventilation air-conditioning system control, and discloses a full fresh air air-conditioning system machine learning control strategy optimization method, which comprises the following steps of: establishing a state space, an action space and a reward function suitable for an air-conditioning system by analyzing elements influencing indoor temperature and humidity of a high-grade biosafety laboratory; and then a deep reinforcement learning method is used for optimizing a control strategy of the high-grade biosafety laboratory all fresh air air conditioning system. According to the invention, strict indoor temperature and humidity control of the high-grade biosafety laboratory can be realized, and meanwhile, certain energy consumption can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of HVAC system control, and in particular to a method for optimizing a machine learning control strategy for a fresh air conditioning system. Background Art

[0002] As the country's emphasis on energy conservation and carbon reduction in the building sector continues to increase, the air conditioning systems of some specialized buildings, such as high-level biosafety laboratories, have high ventilation rates, utilize fresh air, and rely on low-temperature chilled water for supercooling, dehumidification, and reheating. This leads to fresh air loads exceeding five to six times those of ordinary buildings, which contradicts the broader trend of energy conservation and emission reduction. Consequently, various energy-saving measures, such as fresh air dehumidifiers, are emerging. Currently, many fresh air air conditioning systems widely utilize heat recovery technology. By deploying multiple sets of heat recovery heat exchangers and circulating water pumps in different locations, heat from fresh and exhaust air is continuously recovered to the fresh air supply for pre-cooling, preheating, or reheating, reducing the fresh air load. However, the addition of multiple heat recovery heat exchangers makes the control problem associated with the combination of the heat recovery system and the existing fresh air system more complex and nonlinear. Control of the heat recovery system and the fresh air supply is not optimized for energy consumption, failing to fully tap the system's energy-saving potential. Therefore, an air conditioning system control strategy optimization method is needed to fully tap the energy-saving potential and dynamically control the medium flow input of the heat recovery system and the fresh air system according to the outdoor climate conditions and indoor heat and humidity requirements of the building. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for optimizing the machine learning control strategy of a fresh air air conditioning system. The present invention considers the state of the heat recovery system, the indoor heat source and humidity source of the high-level biosafety laboratory as one of the state quantities, and the controllable factors affecting the indoor temperature and humidity of the high-level biosafety laboratory as the control action quantity. The reward function that balances the total energy consumption of the air conditioning system equipment and the indoor temperature and humidity requirements is a Gaussian function. The deep reinforcement learning algorithm DQN is used to construct a reinforcement learning model suitable for optimizing the control strategy of the fresh air air conditioning system of the high-level biosafety laboratory. While achieving strict indoor temperature and humidity control of the high-level biosafety laboratory, the energy consumption of the air conditioning system is reduced, and the indoor temperature and humidity control and energy saving problems of the fresh air air conditioning system of the high-level biosafety laboratory after adding a multi-stage circulating heat recovery system under the strict temperature and humidity control of the high-level biosafety laboratory are solved.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: A machine learning control strategy optimization method for a fresh air air conditioning system includes the following steps: Establish a reinforcement learning state space based on the outdoor meteorological parameters of the fresh air air conditioning system, the circulating heat recovery system parameters, and the indoor heat source and indoor humidity source parameters; Establish a reinforcement learning action space based on the fresh air temperature and humidity processing parameters of the fresh air air conditioning system for heat recovery, cooling and dehumidification, reheating and humidification; A Gaussian function is used to establish a reinforcement learning reward function that balances the indoor environment temperature and humidity targets and the air conditioning system energy consumption targets; Based on the reinforcement learning state space, reinforcement learning action space and reinforcement learning reward function, a deep reinforcement learning method is used to interact with the fresh air air conditioning system environment, and the reinforcement learning agent is trained to control the fresh air air conditioning system.

[0005] Furthermore, the outdoor meteorological parameters include outdoor temperature and humidity; The circulating heat recovery system parameters include the heat recovery pre-cooling heat exchanger inlet temperature and the exhaust heat recovery heat exchanger outlet temperature.

[0006] Furthermore, the fresh air temperature and humidity processing parameters include the heat recovery system circulation pump flow, the heat recovery system three-way valve diversion ratio, the fresh air chilled water subcooling dehumidification heat exchanger flow, the fresh air reheating water heat exchanger flow and the humidification capacity of the humidifier.

[0007] Furthermore, the reinforcement learning reward function is:

[0008]

[0009]

[0010]

[0011]

[0012] in, R is the total reward function; The total power of the air-conditioning system equipment excluding the heat recovery system; The power of the heat recovery system circulation pump; is the power of the refrigeration and dehumidification unit; is the power of the reheat source equipment; is the humidifier power; Bonus item for heat recovery system; The temperature drop of fresh air pre-cooling by the heat recovery system; The temperature rise of supercooled fresh air caused by the heat recovery system; For indoor temperature bonus items; is the current indoor temperature; is the optimal indoor temperature value; The minimum indoor temperature limit; The maximum indoor temperature limit; Bonus item for indoor humidity; is the current indoor humidity; For the optimal indoor humidity content; It is the minimum indoor humidity limit; It is the maximum limit of indoor humidity; are the weight coefficients of heat recovery bonus, indoor temperature bonus and indoor humidity bonus respectively, and is the Gaussian function parameter, and exp is the natural exponential.

[0013] The present invention has the following beneficial effects: The present invention adopts the reinforcement learning method in machine learning. By defining the state space, action space, and reward function, the deep reinforcement learning algorithm DQN is used to construct a reinforcement learning model for optimizing the control strategy of the fresh air air-conditioning system in high-level biosafety laboratories that adopt supercooling dehumidification and fresh air heat recovery technology. While achieving strict control of the indoor temperature and humidity in high-level biosafety laboratories, the energy consumption of the air-conditioning system is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the process diagram of the fresh air air conditioning system for high-level biosafety laboratories; Figure 2 A flow chart of a machine learning control strategy optimization method for a fresh air air conditioning system; Figure 3 This is a diagram of the basic principles of reinforcement learning; Figure 4 This is the DQN algorithm control framework diagram for the high-level biosafety laboratory air conditioning system; Figure 5 This is a graph showing how the temperature reward item changes with the indoor temperature; Figure 6 This is a graph of outdoor temperature and humidity on a typical summer day; Figure 7 Graph showing cumulative rewards and total device power changes during training; Figure 8 This is the effect diagram of indoor temperature and humidity control. DETAILED DESCRIPTION

[0015] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0016] An embodiment of the present invention provides a temperature and humidity adjustment control method for a fresh air air conditioning system in a high-level biosafety laboratory, which is used to support strict indoor temperature control in a high-level biosafety laboratory and maximize energy saving of the air conditioning system on this basis.

[0017] like Figure 1 The figure shows the process diagram of the fresh air conditioning system for a high-level biosafety laboratory. This fresh air conditioning system delivers a fixed exhaust air volume and employs a circulating heat recovery technology that combines condensation dehumidification with heat exchangers, fresh air heat recovery, and exhaust air heat recovery. The air conditioning system primarily controls fresh air temperature and humidity through two components: the fresh air treatment system and the heat recovery system. The fresh air treatment system, consisting of a chilled water subcooling and dehumidification heat exchanger (heat exchanger #2), a fresh air reheating heat exchanger (heat exchanger #4), and a humidifier, serves as the primary device for regulating fresh air temperature and humidity. The circulating heat recovery system, comprised of a heat recovery precooling heat exchanger (heat exchanger #1), a heat recovery reheating heat exchanger (heat exchanger #3), an exhaust air heat recovery heat exchanger (heat exchanger #5), and a circulating water pump, recovers exhaust air cooling and fresh air subcooling and dehumidification cooling to precool and reheat the subcooled fresh air, reducing the fresh air load.

[0018] The system operation process under summer cooling conditions is as follows: 1. Outdoor fresh air enters the fresh air duct and heat recovery pre-cooling heat exchanger (1# heat exchanger) for heat exchange, and the outdoor fresh air temperature is reduced once; 2. The fresh air and chilled water dehumidification heat exchanger (2# heat exchanger) exchange heat and moisture, and the fresh air temperature is reduced and dehumidified for the second time; 3. The fresh air exchanges heat with the heat recovery and reheat heat exchanger (3# heat exchanger) after the three-way valve adjustment and diversion. The fresh air temperature rises and the subcooling and dehumidification cooling capacity is recovered. 4. The fresh air and the fresh air reheating water heat exchanger (4# heat exchanger) perform heat exchange, and the fresh air temperature rises again and is adjusted to a certain supply air temperature; 5. Fresh air is cleaned and delivered to the work area to change the indoor temperature and humidity.

[0019] 6. The exhaust air in the working area exchanges heat with the exhaust heat recovery heat exchanger (5# heat exchanger), which increases the exhaust air temperature and recovers the cold energy. The exhaust air is then filtered through activated carbon and discharged from the system.

[0020] like Figure 2 The figure shows a flow chart of a method for optimizing a machine learning control strategy for a fresh air air conditioning system. The embodiment of the present invention provides a method for optimizing a machine learning control strategy for a fresh air air conditioning system, comprising the following steps S1 to S4: S1. Establish a reinforcement learning state space based on the outdoor meteorological parameters of the fresh air air conditioning system, the circulating heat recovery system parameters, and the indoor heat source and indoor humidity source parameters; S2. Establish a reinforcement learning action space based on the fresh air temperature and humidity processing parameters of the fresh air air conditioning system for heat recovery, cooling and dehumidification, reheating, and humidification; S3. Use Gaussian function to establish a reinforcement learning reward function that balances the indoor environment temperature and humidity targets and the air conditioning system energy consumption targets; S4. Based on the reinforcement learning state space, reinforcement learning action space and reinforcement learning reward function, a deep reinforcement learning method is used to interact with the fresh air air conditioning system environment and train the reinforcement learning agent to control the fresh air air conditioning system.

[0021] Reinforcement learning (RL) is an important research area in machine learning. Standard RL consists primarily of an environment, an agent, actions (A={a1,a2,a3,…,at,…}), rewards (R={r1,r2,r3,…,rt,…}), and states (S={s1,s2,s3,…,st,…}). Reinforcement learning agents learn to perform continuous and optimal actions by interacting with their environment (e.g., buildings and HVAC systems affected by changing weather and indoor conditions). The fundamental principle of RL control is that when a system performs an action and receives a positive (large) reward from the environment, it reinforces that action; otherwise, it weakens that action later in the process.

[0022] like Figure 3 The figure shows the principle diagram of reinforcement learning. At time t, the agent receives the state st and reward rt from the environment. The agent will perform an action at from a set of candidate actions based on the state and reward. As a result, the environment changes to the next state st+1 and gives an immediate reward for the next state rt+1 based on the action at received from the RL agent. This process is repeated until the end. Then, a series of states, rewards, and actions {s1, a1, r1, s2…, st, rt, at,…} are obtained from this process. This is a Markov decision process (MDP). The state st+1 at time t+1 is determined by the state st and action at time t, and is independent of the previous historical states and actions. The goal of the Markov decision process is to find an optimal policy. By repeatedly interacting with the environment and updating the policy, the agent can eventually learn a policy for selecting the optimal action in each state, thereby maximizing the cumulative reward.

[0023] Reinforcement learning problems can usually be modeled as a Markov decision process (MDP), represented by a four-tuple: M =<S,A,R,P> ; where S represents the state space, A represents the action space, R represents the immediate reward received after transitioning from state st to state st+1, and P represents the probability that action at in state st will lead to state st+1.

[0024] The summer cooling conditions of the air-conditioning system of a high-level biosafety laboratory must achieve strict control of the indoor temperature and humidity of the high-level biosafety laboratory and tap the maximum energy-saving potential. This requires complex regulation of the heat recovery system and fresh air system equipment. There are many control variables and a high degree of nonlinearity. Traditional control methods have poor control effects and cannot achieve the optimal control effect and energy-saving goals.

[0025] Therefore, the control problem of the fresh air conditioning system in a high-level biosafety laboratory can be modeled as a Markov decision process. By establishing a suitable state space, action space, and reward function, the intelligent agent can interact with the environmental model and continuously learn and iterate to obtain the optimal control strategy.

[0026] The Deep Reinforcement Learning (DQN) algorithm is a deep reinforcement learning (DRL) algorithm that combines deep learning and reinforcement learning. DQN uses a deep neural network Q(s, a; θ) to approximate a state-action value (Q value) to address the curse of dimensionality caused by high-dimensional state and action spaces. This approach estimates the Q value of each action in the current state. Based on this Q value, the reinforcement learning agent selects the action a with the highest Q value in the current state s.

[0027] The iterative update of the neural network to approximate state-action value is based on the Bellman equation:

[0028] like Figure 4 The following is a DQN control framework diagram of the fresh air conditioning system in a high-level biosafety laboratory, which can be briefly described as follows: (1) Create two neural networks with the same structure: Q (policy network) and Q (target network), and initialize the parameters of the two networks; (2) The agent (controller) randomly selects an action with a probability of ε based on the current state (ε will decrease to εmin after a certain number of steps with a decreasing factor Δε). Otherwise, the agent selects the action with the highest Q value calculated by the Q-policy network; (3) After the agent takes action, it observes the next state s′ and rewards r, and stores the entire decision process (s, a, r, s′) in the experience replay area D; (4) The next round begins when the state terminates. When the experience replay area is full, each time the agent makes a decision, it will randomly select a small batch of samples from the experience replay area to update the network parameters. The update method is to minimize the Bellman error function:

[0029]

[0030] Where y is the estimated value of the target network Q, and the parameters are updated using stochastic gradient descent , each fixed step size makes θ equal to θ′.

[0031] The deep reinforcement learning (DQN) algorithm uses a deep neural network to approximate the Q-value function, effectively learning optimal or near-optimal strategies in scenarios with high-dimensional environments and complex features. The algorithm utilizes mechanisms such as the Experience Replay pool and the Target Network during training, making the training process more stable, reducing the risk of gradient divergence and over-estimation, and significantly improving the model's convergence speed and performance.

[0032] Based on the temperature and humidity control process of the fresh air air conditioning system of the high-level biosafety laboratory and the principle of reinforcement learning, a suitable reinforcement learning model state space, action space, and reward function are established.

[0033] State space: According to the temperature and humidity control process flow of the fresh air air conditioning system of the high-level biosafety laboratory, the inlet temperatures of the fresh air chilled water subcooling and dehumidification heat exchanger and the fresh air reheating water heat exchanger are fixed values, while the inlet temperatures of the heat recovery precooling heat exchanger, the heat recovery reheat heat exchanger and the exhaust heat recovery heat exchanger are all dynamically changing. Therefore, the state quantities that affect the indoor temperature and humidity of the high-level biosafety laboratory include outdoor temperature, outdoor humidity, the inlet temperature of the heat recovery precooling heat exchanger, the outlet temperature of the exhaust heat recovery heat exchanger, and the indoor heat source and humidity source of the high-level biosafety laboratory.

[0034]

[0035] in, is the outdoor temperature, °C; is the outdoor humidity, kg / kg; is the heat recovery pre-cooling heat exchanger inlet temperature, °C; is the outlet temperature of the heat recovery heat exchanger of the heat recovery system, °C; is the indoor heat source, kw; is the indoor humidity source, kg / s.

[0036] Action space: According to the controllable control quantities of the fresh air system and heat recovery system, select the heat recovery system circulation flow, the heat recovery system three-way valve diversion ratio, the fresh air chilled water subcooling dehumidification heat exchanger flow, the fresh air reheating water heat exchanger flow, and the humidifier humidification capacity respectively.

[0037]

[0038] in, is the total flow rate of the heat recovery system circulation pump, m3 / h; The diversion ratio of the three-way valve of the heat recovery system; is the fresh air subcooling dehumidification heat exchanger flow rate, m3 / h; is the fresh air reheat heat exchanger flow rate, m3 / h; is the fresh air humidification capacity, kg / s.

[0039] Reward function: The reduction of fresh air load in the fresh air conditioning system of a high-level biosafety laboratory mainly relies on the heat recovery system. The goal is to reduce energy consumption and achieve temperature and humidity control. Therefore, the reinforcement learning reward function that balances the indoor temperature and humidity of the high-level biosafety laboratory and the energy consumption of the air conditioning system should consist of four parts: energy consumption term, heat recovery term, indoor temperature term, and indoor humidity term. The design is as follows:

[0040]

[0041]

[0042]

[0043]

[0044] in, R is the total reward function; The total power of the air-conditioning system equipment excluding the heat recovery system; The power of the heat recovery system circulation pump; is the power of the refrigeration and dehumidification unit; is the power of the reheat source equipment; is the humidifier power; Bonus item for heat recovery system; The temperature drop of fresh air pre-cooling by the heat recovery system; The temperature rise of supercooled fresh air caused by the heat recovery system; For indoor temperature bonus items; is the current indoor temperature; is the optimal indoor temperature value; The minimum indoor temperature limit; The maximum indoor temperature limit; Bonus item for indoor humidity; is the current indoor humidity; For the optimal indoor humidity content; It is the minimum indoor humidity limit; It is the maximum limit of indoor humidity; are the weight coefficients of heat recovery bonus, indoor temperature bonus and indoor humidity bonus respectively, and is the Gaussian function parameter, and exp is the natural exponential.

[0045] like Figure 5 The figure below shows the temperature reward variation for a Gaussian function with a temperature penalty. 𝜆1 corresponds to the precision of the Gaussian function, that is, the steepness of the peak. 𝜆2 controls the slope of the trapezoidal function. When the target value is close and the interval constraint is satisfied, the function value is large. However, once the interval is exceeded, the function value decays sharply. Using a Gaussian function to incentivize the agent to control actions to maintain the indoor temperature (humidity) close to the central temperature (humidity) of the high-level biosafety laboratory is easier for the agent to learn than a simple trapezoidal reward function.

[0046] Taking the fresh air air conditioning system of a high-level biosafety laboratory as an example, the building area is 130m2, the height is 4m, the supply air volume and exhaust air volume are 6240m3 / h, the flow adjustment range of the heat recovery system circulation pump is 2.4m3 / h~4.3m3 / h, the flow adjustment range of the low-temperature chilled water of the fresh air subcooling dehumidification heat exchanger is 1m3 / h~4m3 / h, and the flow adjustment range of the fresh air reheating water heat exchanger is 0.5m3 / h~2m3 / h.

[0047] The indoor temperature and humidity standard requirements for the high-level biosafety laboratory are: Temperature: 22~24℃ Relative humidity: 40~50% For indoor heat sources and indoor moisture sources, a fixed schedule was established to simulate indoor thermal and moisture behaviors at different times.

[0048] like Figure 6 The figure shows the outdoor temperature and humidity changes under a typical summer day cooling condition. The outdoor temperature and humidity data and the indoor heat and humidity source schedule are used as state inputs.

[0049] like Figure 7 As shown in the figure, after 50 training set iterations, the cumulative reward gradually increases, the total energy consumption of the system continues to decrease, and converges after 20 training sets.

[0050] like Figure 8 As shown in the figure, the final trained reinforcement learning agent is able to control the indoor temperature between 22~24℃ and the indoor relative humidity between 40~50%.

[0051] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0052] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0054] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0055] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A machine learning control strategy optimization method for a fresh air air conditioning system, characterized in that: The following steps are involved: Establish a reinforcement learning state space based on the outdoor meteorological parameters of the fresh air air conditioning system, the circulating heat recovery system parameters, and the indoor heat source and indoor humidity source parameters; Establish a reinforcement learning action space based on the fresh air temperature and humidity processing parameters of the fresh air air conditioning system for heat recovery, cooling and dehumidification, reheating and humidification; A Gaussian function is used to establish a reinforcement learning reward function that balances the indoor environment temperature and humidity targets and the air conditioning system energy consumption targets; Based on the reinforcement learning state space, reinforcement learning action space and reinforcement learning reward function, a deep reinforcement learning method is used to interact with the fresh air air conditioning system environment, and the reinforcement learning agent is trained to control the fresh air air conditioning system.

2. The machine learning control strategy optimization method for a fresh air air conditioning system according to claim 1 is characterized in that: Outdoor meteorological parameters include outdoor temperature and humidity; The circulating heat recovery system parameters include the heat recovery pre-cooling heat exchanger inlet temperature and the exhaust heat recovery heat exchanger outlet temperature.

3. The method for optimizing a machine learning control strategy for a fresh air air conditioning system according to claim 1, wherein: The fresh air temperature and humidity processing parameters include the heat recovery system circulation pump flow, the heat recovery system three-way valve diversion ratio, the fresh air chilled water subcooling dehumidification heat exchanger flow, the fresh air reheating water heat exchanger flow and the humidifier humidification capacity.

4. The method for optimizing a machine learning control strategy for a fresh air conditioning system according to claim 1, wherein: The reinforcement learning reward function is: in, R is the total reward function; The total power of the air-conditioning system equipment excluding the heat recovery system; The power of the heat recovery system circulation pump; is the power of the refrigeration and dehumidification unit; is the power of the reheat source equipment; is the humidifier power; Bonus item for heat recovery system; The temperature drop of fresh air pre-cooling by the heat recovery system; The temperature rise of supercooled fresh air caused by the heat recovery system; For indoor temperature bonus items; is the current indoor temperature; is the optimal indoor temperature value; The minimum indoor temperature limit; The maximum indoor temperature limit; Bonus item for indoor humidity; is the current indoor humidity; For the optimal indoor humidity content; It is the minimum indoor humidity limit; It is the maximum limit of indoor humidity; are the weight coefficients of heat recovery bonus, indoor temperature bonus and indoor humidity bonus respectively, and is the Gaussian function parameter, and exp is the natural exponential.

Images