Building heating energy-saving safety control method and system based on hierarchical reinforcement learning

Abstract

The present application relates to the technical field of building HVAC intelligent control, in particular to a building HVAC energy-saving and safety control method and system based on hierarchical reinforcement learning, which adopts an asynchronous hierarchical architecture of a water-side upper global agent and a wind-side lower district agent, the upper agent performs macroscopic scheduling of chilled water and supply air temperature based on a soft actor-critic algorithm at a low frequency, and the lower agent completes precise adjustment of end air valves based on a multi-agent near-end policy optimization algorithm at a high frequency; the safety boundary of the control instruction is constrained by a deterministic physical inverse mapping and a dead zone truncation mechanism, a composite reward function embedded with a predicted mean thermal sensation index and a time-of-use electricity price is constructed, and closed-loop optimization is realized by combining digital twin pre-training and online fine-tuning of the actual machine. The present application effectively decouples the water-side and wind-side control logic, overcomes the curse of dimensionality, ensures the safety of equipment operation and indoor thermal comfort, realizes economic and energy-saving regulation and control of the building HVAC system, and significantly improves the engineering landing feasibility.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for building HVAC systems, specifically a building HVAC energy-saving and safety control method and system based on hierarchical reinforcement learning. Background Technology

[0002] With the advancement of global energy conservation and emission reduction goals, building energy consumption accounts for an increasingly prominent proportion of total social energy consumption, with HVAC systems typically accounting for more than 50% of the total energy consumption of commercial buildings. An HVAC system is a typical high-dimensional, nonlinear, complex thermodynamic system with large time delays and strong thermal inertia coupling characteristics.

[0003] Currently, building automation systems widely used in industry generally employ static rules based on expert experience and underlying proportional-integral-derivative feedback control loops. In recent years, with the improvement of IoT data acquisition capabilities and the development of artificial intelligence technology, data-driven intelligent control algorithms have begun to be widely introduced into the field of building energy management. Reinforcement learning, as a machine learning algorithm that maximizes long-term cumulative rewards through trial and error in unknown environments, can effectively handle nonlinear system optimization problems. This technology models the operation of building HVAC systems as a Markov decision process, deploying intelligent agents to observe environmental conditions in real time and output control actions, enabling the agents to autonomously learn and approach the optimal energy dispatch and setpoint distribution strategies through continuous interaction with the building's thermal environment.

[0004] In existing implementation schemes, centralized HVAC control methods based on single-agent deep reinforcement learning adopt a flattened centralized decision-making mode. A single global agent simultaneously receives state data from all floor zones of the entire building and outputs action commands for all controllable devices at the bottom level. Since the building HVAC system includes water-side devices with large thermal inertia and slow dynamic response, and wind-side devices with small thermal inertia and fast dynamic response, the severe coupling of spatiotemporal scales will cause the state space and action space of the Markov decision process to expand exponentially, causing the curse of dimensionality problem. This makes the algorithm extremely difficult to converge and has huge computational overhead, making it impossible to truly implement in large physical buildings containing dozens of independent zones.

[0005] Meanwhile, reinforcement learning has an unexplainable black box characteristic in the exploration and trial-and-error phase. Traditional energy-saving algorithms often only take the lowest absolute value of energy consumption as the sole reward orientation, lacking hard constraints on the underlying physical common sense. When pursuing extreme energy saving, the model is prone to outputting control commands that exceed the limits, which will not only aggravate the mechanical wear of the underlying physical hardware, but also violate the international minimum fresh air exchange standards, causing a sharp drop in the thermal comfort of indoor occupants. In addition, simply focusing on power consumption while ignoring the real-time time-of-use electricity price of the grid cannot achieve economic optimization at the billing level, and it is difficult to balance the multiple objectives of equipment safety, human experience and operating costs.

[0006] Therefore, in view of the above situation, there is an urgent need to develop a building HVAC energy-saving and safety control method and system based on hierarchical reinforcement learning to overcome the shortcomings in current practical applications. Summary of the Invention

[0007] The purpose of this invention is to provide a building HVAC energy-saving and safety control method and system based on hierarchical reinforcement learning to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A hierarchical reinforcement learning-based energy-saving and safety control method for building HVAC systems is applied to building HVAC systems containing multiple independent variable air volume terminal protection zones, including the following steps: An asynchronous hierarchical reinforcement learning control architecture is adopted, which includes a water-side upper-layer global agent and a wind-side lower-layer multi-zone agent. The upper-level global intelligent agent on the water side receives global status data at a first decision frequency and outputs chilled water supply temperature setting command and global air supply temperature setting command. The lower-level multi-zone agent on the wind side receives local status data of the corresponding zone and the global air supply temperature setting command at a second decision frequency, and outputs variable air volume valve opening commands for each zone; wherein, the second decision frequency is higher than the first decision frequency. Deterministic physical inverse mapping and safety boundary truncation are performed on the chilled water supply temperature setting command, the global air supply temperature setting command, and the variable air volume damper opening command, respectively, to constrain each control command within the physical safety operating range of the corresponding equipment. The water-side upper-layer global agent and the wind-side lower-layer multi-zone agent are both trained and optimized based on a composite reward function. The composite reward function includes at least an energy consumption cost term based on time-of-use electricity pricing, a thermal comfort penalty term based on the predicted average thermal sensation index, and an equipment safety penalty term.

[0009] As a further aspect of the present invention: the water-side upper-layer global agent is constructed based on the soft actor-critic algorithm, including a policy network, a double-Q value network, and a physical instruction inverse mapping unit; The policy network receives a global state vector, outputs a continuous action tensor, and maps it to a preset numerical range after Tanh activation. The dual-Q value network uses two independently initialized value networks with non-shared parameters. The target value is calculated by taking the minimum Q value output by the two value networks to suppress the overestimation bias of the value function. The physical command inverse mapping unit converts continuous action tensors into physical control commands through a deterministic linear mapping relationship, wherein: The mapping formula for the chilled water supply temperature setpoint is:

[0010] In the formula, Set the chilled water supply temperature. The first dimension of the continuous action output by the policy network; The mapping formula for the global supply air temperature setpoint is:

[0011] In the formula, This is the actual supply air temperature setting. The second dimension of the continuous action output by the policy network.

[0012] As a further aspect of the present invention: the wind-side lower-layer multi-zone agent is constructed based on a multi-agent near-end policy optimization algorithm and adopts a centralized training and distributed execution architecture; Each zone is set up with a local policy network. All local policy networks share network parameters. Each local policy network receives the local state vector of the corresponding zone and the global air supply temperature setting command, and outputs a one-dimensional continuous action. During the training phase, a global value network is configured. The global value network receives global state data and local state data of all defense zones, outputs global state value, and calculates the advantage function in combination with the generalized advantage estimation method to guide the parameter update of each local policy network. The opening commands for variable air volume dampers in each defense zone are deterministically mapped and truncated using the following formula:

[0013] in, The physical damper opening ratio for the i-th defense zone; For the continuous actions output by the strategy network of the i-th defense zone, the opening degree of the damper is constrained within the range from the preset minimum opening degree to the fully open degree.

[0014] As a further aspect of the present invention: the calculation formula for the composite reward function is as follows:

[0015] in, The output value of the system's joint reward at the current time step. This represents the weighting coefficient for the energy-saving and economical items. This represents the actual total power consumption of the chiller, water pumps, and fans within the current time step. This refers to the real-time time-of-use electricity price rate published by the power grid at the current moment. This is the weighting coefficient for the thermal comfort penalty term. Let i be the predicted average thermal sensation index calculated based on the current indoor temperature and humidity for the i-th defense zone. This is a physical penalty item for equipment safety. When the fluctuation of the damper's movement is detected to exceed a preset threshold, the corresponding penalty value is triggered.

[0016] As a further aspect of the present invention: the training and deployment steps of the upper-layer global intelligent agent on the water side and the lower-layer multi-zone intelligent agent on the wind side include: Based on building parameters and historical operating data, a digital twin simulation environment is constructed through a building energy consumption simulation engine. Each agent is pre-trained offline in the digital twin simulation environment. After the composite reward function converges, the trained agent model is deployed to the actual machine for operation. During actual operation, real-world trajectory data is continuously collected, and the parameters of the agent model are updated online in small increments at a preset low learning rate to adapt to the dynamic characteristics of real buildings, forming a closed-loop optimization of offline pre-training and online fine-tuning.

[0017] A building HVAC energy-saving and safety control system based on hierarchical reinforcement learning is applied to building HVAC systems containing multiple independent variable air volume terminal protection zones, including: The data acquisition module is used to collect global status data and local status data of each defense zone, and perform standardized preprocessing. The upper-level global scheduling module has a built-in water-side upper-level global intelligent agent, which is used to output chilled water supply temperature setting instructions and global air supply temperature setting instructions based on global state data at the first decision frequency. The lower-level local execution module has a built-in wind-side lower-level multi-zone intelligent agent, which is used to output the variable air volume valve opening command of each zone based on the local state data of the corresponding zone and the global air supply temperature setting command at a second decision frequency; wherein, the second decision frequency is higher than the first decision frequency. The safety constraint module is used to perform deterministic physical inverse mapping and safety boundary truncation processing on the chilled water supply temperature setting command, the global air supply temperature setting command, and the variable air volume valve opening command, respectively, to constrain each control command within the physical safety operating range of the corresponding equipment. The reward calculation module is used to calculate a composite reward function to train and optimize each agent. The composite reward function includes at least an energy consumption cost term based on time-of-use electricity pricing, a thermal comfort penalty term based on the predicted average thermal sensation index, and an equipment safety penalty term.

[0018] As a further aspect of the present invention: the water-side upper-layer global agent in the upper-layer global scheduling module is constructed based on the soft actor-critic algorithm, including a policy network unit, a double-Q value network unit, and a physical inverse mapping unit; The policy network unit is used to receive the global state vector and output a continuous action tensor. The dual-Q value network unit uses two independent value networks and takes the minimum Q value of the two outputs to calculate the target value, so as to suppress the overestimation bias of the value function. The physical inverse mapping unit is used to convert the continuous motion tensor into chilled water supply temperature setpoint and global air supply temperature setpoint through deterministic linear mapping, and to constrain the setpoints within the safe operating range of the equipment.

[0019] As a further aspect of the present invention: the wind-side lower-layer multi-zone agent in the lower-layer local execution module is constructed based on the multi-agent near-end policy optimization algorithm and adopts a centralized training and distributed execution architecture; The wind-side lower-layer multi-zone intelligent agent includes multiple local policy network units that share parameters. Each local policy network unit corresponds to a zone and is used to output the wind valve control action of the corresponding zone. During the training phase, a global value network unit is configured to output the global state value based on global state data and local state data of all defense zones, and to guide the parameter update of each local policy network unit in combination with the generalized advantage estimation method. The safety constraint module performs dead zone truncation processing on the valve opening command, forcibly retaining the minimum fresh air exchange opening.

[0020] As a further aspect of the present invention: the composite reward function calculation formula executed by the reward calculation module is as follows:

[0021] in, The output value of the system's joint reward at the current time step. This represents the weighting coefficient for the energy-saving and economical items. This represents the actual total power consumption of the chiller, water pumps, and fans within the current time step. This refers to the real-time time-of-use electricity price rate published by the power grid at the current moment. This is the weighting coefficient for the thermal comfort penalty term. Let i be the predicted average thermal sensation index calculated based on the current indoor temperature and humidity for the i-th defense zone. This is a physical penalty item for equipment safety. When the fluctuation of the damper's movement is detected to exceed a preset threshold, the corresponding penalty value is triggered.

[0022] As a further aspect of the present invention, it also includes a model iteration deployment module, which comprises a digital twin pre-training unit and a live online fine-tuning unit; The digital twin pre-training unit is used to construct a digital twin simulation environment based on the building energy consumption simulation engine, and to complete the offline pre-training and convergence of each intelligent agent; The real-machine online fine-tuning unit is used to collect real-world trajectory data during real-machine operation and make minor updates to the parameters of the intelligent agent model with a low learning rate to adapt to the dynamic characteristics changes of the real building.

[0023] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs an asynchronous hierarchical control architecture of "water-side global low frequency - wind-side local high frequency", which logically decouples the control cycle of building thermal inertial equipment and terminal fast dynamic equipment. It breaks down the originally huge joint action space into upper-layer macro temperature setting and lower-layer micro valve adjustment, fundamentally overcoming the dimensionality curse faced by single-layer reinforcement learning in multi-node collaboration of large buildings. This dimensionality reduction decision-making method enables the algorithm to achieve rapid convergence under limited edge gateway computing power, greatly improving the feasibility of the model in engineering implementation under complex working conditions containing dozens of independent defense zones. This invention rigidly embeds a deterministic physical inverse mapping and dead zone truncation mechanism between the network output and the underlying programmable logic controller. It also integrates a cliff-like penalty for the predicted average thermal sensation index and time-of-use electricity price constraints into the reward function. This completely prevents dangerous actions that may be taken by artificial intelligence algorithms during the exploration and trial-and-error process. Without the need to build a high-cost and high-precision building thermodynamic model, it achieves the economic optimization of peak shaving and valley filling using building thermal inertia, while effectively ensuring the absolute mechanical safety of the underlying hardware and the health bottom line of indoor personnel. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the energy-saving and safety control method for building HVAC based on hierarchical reinforcement learning in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the hierarchical intelligent agent asynchronous multi-frequency control network architecture in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of deterministic physical inverse mapping and secure truncation logic in an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the electronic device structure for implementing the control method in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0030] Please see Figures 1-4 The energy-saving and safety control method and system for building HVAC based on hierarchical reinforcement learning provided in this invention are applied to commercial building HVAC systems containing multiple independent variable air volume (VAV) terminal protection zones. The system includes a chiller unit, a chilled water circulating pump, a cooling water circulating pump, an air conditioning unit, VAV terminal devices distributed in each protection zone, environmental and equipment sensors, an edge computing gateway, and a programmable logic controller for each protection zone. This embodiment uses a building scenario with 10 independent VAV terminal protection zones as an example for illustration; the scope of protection of this invention is not limited thereto.

[0031] The control method in this embodiment specifically includes the following steps: Step S1: Construction of multi-source heterogeneous datasets and alignment of state space tensors The system uses the standard BACnet / IP protocol of building automation systems to collect real-time operating data from various sensors at the bottom layer, and constructs the global state vector and local state vector of each defense zone required for reinforcement learning, providing standardized input features for subsequent hierarchical intelligent agents.

[0032] Specifically, a 65-dimensional global state vector is constructed, which contains four types of feature data: The first category is 3D outdoor meteorological parameters, specifically including outdoor dry-bulb temperature, relative humidity, and solar radiation illuminance. The second category is 21-dimensional macro-building features, specifically including the average return air temperature of 10 defense zones, the total active power meter reading of the building, and the personnel occupancy rate of 10 defense zones; The third category consists of 37 core equipment and historical status parameters, specifically including 6 current core equipment parameters: real-time operating power of the chiller, chilled water supply temperature, chilled water return temperature, cooling water supply temperature, cooling water return temperature, and frequency feedback value of the main circulating water pump inverter. A unique thermal encoding is used to represent the 3-dimensional signal of the current time-of-use electricity price being in the peak, flat, or valley segment, as well as 28 historical lag characteristics. The 28 historical lag characteristics are composed of the total active power of the building and the lag values ​​of the above 6 core equipment parameters in the first 4 time steps, totaling 7 multiplied by 4 dimensions. The fourth category is 4-dimensional time features. The values ​​for the current 0-24 hours and 1-12 months are converted into sine and cosine values ​​respectively, forming 4-dimensional time coding features to represent diurnal and seasonal rhythms. For each defense zone, a 16-dimensional local defense zone state vector is constructed. For the i-th defense zone, the local defense zone state vector includes real-time indoor temperature probe values, thermostat set target temperature, duct static pressure sensor values, current variable air volume box valve opening feedback values, local personnel carbon dioxide concentration values, and the lag sequences of the above five variables in the previous two time steps, totaling 15 dimensions, plus a 1-dimensional defense zone identification code. After data collection, the system processes all continuous floating-point input data using a maximum-minimum value normalization method. The data processing formula is:

[0033] in, is the dimensionless value input to the neural network after normalization; x is the original value collected by the sensor. This represents the physical minimum value of the sensor over a historical year; This represents the physical maximum value of the sensor over a historical year.

[0034] After normalization, all state tensors are strictly mapped to the dimensionless interval between -1 and 1. Maximum and minimum value normalization is a common technique in deep neural network input processing, effectively improving the convergence speed and prediction stability of network training.

[0035] Through the classification and normalization of the above multi-source features, heterogeneous data with different dimensions and sampling frequencies can be unified into a standard tensor format that can be directly processed by neural networks. The global state vector covers four types of information: macro-meteorology, overall building, core equipment and time rhythm, providing a complete decision basis for upper-level global scheduling. The local state vector focuses on the real-time operating status of a single defense zone, providing precise local sensing input for lower-level end-point adjustment.

[0036] Step S2: Construction of the upper-layer global resource scheduling agent on the waterside based on the soft actor-critic algorithm A high-level global agent based on the soft actor-critic algorithm is deployed in the building edge computing gateway. This high-level global agent is responsible for low-frequency macroscopic energy consumption decisions, with a fixed decision frequency of forward propagation calculation every 15 minutes, matching the physical characteristics of the chiller unit's high thermal inertia and slow dynamics. The corresponding hierarchical control architecture for this step is as follows: Figure 2 As shown, the physical inverse mapping logic is as follows: Figure 3 As shown.

[0037] The upper-layer global agent comprises three parts: a policy network, a value network, and a physical instruction inverse mapping unit. (1) Policy Network Construction: The policy network adopts a multilayer perceptron architecture. The input layer receives a 65-dimensional global state vector; the number of nodes in the first hidden layer is set to 256, and the activation function is ReLU; the number of nodes in the second hidden layer is set to 128, and the activation function is ReLU. The output layer is connected to the second hidden layer and consists of two sets of parallel linear neurons fully connected layers, which output a Gaussian distributed mean vector and a logarithmic standard deviation vector, respectively. The system uses the reparameterization technique to introduce a standard normal distribution noise vector for sampling, and finally outputs a 2-dimensional continuous action tensor through the Tanh activation function. The numerical distribution range of the action tensor is -1 to 1. The reparameterization technique is a conventional sampling method for continuous action space reinforcement learning policy networks, which can ensure the effective backpropagation of policy gradients.

[0038] (2) Value network construction The value network employs a dual-Q network architecture to eliminate action overestimation. This involves instantiating two identical value networks, a first value network and a second value network, in parallel, but with independently initialized internal weight parameters. The input layer of both value networks is a concatenation of a 65-dimensional global state vector and a 2-dimensional action tensor, resulting in a total dimension of 67. Both value networks contain three fully connected hidden layers with 256, 256, and 128 nodes respectively, and ReLU is used as the activation function. The output layer of each network consists of a single linear neuron, outputting the first scalar Q-value and the second scalar Q-value, respectively.

[0039] When calculating the target value update loss function, the system extracts the Q-values ​​of the two network outputs and uses the formula... The minimum of the two values ​​is used in subsequent calculations, thereby forcibly suppressing the positive overestimation bias generated by reinforcement learning in value function approximation and improving the accuracy of policy evaluation.

[0040] (3) Physical instruction inverse mapping operation The 2D action tensor output by the neural network is a dimensionless value and cannot directly control physical devices. The system converts it into a real temperature setpoint through a deterministic linear mapping relationship and sends it to the underlying controller via the ModbusTCP protocol.

[0041] The first dimension of the action is mapped to the global chilled water supply temperature setpoint, and the mapping formula is as follows:

[0042] In the formula, This is the actual chilled water supply temperature setting, in degrees Celsius. This is the first-dimensional continuous action output by the strategy network. This formula strictly maps the action value output by the network to a range of 5 to 9 degrees Celsius, corresponding to the safe and efficient operating range of the compressor.

[0043] The second-dimensional action is mapped to the global supply air temperature setpoint, and the mapping formula is as follows:

[0044] In the formula, This is the actual supply air temperature setting, in degrees Celsius. This is the second-dimensional continuous action output by the strategy network. This formula strictly maps the action values ​​output by the network to a safe air supply range of 12 to 16 degrees Celsius.

[0045] By constructing a high-level global intelligent agent based on the soft actor-critic algorithm, a low-frequency decision cycle of 15 minutes is used to match the thermal inertia characteristics of the chiller unit, avoiding equipment damage and energy waste caused by high-frequency adjustment. The dual-Q network architecture effectively suppresses the overestimation problem of the value function and improves the accuracy of policy evaluation. Most importantly, through a deterministic linear inverse mapping formula, the dimensionless actions output by the neural network are strictly limited to the physical safety range of the equipment. This completely blocks the possibility of extreme out-of-limit instructions being output during the reinforcement learning exploration phase from the execution level, ensuring the operational safety of the chiller and air conditioning unit and solving the physical safety risks caused by the black-box output of pure data-driven algorithms.

[0046] In addition to the soft actor-critic algorithm used in this embodiment, the global agent responsible for macro-level cold allocation at the upper layer can also use a deep deterministic policy gradient algorithm or a near-end policy optimization algorithm, which can also achieve the technical effect of low-frequency global scheduling.

[0047] Step S3: Construction of a wind-side lower-level defense zone cooperative execution network based on a multi-agent proximal policy optimization algorithm Within the programmable logic controllers of 10 independent zones, lower-level agents based on a multi-agent near-end policy optimization algorithm are deployed. These lower-level agents are responsible for high-frequency and precise tracking of cooling commands issued from the upper layer, with a fixed decision-making frequency of once per minute, matching the low thermal inertia and fast dynamic physical characteristics of variable air volume terminals. This step employs a centralized training and distributed execution architecture. During the training phase, global information is used to optimize the strategy; during the runtime phase, each zone makes independent decisions based only on its local state. The corresponding architecture is as follows: Figure 2 As shown.

[0048] Specifically, ten local policy networks with shared network parameters are instantiated. The input layer of each local policy network receives a 16-dimensional local defense zone state vector and a normalized scalar of the air supply temperature setpoint from the upper layer, resulting in a total input dimension of 17. The hidden layer consists of two fully connected layers, each with 64 nodes, and uses ReLU as the activation function. The output layer outputs a 1-dimensional continuous action using the Tanh activation function, with a numerical distribution ranging from -1 to 1. This parameter-sharing multi-agent setup significantly reduces the number of network parameters, improves training efficiency and cross-defense zone generalization ability, and is a common implementation method for multi-agent reinforcement learning.

[0049] The lower-level value network is used only during the training phase. It employs a global perspective design, with its input layer consisting of a 65-dimensional global state vector and 160-dimensional local state vectors from 10 defense zones, for a total dimension of 225. The lower-level value network outputs a global state value and, combined with a generalized advantage estimation method, calculates an advantage function to guide gradient updates in the policy networks of each defense zone. The advantage function is defined as follows:

[0050] in, The dominant function value of defense zone i at time step t; This is a discount factor for future rewards. To estimate the smoothing parameters for generalized dominance; The timing difference error is calculated using the following formula:

[0051] in, For the instant reward of time step t, and These are the global state value scalar predictions output by the lower-level value network at time step t and time step t+1, respectively.

[0052] Furthermore, the physical command inverse mapping operation of the terminal air valve of the defense zone is executed, and the corresponding logic is as follows: Figure 3 As shown. The mapping formula for the variable air volume valve opening command of the i-th zone is:

[0053] in, The physical damper opening ratio for the i-th defense zone; The continuous actions output by the policy network for the i-th defense zone.

[0054] This formula ensures that the opening of the damper is physically limited to between 20% and 100%, reserving 20% ​​opening as the minimum dead zone for the fresh air exchange valve.

[0055] In addition to the deterministic linear inverse mapping and dead zone truncation method used in this embodiment, a restricted Markov decision process framework can also be adopted, introducing the Lagrange multiplier method to dynamically constrain the actions that exceed the limits during the network parameter update stage; or a control barrier function can be introduced, and the unsafe actions output by the neural network can be projected in real time to the safe and feasible set allowed by the compressor and fan at each time step through a quadratic programming algorithm, which can also achieve the purpose of preventing damage to the underlying physical equipment and violating the fresh air standard.

[0056] By constructing a lower-level execution network based on a multi-agent near-end policy optimization algorithm, a high-frequency decision cycle of 1 minute is used to match the dynamic characteristics of variable air volume terminals, achieving accurate real-time response to the cooling demand of each zone. A centralized training distributed execution architecture is adopted, which not only utilizes global information to ensure the collaborative optimization effect of multiple zones, but also realizes distributed rapid decision-making during the operation phase, reducing the dependence on the computing power of the edge gateway. By using a hard dead zone cutoff of the air valve opening, the minimum fresh air exchange rate is forcibly retained, which fundamentally eliminates the indoor air quality degradation and personnel health risks caused by the complete closure of the air valve, while avoiding the extreme behavior of reinforcement learning to shut off the fresh air in pursuit of ultimate energy saving.

[0057] Thus, by using an asynchronous layered architecture of low-frequency global scheduling at the upper layer and high-frequency local execution at the lower layer, the originally coupled water-side and wind-side control logic is completely decoupled, significantly compressing the state and action space of each layer of the network. This fundamentally overcomes the dimensionality curse problem faced by single-layer centralized reinforcement learning in large buildings, and significantly improves the algorithm's convergence speed and engineering feasibility.

[0058] In addition to the multi-agent near-end policy optimization algorithm used in this embodiment, the multi-agents responsible for the micro-execution at the end of the defense zone can also adopt the multi-agent deep deterministic policy gradient algorithm or a distributed control architecture based on independent Q-learning, which can also achieve the technical effects of high-frequency end-end adjustment and multi-zone coordination.

[0059] Step S4: Design of a composite reward function for hard embedding the predicted average thermal sensation index and the physical boundary of electricity price. To guide the network gradients of upper and lower layer agents towards updates that balance human thermal comfort and electricity cost efficiency, a composite objective reward function is constructed within a reinforcement learning interactive environment to constrain the algorithm's optimization direction at the objective level. The calculation formula for the composite objective reward function is as follows:

[0060] in, The scalar output value of the system's joint reward at the current time step; The weighting coefficient for the energy-saving and economical items is fixed at 1.0. This represents the actual total power consumption of the chiller, water pumps, and fans within the current time step, in kilowatt-hours. The real-time time-of-use electricity price rate published by the power grid at the current moment is expressed in yuan per kilowatt-hour. The weighting coefficient for the thermal comfort penalty is fixed at 5.0; The predicted average thermal comfort index for the i-th zone is calculated based on the current indoor temperature and humidity. The specific calculation is completed by calling the standard Fanger thermal comfort model. The input variables are directly the indoor temperature in the local zone state vector, and the indoor wind speed is fixed at 0.15 m / s, relative humidity at 50%, standard human metabolic rate at 1.0 Mt, and clothing thermal resistance at 0.5 clo. Physical penalties for equipment safety.

[0061] When the absolute value of the predicted average thermal sensation index of the i-th defense zone is less than or equal to 0.5, it meets the international thermal comfort standard. A value of 0 results in no comfort penalty; however, once the absolute value of the predicted average thermal sensation index exceeds the 0.5 boundary, a quadratic exponential cliff penalty will be triggered.

[0062] In addition, the system monitors the fluctuations in the operation of air valves in each defense zone in real time. If the fluctuation in the air valve operation output by any lower-level agent exceeds 30% within three consecutive time steps, then a setting is implemented. The value is 10.0, otherwise it is set to 0.0.

[0063] The fluctuation in valve action is determined by the absolute difference between the physical valve opening ratios of adjacent time steps; whereby the physical valve opening ratio is the actual control quantity obtained after deterministic physical inverse mapping and safety boundary truncation of the continuous actions output by the lower-level agent. The formula for calculating the fluctuation in valve action of the i-th zone at time step t is:

[0064] In the first equation, the left side represents the fluctuation of the damper action in the i-th zone at time step t; the two terms on the right side represent the ratio of the physical damper opening in the current time step and the previous time step, respectively, with values ​​ranging from 0.20 to 1.00. When any zone has an absolute difference greater than 0.30 in the above terms for three consecutive time steps, the equipment safety physical penalty term is set to 10.0; otherwise, it is set to 0.0.

[0065] By designing the aforementioned composite reward function, the three optimization objectives of energy consumption cost, thermal comfort, and equipment safety are unified and shaped: time-of-use electricity rates are introduced to replace simple energy consumption values, guiding the algorithm to utilize building thermal inertia to store cold during off-peak hours and release cold during peak hours, achieving peak shaving and valley filling for economic optimization, rather than simply pursuing the lowest absolute value of energy consumption; a cliff-like secondary penalty mechanism is adopted to rigidly constrain the compliance range of the predicted average thermal sensation index, ensuring that indoor thermal comfort always meets international standards and avoiding the algorithm sacrificing human experience for energy saving; an equipment motion fluctuation penalty item is added to force the network to output smooth control commands, reduce frequent mechanical structure movements, extend equipment lifespan, and reduce the risk of failures such as motor burnout.

[0066] In addition to the standard Fanger thermodynamic model and fixed-weight composite reward function used in this embodiment, thermal comfort evaluation can also use a data-driven thermal sensation voting prediction model trained based on historical real-person voting feedback data as the evaluation benchmark; the optimization strategy can also adopt a multi-objective reinforcement learning framework to directly solve the Pareto front of energy consumption and comfort, allowing building managers to switch different operating modes in actual operation, which can also achieve the effect of multi-objective balance optimization.

[0067] Step S5: Iteration of closed-loop model based on digital twin pre-training and deployment of security fine-tuning at the edge of the actual machine The system extracts building material parameters, physical orientation, and historical one-year operation datasets of real buildings, and uses the EnergyPlus building energy consumption simulation engine in the cloud to build a high-fidelity digital twin Markov decision-making process interactive environment to avoid damage to physical equipment caused by random exploration during the cold start phase of reinforcement learning models.

[0068] In addition to the EnergyPlus simulation engine used in this embodiment, other professional building thermophysics simulation software such as Modelica, TRNSYS, or DeST can also be used; or the white-box physical mechanism can be completely abandoned, and the massive operation dataset of the building automation system over many years can be used to train a pure data-driven deep sequence agent model such as a long short-term memory network, a gated recurrent unit, or a Transformer as a virtual interactive environment, which can also complete the closed-loop pre-training of the cold start phase of the reinforcement learning model.

[0069] In the offline high-fidelity pre-training phase, the Adam optimizer is used to update network weights. The initial learning rate for the upper-layer soft actor-critic algorithm network is set to 0.03%, and the initial learning rate for the lower-layer multi-agent proximal policy optimization algorithm network is set to 0.05%. The experience replay pool capacity is set to 1,000,000, the batch size is set to 256, and the future reward discount factor is set. The smoothing parameter for generalized dominance estimation is set to 0.99. The value is set to 0.95, and the policy pruning factor is set to 0.2. The Adam optimizer and experience replay mechanism are standard configurations for deep reinforcement learning training.

[0070] For the upper-level soft actor-critic algorithm, to stabilize the training of the double-Q network, in addition to instantiating the first and second value networks, a first target value network and a second target value network with identical structures are also instantiated. During training, a soft update mechanism is used to smoothly synchronize the parameters of these two target value networks. The update formula is as follows:

[0071] Where k takes the value of 1 or 2, representing the first or second value network, respectively; The weight parameters are for the target value network k; These are the weight parameters of the current value network k; The target network soft update coefficient is fixed at 0.005. The target network soft update mechanism is a common method for stable training in deep reinforcement learning, which can avoid training oscillations caused by drastic fluctuations in the target value.

[0072] The model underwent 20,000 rounds of exploration in the simulation environment until the composite objective reward function converged, completing closed-loop training, and finally solidifying the weights of each network layer into a standard ONNX format file.

[0073] After training, the solidified weight file is distributed to the edge computing gateway and each zone controller of the actual building. During online operation, the network only outputs predicted actions through forward propagation, and then issues control commands after performing inverse mapping. Simultaneously, the experience replay pool within the gateway continuously collects the state, actions, and actual energy consumption reward trajectories of the real building. Utilizing the idle computing power at night, the system retrieves the collected real trajectory data and performs online adaptive micro-updates of the policy network parameters at an extremely low learning rate of one in ten thousand. This eliminates the model gap between the simulation environment and the real physical building, enabling adaptive closed-loop control of long-term mechanical aging of HVAC equipment and seasonal changes in building occupancy rates.

[0074] By pre-training in a digital twin simulation environment, reinforcement learning models can complete most of their exploration and learning processes in a virtual environment. This avoids the impact and damage to physical equipment caused by random strategies during the cold start phase, significantly improving the security and efficiency of physical deployment. Mechanisms such as target network soft updates and experience replay are used to ensure the stability and convergence of the training process. Through online fine-tuning with a low learning rate during the physical operation phase, the system can continuously adapt to the dynamic changes in the characteristics of real buildings, ensuring the long-term effectiveness of the control effect and forming a complete closed loop of "offline pre-training - physical deployment - online iteration".

[0075] This embodiment also provides a building HVAC energy-saving and safety control system based on hierarchical reinforcement learning, used to implement the above control method. The system includes: The data acquisition module is used to perform multi-source heterogeneous data acquisition and state tensor alignment processing; The upper-level global scheduling module is used to execute the inference and control commands issued by the water-side global resource scheduling intelligent agent; The lower-level local execution module is used to execute the inference and wind valve control commands of the end-control agents in each wind-side defense zone. The reward calculation module is used to perform compound reward function calculations, providing optimization objectives for model training; The model iteration deployment module is used to perform digital twin pre-training, model solidification and distribution, and online fine-tuning on real machines.

[0076] Furthermore, this embodiment also provides an electronic device for implementing the above-described building HVAC energy-saving and safety control method, such as... Figure 4 As shown, the electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0077] The memory is used to store computer programs and may include high-speed random access memory or non-volatile memory, such as at least one disk storage device.

[0078] The communication interface is used to communicate with hardware devices such as building automation systems, edge computing gateways, low-level programmable logic controllers of various defense zones, and external IoT sensors. It uses standard industrial protocols such as BACnet / IP or ModbusTCP to collect status data and issue control commands after physical inverse mapping.

[0079] The processor may be a central processing unit, a special purpose integrated circuit, or one or more integrated circuits configured to implement embodiments of the present invention. The processor is used to call and execute programs stored in memory to implement the steps of the above-described building HVAC energy-saving and safety control method based on hierarchical reinforcement learning.

[0080] This embodiment also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the above-described building HVAC energy-saving and safety control method based on hierarchical reinforcement learning. The storage medium includes various media capable of storing program code, such as read-only memory, random access memory, magnetic disk, or optical disk.

[0081] It should be noted that, in this invention, although the specification describes the embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A building HVAC energy-saving and safety control method based on hierarchical reinforcement learning, applied to a building HVAC system containing multiple independent variable air volume terminal protection zones, characterized in that, Includes the following steps: An asynchronous hierarchical reinforcement learning control architecture is adopted, which includes a water-side upper-layer global agent and a wind-side lower-layer multi-zone agent. The upper-level global intelligent agent on the water side receives global status data at a first decision frequency and outputs chilled water supply temperature setting command and global air supply temperature setting command. The lower-level multi-zone agent on the wind side receives local status data of the corresponding zone and the global air supply temperature setting command at a second decision frequency, and outputs variable air volume valve opening commands for each zone; wherein, the second decision frequency is higher than the first decision frequency. Deterministic physical inverse mapping and safety boundary truncation are performed on the chilled water supply temperature setting command, the global air supply temperature setting command, and the variable air volume damper opening command, respectively, to constrain each control command within the physical safety operating range of the corresponding equipment. The water-side upper-layer global agent and the wind-side lower-layer multi-zone agent are both trained and optimized based on a composite reward function. The composite reward function includes at least an energy consumption cost term based on time-of-use electricity pricing, a thermal comfort penalty term based on the predicted average thermal sensation index, and an equipment safety penalty term.

2. The building HVAC energy-saving and safety control method based on hierarchical reinforcement learning according to claim 1, characterized in that, The waterside upper-layer global agent is constructed based on the soft actor-critic algorithm and includes a policy network, a double-Q value network, and a physical instruction inverse mapping unit. The policy network receives a global state vector, outputs a continuous action tensor, and maps it to a preset numerical range after Tanh activation. The dual-Q value network uses two independently initialized value networks with non-shared parameters. The target value is calculated by taking the minimum Q value output by the two value networks to suppress the overestimation bias of the value function. The physical command inverse mapping unit converts continuous action tensors into physical control commands through a deterministic linear mapping relationship, wherein: The mapping formula for the chilled water supply temperature setpoint is: In the formula, Set the chilled water supply temperature. The first dimension of the continuous action output by the policy network; The mapping formula for the global supply air temperature setpoint is: In the formula, This is the actual supply air temperature setpoint. The second dimension of the continuous action output by the policy network.

3. The building HVAC energy-saving and safety control method based on hierarchical reinforcement learning according to claim 1, characterized in that, The wind-side lower-layer multi-zone agent is constructed based on a multi-agent near-end policy optimization algorithm and adopts a centralized training and distributed execution architecture. Each zone is set up with a local policy network. All local policy networks share network parameters. Each local policy network receives the local state vector of the corresponding zone and the global air supply temperature setting command, and outputs a one-dimensional continuous action. During the training phase, a global value network is configured. The global value network receives global state data and local state data of all defense zones, outputs global state value, and calculates the advantage function in combination with the generalized advantage estimation method to guide the parameter update of each local policy network. The opening commands for variable air volume dampers in each defense zone are deterministically mapped and dead-zone truncation processed using the following formula: in, The physical damper opening ratio for the i-th defense zone; For the continuous actions output by the strategy network of the i-th defense zone, the opening degree of the damper is constrained within the range from the preset minimum opening degree to the fully open degree.

4. The building HVAC energy-saving and safety control method based on hierarchical reinforcement learning according to claim 1, characterized in that, The formula for calculating the composite reward function is as follows: in, The output value of the system's joint reward at the current time step. This represents the weighting coefficient for the energy-saving and economical items. This represents the actual total power consumption of the chiller, water pumps, and fans within the current time step. This refers to the real-time time-of-use electricity price rate published by the power grid at the current moment. This is the weighting coefficient for the thermal comfort penalty term. Let i be the predicted average thermal sensation index calculated based on the current indoor temperature and humidity for the i-th defense zone. This is a physical penalty item for equipment safety. When the fluctuation of the damper's movement is detected to exceed a preset threshold, the corresponding penalty value is triggered.

5. The building HVAC energy-saving and safety control method based on hierarchical reinforcement learning according to claim 1, characterized in that, The training and deployment steps for the upper-layer global agent on the water side and the lower-layer multi-zone agent on the wind side include: Based on building parameters and historical operating data, a digital twin simulation environment is constructed through a building energy consumption simulation engine. Each agent is pre-trained offline in the digital twin simulation environment. After the composite reward function converges, the trained agent model is deployed to the actual machine for operation. During actual operation, real-world trajectory data is continuously collected, and the parameters of the agent model are updated online in small increments at a preset low learning rate to adapt to the dynamic characteristics of real buildings, forming a closed-loop optimization of offline pre-training and online fine-tuning.

6. A building HVAC energy-saving and safety control system based on hierarchical reinforcement learning, applied to a building HVAC system containing multiple independent variable air volume terminal protection zones, characterized in that... include: The data acquisition module is used to collect global status data and local status data of each defense zone, and perform standardized preprocessing. The upper-level global scheduling module has a built-in water-side upper-level global intelligent agent, which is used to output chilled water supply temperature setting instructions and global air supply temperature setting instructions based on global state data at the first decision frequency. The lower-level local execution module has a built-in wind-side lower-level multi-zone intelligent agent, which is used to output the variable air volume valve opening command of each zone based on the local state data of the corresponding zone and the global air supply temperature setting command at a second decision frequency; wherein, the second decision frequency is higher than the first decision frequency. The safety constraint module is used to perform deterministic physical inverse mapping and safety boundary truncation processing on the chilled water supply temperature setting command, the global air supply temperature setting command, and the variable air volume valve opening command, respectively, to constrain each control command within the physical safety operating range of the corresponding equipment. The reward calculation module is used to calculate a composite reward function to train and optimize each agent. The composite reward function includes at least an energy consumption cost term based on time-of-use electricity pricing, a thermal comfort penalty term based on the predicted average thermal sensation index, and an equipment safety penalty term.

7. The building HVAC energy-saving and safety control system based on hierarchical reinforcement learning according to claim 6, characterized in that, The water-side global agent in the upper-layer global scheduling module is constructed based on the soft actor-critic algorithm and includes a policy network unit, a double-Q value network unit, and a physical inverse mapping unit. The policy network unit is used to receive the global state vector and output a continuous action tensor. The dual-Q value network unit uses two independent value networks and takes the minimum Q value of the two outputs to calculate the target value, so as to suppress the overestimation bias of the value function. The physical inverse mapping unit is used to convert the continuous motion tensor into chilled water supply temperature setpoint and global air supply temperature setpoint through deterministic linear mapping, and to constrain the setpoints within the safe operating range of the equipment.

8. The building HVAC energy-saving and safety control system based on hierarchical reinforcement learning according to claim 6, characterized in that, The wind-side lower-layer multi-zone agent in the lower-layer local execution module is constructed based on the multi-agent near-end policy optimization algorithm and adopts a centralized training and distributed execution architecture. The wind-side lower-layer multi-zone intelligent agent includes multiple local policy network units that share parameters. Each local policy network unit corresponds to a zone and is used to output the wind valve control action of the corresponding zone. During the training phase, a global value network unit is configured to output the global state value based on global state data and local state data of all defense zones, and to guide the parameter update of each local policy network unit in combination with the generalized advantage estimation method. The safety constraint module performs dead zone truncation processing on the valve opening command, forcibly retaining the minimum fresh air exchange opening.

9. The building HVAC energy-saving and safety control system based on hierarchical reinforcement learning according to claim 6, characterized in that, The formula for calculating the composite reward function executed by the reward calculation module is as follows: in, The output value of the system's joint reward at the current time step. This represents the weighting coefficient for the energy-saving and economical items. This represents the actual total power consumption of the chiller, water pumps, and fans within the current time step. This refers to the real-time time-of-use electricity price rate published by the power grid at the current moment. This is the weighting coefficient for the thermal comfort penalty term. Let i be the predicted average thermal sensation index calculated based on the current indoor temperature and humidity for the i-th defense zone. This is a physical penalty item for equipment safety. When the fluctuation of the damper's movement is detected to exceed a preset threshold, the corresponding penalty value is triggered.

10. The building HVAC energy-saving and safety control system based on hierarchical reinforcement learning according to claim 6, characterized in that, It also includes a model iteration deployment module, which comprises a digital twin pre-training unit and a live online fine-tuning unit; The digital twin pre-training unit is used to construct a digital twin simulation environment based on the building energy consumption simulation engine, and to complete the offline pre-training and convergence of each intelligent agent; The real-machine online fine-tuning unit is used to collect real-world trajectory data during real-machine operation and make minor updates to the parameters of the intelligent agent model with a low learning rate to adapt to the dynamic characteristics changes of the real building.

Images