Multi-room negative pressure gradient automatic control method and system based on reinforcement learning

Abstract

The invention belongs to the field of multi-room negative pressure gradient control, and discloses a multi-room negative pressure gradient automatic control method and system based on reinforcement learning. An environment model containing multi-dimensional dynamic parameters such as an environment pressure state, an access control state, a fan air volume and a room coupling relation is constructed, so that the method can directly obtain an optimal control strategy through interactive learning on the premise of not needing an accurate physical model. Through the self-adaptive decision-making capability of reinforcement learning, linkage adjustment of the exhaust air volume and the fresh air volume of each room and the power of a total exhaust fan and a fresh air fan of the whole system is achieved, so that the stability of gradients of a corridor-5Pa, a ward-10Pa, a semi-polluted area-15Pa, a polluted area-20Pa and the like is kept, and it is ensured that the pressure difference of all areas is always within the safety range of-7Pa to-2Pa.

Description

Technical Field

[0001] This invention belongs to the field of multi-room negative pressure gradient control, specifically relating to a multi-room negative pressure gradient automatic control method and system based on reinforcement learning. Background Technology

[0002] Negative pressure wards, isolation wards, laboratories, and infectious disease control facilities commonly employ zoned differential pressure control. This effectively blocks air recirculation and contamination diffusion by creating a gradually decreasing pressure gradient from clean to contaminated areas. Common pressure gradients are -5 Pa or -10 Pa increments, with typical examples including: corridor -5 Pa, ward -10 Pa, semi-contaminated area -15 Pa, and contaminated area -20 Pa. Maintaining pressure differences between zones by adjusting room exhaust air volume, fresh air intake air volume, and the airflow output of main exhaust fans and fresh air units is crucial for ensuring the safety of the isolation environment. However, in actual engineering scenarios, negative pressure systems exhibit the following characteristics:

[0003] (1) Strong coupling of multiple rooms: A floor usually contains more than ten rooms, each of which contains multiple sub-rooms. All rooms share the main exhaust fan and fresh air fan, and their airflow affects each other, with the pressure difference relationship being complexly coupled.

[0004] (2) Frequent dynamic disturbances: Opening and closing of doors, movement of people, air leakage through window / door gaps, aging of room sealing, changes in filter resistance, and decline in fan efficiency can all have a momentary or continuous impact on pressure difference, putting the system in a high-disturbance environment.

[0005] (3) Significant nonlinear characteristics: There is a nonlinear relationship between room pressure difference and air volume. The leakage flow is significantly affected by door opening and room pressure difference, making it difficult to accurately model using traditional linear control methods.

[0006] Under the aforementioned complex conditions, it is extremely difficult to implement a control system that can maintain a stable pressure gradient over a long period of time, automatically adapt to disturbances, and guarantee safety boundaries.

[0007] Current negative pressure wards in medical facilities generally employ a traditional negative pressure maintenance scheme using constant air volume combined with PID (Proportional-Integral-Derivative) hierarchical control. Specifically, this involves configuring independent exhaust fans in contaminated, semi-contaminated, and clean areas, or using a unified exhaust fan with terminal air valves. PID control, based on pressure sensor feedback, performs closed-loop regulation to maintain a target pressure difference between rooms. This approach typically relies on environments with relatively stable room sealing and infrequent personnel movement, maintaining a simple negative pressure gradient structure through a fixed air volume ratio. However, in shared exhaust fan systems involving multiple rooms and interconnected wards, when frequent door opening and closing, reduced sealing, and fluid coupling changes due to room volume differences occur, traditional PID control struggles to respond quickly to global pressure disturbances. It often only corrects single-point pressure locally, failing to coordinate adjustments to fresh air fan power, exhaust volume in each room, and overall exhaust strategy. This can lead to excessive pressure fluctuations and localized positive pressure backflow. Therefore, this scheme is insufficiently adaptable to complex flow coupling scenarios and cannot achieve precise dynamic negative pressure gradient control.

[0008] In recent years, some negative pressure ward control schemes based on numerical models or CFD (Computational Fluid Dynamics) simulations have emerged. These schemes establish fluid models of the airflow distribution in the environment and combine them with simple rule bases or model predictive control (MPC)-based decision logic to predict and regulate pressure in multiple rooms and pipelines. These schemes typically first estimate the airflow and pressure difference relationship using CFD or drag network models, and then use linear or weakly nonlinear model prediction algorithms to perform rolling optimization of valve opening and fan frequency. However, these methods rely on accurate system models, which are costly to build and difficult to update in real time. In real-world environments, factors such as personnel entry and exit, uncertain door opening and closing, decreased room airtightness, and changes in filter resistance cause the model to rapidly deviate from the actual system, leading to a decline in control performance. Especially when all rooms share exhaust fans and fresh air fans, and the pressure in the rooms has a hierarchical gradient requirement and needs to be maintained in real time within a dynamic safety range of -7Pa to -2Pa, the model prediction method is difficult to cover all scenarios and cannot handle multi-objective trade-offs in high-dimensional state spaces. Therefore, it cannot meet the actual need to maintain a stable negative pressure gradient in complex building environments for a long time.

[0009] In summary, the existing pressure control technologies for medical negative pressure wards generally adopt constant air volume control, PID closed-loop regulation, or predictive control methods based on fixed models. These technologies can maintain the basic negative pressure in an environment with simple structure and small disturbances, but there are obvious limitations in complex spaces that include multiple rooms, multiple small compartments, multiple people flow activities, and frequent opening and closing of room doors. First, traditional PID control can only perform local regulation for a single room and cannot handle the fluid coupling relationship caused by sharing exhaust fans and fresh air fans among rooms. Therefore, when the pressure in a certain area is disturbed, significant manual intervention is often required to restore the global gradient. Second, the existing methods based on CFD or linear prediction models rely on highly accurate environmental models and are difficult to adapt to dynamic uncertain factors such as attenuation of room sealing, change of filter resistance, and disturbance of fresh air volume; once the model deviates from the actual situation, it may lead to the failure of global pressure control. Third, these methods often lack a systematic safety protection and steady-state - dynamic switching mechanism and cannot take global linkage regulation in a timely manner when the pressure deviates from the threshold, resulting in the risk of positive pressure in the room for a short time. In addition, the existing solutions generally do not have the ability of autonomous learning, cannot automatically optimize the strategy through long-term operation data, and are also difficult to cope with extreme disturbance scenarios such as multiple people opening doors simultaneously and a small compartment being temporarily closed. Therefore, there are still significant deficiencies in the stability, adaptability, robustness, and intelligence level of the existing technologies. Summary of the Invention

[0010] The purpose of the present invention is to provide a multi-room negative pressure gradient automatic control method and system based on reinforcement learning, aiming to solve the key pain points existing in the control of negative pressure systems, such as slow dynamic response, insensitivity to disturbances, fixed control parameters, difficulty in maintaining global pressure difference optimization, and inability to automatically adapt to changes in room conditions, and to provide an intelligent control method that can learn, adapt, and operate stably in complex scenarios for a long time.

[0011] To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0012] In the first aspect, a multi-room negative pressure gradient automatic control method based on reinforcement learning is provided, including:

[0013] Construct a multi-room air pressure difference dynamics model and discretize it;

[0014] Obtain the pressure differences and door states between adjacent rooms at the current moment, and input the obtained pressure differences and door states into the discretized multi-room air pressure difference dynamics model to obtain the predicted values of the pressure differences between adjacent rooms at the next moment;

[0015] The pressure difference and door status between adjacent rooms at the current moment, the exhaust air volume and fresh air volume of each room at the current moment, and the total exhaust air volume and total fresh air volume of all rooms at the current moment are used as the state input of the reinforcement learning algorithm at the current moment to obtain the action at the current moment. The action includes the exhaust air increment and fresh air increment of each room, as well as the total exhaust air increment and total fresh air increment of all rooms.

[0016] The reward value for the current moment is calculated based on the predicted pressure difference between adjacent rooms at the next moment and the action at the current moment. The loss function is then calculated based on the reward value, and the reinforcement learning algorithm is updated.

[0017] The training process is iterated until completion. The reinforcement learning algorithm after training is used to output real-time actions based on the real-time collected data, thereby realizing automatic control of negative pressure gradient in multiple rooms.

[0018] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0019] Preferably, the discretized multi-room air pressure difference dynamics model is expressed as follows:

[0020]

[0021] In the formula, for Time Room and room Predicted pressure difference between them for Time Room and room The pressure difference measurement between them For discrete time steps, air density, For the room volume, For the room The set of adjacent rooms, For the room and room The coupling coefficient between them for Time Room and room The pressure difference measurement between them for Time Room Fresh air volume, for Time Room The exhaust volume, for Time Room The amount of air leakage, For the room volume, For the room The set of adjacent rooms, For the room and room The coupling coefficient between them for Time Room and room The pressure difference measurement between them for Time Room Fresh air volume, for Time Room The exhaust volume, for Time Room The amount of air leakage.

[0022] Preferably, the process for obtaining the air leakage is as follows:

[0023] Get the set of adjacent rooms of the current room;

[0024] Take an adjacent room and calculate the pressure difference between the current room and the adjacent room. The power of the power is multiplied by the air leakage coefficient between the current room and the adjacent rooms, as well as the door opening coefficient between the current room and the adjacent rooms, to obtain the air leakage amount corresponding to the current adjacent room;

[0025] The air leakage of the current room is obtained by summing the air leakage of all adjacent rooms.

[0026] Preferably, the door opening coefficient is set as follows:

[0027] If the door status sensor between adjacent rooms outputs a closed door status, the door opening coefficient is set to 0.01; if the door status sensor between adjacent rooms outputs an open door status, the door opening coefficient is set to 1.

[0028] Preferably, the constraints of the multi-room air pressure difference dynamics model are set as follows:

[0029] The sum of the exhaust volumes of all rooms is less than or equal to the maximum total exhaust volume of all rooms;

[0030] The sum of the fresh air volume of all rooms is less than or equal to the maximum total fresh air volume of all rooms.

[0031] Preferably, the reinforcement learning algorithm includes a policy network and two independent value networks. During iterative training, the minimum Q-value in the two independent value networks is taken as the final Q-value.

[0032] Preferably, the step of calculating the reward value for the current moment based on the predicted pressure difference between adjacent rooms at the next moment and the action at the current moment includes:

[0033] Calculate the pressure difference reward function: For two adjacent rooms, take the absolute value of the difference between the predicted pressure difference between the rooms and the target pressure difference, take the square of the larger of the absolute value of the difference, the difference between the deviation threshold and 0, and then weight them according to the weights between the rooms. Sum the weighted values ​​corresponding to all adjacent rooms and take the negative value as the pressure difference reward value.

[0034] Calculate the smooth action reward function: For each room, calculate the square of the difference between the action vector at the current moment and the action vector at the previous moment. Sum the squared values ​​for all rooms, multiply them by the action penalty weight coefficient, and take the negative value as the smooth action reward value.

[0035] Calculate the safety penalty: Based on the predicted pressure difference between each adjacent room at the next moment, count the number of predicted pressure differences that exceed the safe pressure difference range, multiply the number by the safety penalty coefficient and take the negative value as the safety penalty reward value;

[0036] The final reward value is the sum of the differential pressure reward value, the smoothing action reward value, and the safety penalty reward value.

[0037] Secondly, a multi-room negative pressure gradient automatic control system based on reinforcement learning is provided, including a sensor unit, a data acquisition and controller, and an actuator unit;

[0038] The sensor unit is used to monitor the pressure difference between adjacent rooms and the door status;

[0039] The data acquisition and controller is used to construct and discretize a multi-room air pressure difference dynamic model; acquire the pressure difference and door status between adjacent rooms at the current moment, input the acquired pressure difference and door status into the discretized multi-room air pressure difference dynamic model, and obtain the predicted pressure difference between adjacent rooms at the next moment; use the pressure difference and door status between adjacent rooms at the current moment, as well as the exhaust air volume and fresh air volume of each room at the current moment, and the total exhaust air volume and total fresh air volume of all rooms at the current moment as the current moment state of the reinforcement learning algorithm, and obtain the action at the current moment. The action includes the exhaust air increment and fresh air increment of each room, as well as the total exhaust air increment and total fresh air increment of all rooms; calculate the reward value at the current moment based on the predicted pressure difference between adjacent rooms at the next moment and the action at the current moment, calculate the loss function based on the reward value and update the reinforcement learning algorithm; iterate training until the end, and use the trained reinforcement learning algorithm to output real-time actions based on the real-time acquired state, and convert the actions into control output values ​​of the actuator unit;

[0040] The actuator unit is used to execute according to the data acquisition and the control quantity output by the controller to realize automatic control of negative pressure gradient in multiple rooms.

[0041] This invention addresses the shortcomings of existing technologies in complex multi-room negative pressure gradient control scenarios by proposing a reinforcement learning-based automatic control method and system for multi-room negative pressure gradients. The invention constructs an environmental model incorporating multi-dimensional dynamic parameters such as environmental pressure state, access control status, fan airflow, and room coupling relationships. This allows the method to directly obtain the optimal control strategy through interactive learning without requiring a precise physical model. Through the adaptive decision-making capability of reinforcement learning, the method achieves coordinated adjustment of exhaust airflow, fresh airflow, and the power of the total exhaust and fresh air fans in each room, thereby maintaining the stability of pressure gradients such as -5Pa in corridors, -10Pa in wards, -15Pa in semi-contaminated areas, and -20Pa in contaminated areas, and ensuring that the pressure difference in all areas remains within the safe range of -7Pa to -2Pa. Furthermore, by incorporating safety constraints, reward function pruning, and pressure deviation penalty mechanisms, the invention enhances the system's robustness under conditions of frequent door opening and closing, changes in sealing, and even extreme disturbances. This enables the negative pressure system to operate automatically and stably for extended periods, reducing the need for manual adjustments and improving the biosafety level of hospitals, laboratories, and other similar facilities. Attached Figure Description

[0042] Figure 1 This is an example diagram of the invention deployed in a hospital negative pressure ward;

[0043] Figure 2 This is a flowchart of the multi-room negative pressure gradient automatic control method based on reinforcement learning according to the present invention;

[0044] Figure 3This is a schematic diagram illustrating the change of negative pressure over time in the buffer room, ward, and toilet during the simulation experiment of this invention.

[0045] Figure 4 This is a schematic diagram showing the change in the opening degree of the valves of the buffer room exhaust fan, the ward exhaust fan, the toilet exhaust fan, and the ward fresh air fan over time in the simulation experiment of this invention.

[0046] Figure 5 This is a schematic diagram showing the changes in exhaust fan power and fresh air fan power over time in the simulation experiment of this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0049] Existing negative pressure wards and multi-room medical negative pressure systems generally rely on constant air volume (CAV) control, constant air volume valves, or PID-based local regulation strategies. These methods can only maintain a relatively stable pressure differential in single-room or weakly coupled scenarios. However, in actual engineering, a single floor typically contains dozens of rooms with different functions and pressure differential levels. These rooms are highly coupled to the main air system through door gaps, connecting openings, corridors, and other means, resulting in highly nonlinear, multivariable, and multi-disturbance pressure differentials. Furthermore, frequent door opening and closing, transient airflow impacts caused by personnel movement, leakage uncertainties due to aging room seals, and fan efficiency degradation continuously alter the pressure differential relationships between rooms, making it difficult to maintain a stable negative pressure gradient using conventional control methods. Traditional CAV strategies cannot respond to these disturbances in a timely manner; while PID-based methods rely on accurate models and the independence of controllable objects, making dynamic coordination impossible in multi-room coupled systems.

[0050] Therefore, this embodiment proposes a multi-room negative pressure gradient automatic control method and system based on reinforcement learning, including system hardware structure, environmental state modeling method, control strategy network structure, reward function design, safety constraint mechanism, and online / offline training process. This invention mainly addresses how to achieve adaptive, coordinated, and real-time optimized control of exhaust air volume, fresh air volume, and main fan output in complex spaces with strong coupling of multiple rooms, channels, and fans, so that the entire area can maintain a stable and safe negative pressure gradient (typically a -5 Pa difference, with the pressure difference between rooms maintained within the range of -7 Pa to -2 Pa) even under uncertain disturbances. This invention can be deployed in application scenarios requiring the maintenance of multi-area pressure gradients, such as hospital negative pressure wards, isolation wards, laboratories, and microbiological operation areas. Figure 1 The image shows an example of the present invention deployed in a hospital negative pressure ward. In this example, it is necessary to ensure that the pressure difference between the toilet and the ward is maintained at -5Pa, the pressure difference between the ward and the buffer room is maintained at -5Pa, and the pressure difference between the buffer room and the medical staff corridor is maintained at -5Pa.

[0051] like Figure 2 As shown, this embodiment provides a multi-room negative pressure gradient automatic control method based on reinforcement learning, which specifically includes the following steps. It should be noted that the room in this invention should be understood as a space, including not only toilets and wards, but also spaces such as corridors.

[0052] Step 1: Construct a multi-room air pressure difference dynamic model and discretize it.

[0053] To train reinforcement learning algorithms in simulations, this embodiment establishes a multi-room air pressure differential dynamics model, involving room gas balance model, leakage model, door opening model, and fan / valve model, expressed by the following formulas:

[0054]

[0055] In the formula, Indicates room and room Pressure difference between rate of change over time Indicates room air pressure The rate of change of (pressure Pa) over time Indicates room air pressure rate of change over time For the room and room The pressure difference measurement between them For the room Fresh air volume, For the room The exhaust volume, For the room The amount of air leakage, For the room and room The pressure difference measurement between them For the room Fresh air volume, For the room The exhaust volume, For the room The amount of air leakage.

[0056] Furthermore, the constraints of the multi-room air pressure differential dynamics model are set as follows: the sum of the exhaust volumes of all rooms is less than or equal to the maximum total exhaust volume of all rooms, i.e. , The total number of rooms. This is the maximum total exhaust air volume for all rooms. The sum of the fresh air volumes for all rooms is less than or equal to the maximum total fresh air volume for all rooms. , This represents the maximum total fresh air volume for all rooms.

[0057] Discretization using the Euler method:

[0058]

[0059] Substituting the above equation into the continuous-time equation, we obtain the discretized multi-room air pressure difference dynamic model, as follows:

[0060]

[0061] In the formula, for Time Room and room (for the room) Predicted pressure difference between adjacent rooms. for Time Room and room The pressure difference measurement between them denoted as discrete time step (s). air density ( ), For the room volume ( ), For the room The set of adjacent rooms, For the room and room The coupling coefficient between them for Time Room and room The pressure difference measurement between them for Time Room Fresh air volume, for Time Room The exhaust volume, for Time Room The amount of air leakage, For the room volume, For the room The set of adjacent rooms, For the room and room The coupling coefficient between them for Time Room and room The pressure difference measurement between them for Time Room Fresh air volume, for Time Room The exhaust volume, for Time Room The amount of air leakage.

[0062] Among them, rooms For example, calculate the fresh air volume, exhaust air volume, and air leakage volume ( ). )as follows:

[0063] Fresh air volume: Room at the current moment The fresh air volume for the room The formula is: Maximum fresh air volume multiplied by the current fresh air opening. , For the room Maximum fresh air volume express Time Room The fresh air opening is obtained by dividing the required fresh air volume by the maximum fresh air volume.

[0064] Exhaust volume: Room at the current moment The exhaust volume is for the room The maximum exhaust volume multiplied by the exhaust coefficient at the current moment is expressed by the formula: , For the room Maximum exhaust volume, express Time Room The exhaust opening is obtained by dividing the required exhaust volume by the maximum exhaust volume.

[0065] Air leakage: Get the set of neighboring rooms of the current room; take one neighboring room and calculate the pressure difference between the current room and the neighboring room. The air leakage coefficient of the current room is multiplied by the power of ...

[0066]

[0067] In the formula, For the room and room The air leakage coefficient between them This is the flow index, typically ranging from 0.5 to 0.67. for Time Room and room The door opening coefficient between adjacent rooms is set to a value between 0 and 1 (inclusive). In this embodiment, the door opening coefficient is set as follows: if the door status sensor between adjacent rooms outputs a closed door status, the door opening coefficient is set to 0.01; if the door status sensor between adjacent rooms outputs an open door status, the door opening coefficient is set to 1. The formula is expressed as:

[0068]

[0069] In the formula, For the room and room The door status sensor between Output at any moment This indicates that the output gate is closed. This indicates that the output gate is open.

[0070] Step 2: Obtain the pressure difference and door status between adjacent rooms at the current moment. Input the obtained pressure difference, door status, fresh air volume and exhaust air volume of the room into the discretized multi-room air pressure difference dynamic model to obtain the predicted pressure difference between adjacent rooms at the next moment.

[0071] Step 3: Use the pressure difference and door status between adjacent rooms at the current moment, the exhaust air volume and fresh air volume of each room at the current moment, and the total exhaust air volume and total fresh air volume of all rooms at the current moment as the state input of the reinforcement learning algorithm at the current moment to obtain the action at the current moment. The action includes the exhaust air increment and fresh air increment of each room, as well as the total exhaust air increment and total fresh air increment of all rooms.

[0072] In the reinforcement learning control strategy of this embodiment, the state vector contains all the key quantities affecting negative pressure control, including:

[0073]

[0074]

[0075] The action vector is:

[0076]

[0077]

[0078] In the formula, for The state input of the reinforcement learning algorithm is constantly being updated. for Time Room state, express Time Room Pressure difference between the room and all adjacent rooms express Time Room Door opening coefficient between all adjacent rooms for Total fresh air volume in all rooms at any given time. for Total exhaust volume of all rooms at any given time. for The actions output by the reinforcement learning algorithm are constantly reinforced. for Time Room The action, for Time Room The increase in fresh air volume, for Time Room Increase in exhaust volume for The total increase in fresh air volume in all rooms at any given time. for The total exhaust volume increase of all rooms at any given time.

[0079] Specifically, the reinforcement learning algorithm in this embodiment employs a Soft Actor-Critic (SAC) network.

[0080] The policy network (Actor network) generates continuous control actions for each room, outputting fresh air volume, exhaust air volume, and total fan power adjustment values. It maximizes long-term reward at each time step to ensure the negative pressure gradient meets design requirements. The network structure of the policy network used in this embodiment includes an input layer (dimension equal to the state vector length), three fully connected hidden layers (256 neurons per layer), a ReLU activation function, and an output layer (dimension equal to the action vector dimension). The output activation function uses the Tanh activation function to constrain actions within allowable ranges. The input to the policy network is... The output is the probability parameters of the continuous action space:

[0081]

[0082] In the formula, Indicates the policy in the state Down-output continuous action The probability distribution, The action distribution parameters output by the policy network ( Gaussian distribution).

[0083] Among them, the value network (Critic network) is used to analyze a given state. and actions Value assessment is performed, which evaluates the long-term cumulative reward of the output action in the current state. This embodiment employs a dual-Q network to reduce overestimation bias and improve training stability. The network structure specifically includes an input layer (state and action concatenation). The system consists of three fully connected hidden layers (3 layers, 256 neurons per layer), a ReLU activation function, and an output layer (outputting a single real Q-value). During training, the Q-value may be overestimated, causing the policy network to learn incorrect policies. Therefore, this embodiment uses two independent value networks, taking two Q-values ​​for each update. , Minimum value:

[0084]

[0085] In the formula, yes The target Q value at time t. yes Momentary rewards It is a discount factor. These are the Q values ​​output by two independent value networks. It is the entropy coefficient. It is a strategy In state Select action The logarithmic probability.

[0086] Step 4: Calculate the reward value for the current time based on the predicted pressure difference between adjacent rooms at the next time step and the action at the current time step. Calculate the loss function based on the reward value and update the reinforcement learning algorithm. In this embodiment, the reward value consists of three parts, as follows:

[0087] (1) Pressure difference reward function: For two adjacent rooms, take the absolute value of the difference between the predicted pressure difference and the target pressure difference, take the square of the larger of the absolute value of the difference, the difference between the deviation threshold and 0, and weight it according to the weight between the rooms. Sum the weighted values ​​corresponding to all adjacent rooms and take the negative value as the pressure difference reward value; the formula is expressed as follows:

[0088]

[0089] In the formula, for Pressure difference reward value at any given moment. For the room and room The weights between them For the room and room The target pressure difference between them The deviation threshold (Pa) is used; if the pressure difference is less than the deviation threshold, there is no penalty. When the pressure difference deviates from the target, the reward is reduced.

[0090] (2) Smoothing action reward function: For each room, calculate the square of the difference between the action vector at the current time and the action vector at the previous time. Sum the squared values ​​for all rooms, multiply them by the action penalty weight coefficient, and take the negative value as the smoothing action reward value; the formula is expressed as follows:

[0091]

[0092] In the formula, for The reward value for smooth motion at any given moment. This is the weighting coefficient for the action penalty. for Time Room The punishment involves significant changes in movement, encouraging smoother, more fluid movements.

[0093] (3) Safety penalty item: Based on the predicted pressure difference between adjacent rooms at the next moment, count the number of predicted pressure differences that exceed the safe pressure difference range, multiply the number by the safety penalty coefficient and take the negative value as the safety penalty reward value; the formula is expressed as follows:

[0094]

[0095] In the formula, for The security penalty reward value at any time, As a safety penalty coefficient, This represents the lower limit (Pa) of the safe differential pressure range. This represents the upper limit of the safe range for differential pressure. The greater the differential pressure exceeds the safe range, the greater the penalty, forcing the model to take safe actions.

[0096] (4) The sum of the differential pressure reward value, the smoothing action reward value, and the safety penalty reward value is taken as the final reward value; the formula is expressed as follows:

[0097]

[0098] In the formula, for The final reward value at any given moment is determined by the reinforcement learning model, which maximizes the long-term cumulative reward. A higher reward indicates that the pressure difference is close to the target.

[0099] The specific process of calculating the loss function and updating the reinforcement learning algorithm based on the reward value in this embodiment is as follows:

[0100] (1) Initialization: Initialize the Actor network parameters, initialize the dual Critic network parameters, initialize the experience replay pool, and set the training hyperparameters: the Actor network learning rate is The learning rate of the Critic network is Discount factor The soft update coefficient is 0.99. The entropy coefficient is 0.005. For automatic adjustment or 0.2, batch size is 256, and experience playback pool capacity is... The maximum number of training steps is 100k-500k steps.

[0101] (2) Training cycle (each time step) ): Get the current state Actor network output , Constructing Gaussian distribution sampling action After executing the action, the simulation environment returns to the next state. and rewards , experience The data is stored in the experience replay pool, and random samples are taken from the experience replay pool to update the SAC network.

[0102] When updating the Critic network, a dual-Q network is used, with the following objective:

[0103]

[0104] Minimize the mean square error using the Critic network:

[0105]

[0106] When updating the Actor network, the loss function is calculated by maximizing the expected cumulative reward and using entropy regularization.

[0107]

[0108] To ensure stability, the target network parameters are updated softly.

[0109]

[0110] In the formula, It is a soft update coefficient.

[0111] Step 5: Iterate training until completion. Utilize the trained reinforcement learning algorithm to output real-time actions based on the real-time state, achieving automatic control of negative pressure gradients in multiple rooms. After obtaining the action output, for each room, based on... and Calculate the fresh air volume at the next moment. Then based on the maximum fresh air volume The fresh air opening degree at the next moment is calculated. Similarly, the exhaust ventilation opening at the next moment can be calculated. As the control variable for the next moment, it controls the valves of the fresh air unit and the exhaust fan to achieve multi-room negative pressure gradient control.

[0112] To verify the robustness of the control strategy of this invention to various disturbances, random tests were conducted. These included: random changes in door status (simulating air leakage caused by door opening and closing); room volume or airtightness disturbances (simulating modeling errors); and fresh air / exhaust fan power disturbances (simulating fan performance fluctuations). The simulation results are as follows: Figures 3-5 As shown, the dynamic pressure response process of the negative pressure ward system under steady-state operation and door opening disturbance conditions is demonstrated. Figure 3 This indicates the change in negative pressure in the buffer room, ward, and toilet over time. Figure 4 This indicates the change in the opening degree of the buffer exhaust valve, ward exhaust valve, toilet exhaust valve, and ward fresh air valve over time. Figure 5 This indicates the change in exhaust fan power and fresh air fan power over time.

[0113] As shown in the figure, in the initial stage of the simulation, the pressure in the buffer zone, wards, and toilets remained stable near the set negative pressure target value, consistently maintaining a gradient relationship where the negative pressure in the toilets was greater than that in the wards, and the negative pressure in the wards was greater than that in the buffer zone, indicating that the system was in a normal negative pressure control state. After the toilet door opening event was introduced, the area pressure fluctuated significantly, the local negative pressure level temporarily decreased, and the pressure gradient was disrupted, reflecting the instantaneous air leakage and airflow disturbance caused by the door opening. At the same time, the system control logic drove the exhaust valves and exhaust fan power to change constantly to keep the negative pressure gradient within a reasonable range. As the toilet door opening event ended, the negative pressure control system, under the coordinated adjustment of the fan and valves, gradually restored the pressure in each area to the target negative pressure range, and no pressure instability or gradient reversal occurred during the recovery process, demonstrating the system's good stability and self-recovery capability under typical disturbance conditions.

[0114] After multiple simulation tests, the statistical negative pressure gradient remained within a safe range (-7Pa to -2Pa). Random test results verified that the invention has high stability in actual operation, ensuring safety and preventing the gradient target from being destroyed.

[0115] In another embodiment, a multi-room negative pressure gradient automatic control system based on reinforcement learning is provided, including a sensor unit, a data acquisition and controller, and an actuator unit.

[0116] The sensor unit includes a differential pressure sensor with a sampling frequency of 1–5 Hz, measuring the pressure difference between adjacent spaces, and a door status sensor, measuring the door opening degree (0 for closed, 1 for fully open). The sensor unit is used to monitor the pressure difference and door status between adjacent rooms.

[0117] The data acquisition and controller includes a reinforcement learning module, a low-level precision control module (DDC), a database, and a front-end display interface. It is used to construct and discretize a multi-room air pressure differential dynamics model; collect the pressure differential and door status between adjacent rooms at the current moment; input the collected pressure differential and door status into the discretized multi-room air pressure differential dynamics model to obtain the predicted pressure differential between adjacent rooms at the next moment; use the pressure differential and door status between adjacent rooms at the current moment, as well as the exhaust and fresh air volumes of each room at the current moment, and the total exhaust and fresh air volumes of all rooms at the current moment, as the current moment's state for the reinforcement learning algorithm to obtain the current moment's action. The action includes the exhaust and fresh air increments of each room, and the total exhaust and fresh air increments of all rooms; calculate the reward value for the current moment based on the predicted pressure differential between adjacent rooms at the next moment and the current moment's action; calculate the loss function based on the reward value and update the reinforcement learning algorithm; iterate training until completion; use the trained reinforcement learning algorithm to output real-time actions based on the real-time state, and convert the actions into control output values ​​for the actuator unit;

[0118] The actuator unit includes room exhaust valves, room fresh air valves, exhaust fans, and fresh air fans. All actuators use continuously adjustable setpoints from 0 to 100%, with a control cycle of 5 to 10 seconds. The actuator unit is used to execute commands based on data acquisition and the control output from the controller, achieving automatic control of negative pressure gradients in multiple rooms.

[0119] To ensure data interaction, this embodiment also includes a communication layer, including necessary connections such as switches and network cables.

[0120] For specific limitations on the reinforcement learning-based multi-room negative pressure gradient automatic control system, please refer to the limitations on the reinforcement learning-based multi-room negative pressure gradient automatic control method mentioned above, which will not be repeated here.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for automatic control of multi-room negative pressure gradient based on reinforcement learning, characterized in that, The multi-room negative pressure gradient automatic control method based on reinforcement learning comprises the following steps: A multi-room air pressure difference dynamics model is constructed and discretized; The pressure difference and door state between each adjacent room at the current time are obtained, and the obtained pressure difference and door state are input into the discretized multi-room air pressure difference dynamics model to obtain the pressure difference prediction value between each adjacent room at the next time; The pressure difference and door state between each adjacent room at the current time, the exhaust and fresh air volume of each room at the current time, and the total exhaust and fresh air volume of all rooms at the current time are taken as the state input of the reinforcement learning algorithm at the current time to obtain the action at the current time, wherein the action comprises the exhaust increment and fresh air increment of each room, and the total exhaust increment and fresh air increment of all rooms; The reward value at the current time is calculated according to the pressure difference prediction value between each adjacent room at the next time and the action at the current time, the loss function is calculated based on the reward value, and the reinforcement learning algorithm is updated; Iterative training is performed until the end, and the trained reinforcement learning algorithm is used to output real-time action according to the real-time collected state to realize multi-room negative pressure gradient automatic control.

2. The method of claim 1, wherein, The discretized multi-room air pressure difference dynamics model is represented as follows: wherein is the predicted value of the pressure difference between room and room at time instant is the measured value of the pressure difference between room and room at time instant is the discrete time step, is the air density, is the volume of room , is the set of adjacent rooms of room , is the coupling coefficient between room and room , is the measured value of the pressure difference between room and room at time instant , is the fresh air rate of room at time instant , is the exhaust air rate of room at time instant , is the air leakage rate of room at time instant , is the volume of room , is the set of adjacent rooms of room , is the coupling coefficient between room and room , is the measured value of the pressure difference between room and room at time instant , is the fresh air rate of room at time instant , is the exhaust air rate of room at time instant , is the air leakage rate of room at time instant .

3. The method of claim 2, wherein, The air leakage amount acquisition process is as follows: A set of adjacent rooms of the current room is obtained; Taking an adjacent room, multiplying the pressure difference between the current room and the adjacent room by the power of 0.5, the air leakage coefficient between the current room and the adjacent room, and the door opening degree coefficient between the current room and the adjacent room, as the air leakage amount corresponding to the current adjacent room ​ The air leakage amounts corresponding to all adjacent rooms are added to obtain the air leakage amount of the current room.

4. The method of claim 3, wherein, The door opening degree coefficient is set as follows: If the door state output by the door state sensor between adjacent rooms is closed, the door opening degree coefficient is set to 0.01; if the door state output by the door state sensor between adjacent rooms is open, the door opening degree coefficient is set to 1.

5. The method of claim 1, wherein, The constraints of the multi-room air pressure difference dynamics model are set as follows: The sum of the exhaust volumes of all rooms is less than or equal to the maximum total exhaust volume of all rooms; The sum of the fresh air volumes of all rooms is less than or equal to the maximum total fresh air volume of all rooms.

6. The method of claim 1, wherein, The reinforcement learning algorithm comprises a policy network and two independent value networks, and during iterative training, the minimum Q value in the two independent value networks is taken as the final Q value.

7. The method of claim 1, wherein, The calculation of the reward value at the current time according to the pressure difference prediction value between each adjacent room at the next time and the action at the current time comprises the following steps: A pressure difference reward function is calculated: for two adjacent rooms, the absolute value of the difference between the pressure difference prediction value and the target pressure difference is taken, the square of the larger one of the difference and the difference between the absolute value and the deviation threshold value is taken, and the weighted value is obtained by weighting the room weight, the sum of the weighted values of all adjacent rooms is taken, and the negative value is taken as the pressure difference reward value; A smooth action reward function is calculated: for each room, the square of the difference between the action vector at the current time and the action vector at the last time is calculated, the sum of the square values of all rooms is multiplied by the action penalty weight coefficient and taken as the negative value, and the negative value is taken as the smooth action reward value; A safety penalty term is calculated: based on the pressure difference prediction value between each adjacent room at the next time, the number of pressure difference prediction values exceeding the pressure difference safety range is counted, the number is multiplied by the safety penalty coefficient and taken as the negative value, and the negative value is taken as the safety penalty reward value; The sum of the pressure difference reward value, the smooth action reward value and the safety penalty reward value is taken as the final reward value.

8. A multi-room negative pressure gradient automatic control system based on reinforcement learning, characterized in that, The system comprises a sensor unit, a data acquisition and controller, and an actuator unit. The sensor unit is configured to monitor the pressure difference between adjacent rooms and the door state. The data acquisition and controller is configured to construct a multi-room air pressure difference dynamics model and discretize the model; acquire the pressure difference between adjacent rooms and the door state at the current time, input the acquired pressure difference and door state into the discretized multi-room air pressure difference dynamics model to obtain the pressure difference prediction value between adjacent rooms at the next time; take the pressure difference between adjacent rooms and the door state at the current time, the exhaust air volume and fresh air volume of each room at the current time, and the total exhaust air volume and total fresh air volume of all rooms at the current time as the state of the reinforcement learning algorithm at the current time, obtain the action at the current time, the action comprising the exhaust air increment and fresh air increment of each room, and the total exhaust air increment and total fresh air increment of all rooms; calculate the reward value at the current time according to the pressure difference prediction value between adjacent rooms at the next time and the action at the current time, calculate the loss function based on the reward value, and update the reinforcement learning algorithm; iteratively train until the end, use the trained reinforcement learning algorithm to output the real-time action according to the real-time acquired state, and convert the action into a control amount output value to execute the actuator unit; The actuator unit is configured to execute according to the control amount output by the data acquisition and controller to realize multi-room negative pressure gradient automatic control.

Images