A Distributed Hydraulic Autonomous Control System and Method for Centralized Heating Networks Based on Multi-Agent Reinforcement Learning

By deploying a distributed hydraulic autonomous control system based on multi-agent reinforcement learning in a centralized heating system, the problems of computational dimensionality curse and communication delay in the centralized scheduling mode are solved, and the entire network achieves second-level self-balancing when the central system loses contact, ensuring the safety and stability of the heating system.

CN123113369APending Publication Date: 2026-11-17HUANENG CLEAN ENERGY RES INST +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611446996.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-17
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing centralized heating systems suffer from problems such as the curse of computational dimensionality, large communication delays, and high risk of single-point collapse under centralized scheduling, as well as the lack of horizontal coordination between nodes in traditional distributed PID control and the inability of the entire network to achieve autonomous balance when the central control is lost.

Method used

A distributed hydraulic autonomous control system based on multi-agent reinforcement learning is adopted. By deploying load-end autonomous control agents and network-end collaborative topology agents at the user end and heat exchange station, local information interaction is carried out using a distributed peer-to-peer (P2P) communication network, and a deep Actor neural network is mounted on the edge agents. Combined with physical security heuristic fence operators and the MAPPO algorithm, decentralized global hydraulic balance control is achieved.

Benefits of technology

When the central cloud loses connection or malfunctions, the system can reconstruct the entire network's safe hydraulic balance steady state within seconds, reducing the dimensions of the state space and action space, reducing long-distance communication delays and cloud computing pressure, and ensuring the safety and stability of the heating system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113369A_ABST
    Figure CN123113369A_ABST
Patent Text Reader

Abstract

This invention belongs to the field of intelligent control of centralized heating, and discloses a distributed hydraulic autonomous control system and method for centralized heating networks based on multi-agent reinforcement learning. The system includes a load-end autonomous control agent, a network-end collaborative topology agent, a distributed peer-to-peer (P2P) communication network, and a physical security heuristic fence operator. The system is divided into local sub-grids according to the physical network topology. Edge agents only interact with first-order neighbors, and output valve opening corrections through the MAPPO algorithm. The reward function includes a penalty for balancing neighbor heating quality. The fence operator performs hydraulic security checks on the actions. When the central cloud loses connection, decentralized autonomy is switched, and near-end agents perform proportional collaborative step-down, achieving second-level self-balancing of the entire network under network outage conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology for centralized heating networks, specifically relating to a distributed hydraulic autonomous control system and method for centralized heating networks based on multi-agent reinforcement learning. Background Technology

[0002] As modern urban centralized heating networks evolve towards ultra-large scale, long distances, and multiple branches, the hydraulic conditions of heating pipelines are becoming increasingly complex. Traditional smart heating systems mainly rely on a centralized scheduling model of "cloud-based central computing power," where a central server collects the operating status of tens of thousands of electric valves and pump stations across the network, centrally solves the global hydraulic balance strategy, and then issues it for execution. However, this centralized scheduling model faces three major control bottlenecks that are difficult to overcome in actual operation: First, when dealing with tens of thousands of electric valves and pumping stations across the network, the dimensions of the state space and action space of centralized computing grow exponentially with the number of nodes, causing a severe dimensionality curse and making it difficult for cloud computing power to meet the real-time solution requirements. At the same time, long-distance communication delays can easily cause overshooting and violent oscillations in the hydraulic system of the pipeline network, making it difficult to achieve second-level real-time control. Secondly, centralized systems have an extremely high risk of single-point failure. Once the central server suddenly fails or the city's backbone communication network is interrupted, the entire heating network will instantly lose its regulation capacity and face the danger of large-scale hydraulic paralysis, which is unacceptable for residential heating systems. Finally, traditional distributed PID control with decentralized authority suffers from blindness and fragmentation. Due to the strong global coupling characteristics of pipeline hydraulics, local adjustments can change the flow and pressure difference of adjacent nodes, easily leading to frequent conflicts and control chases among neighboring nodes. It lacks the ability for horizontal coordination between nodes and makes it difficult to achieve global hydraulic balance.

[0003] In recent years, some studies have attempted to introduce multi-agent reinforcement learning into the field of heating control. For example, patent document CN122590339A discloses a control method for a multi-split water-loop heat pump based on heating engineering. Its upper layer uses model predictive control to provide the globally optimal water temperature and pump frequency trajectory, while the lower layer deploys multiple agents, each employing a deep reinforcement learning model. This method uses a "centralized training, distributed execution" framework for policy learning and online decision-making. However, the lower-layer agents in this scheme aim to track the optimal trajectory provided by the upper-layer MPC, essentially remaining a hierarchical follower control and failing to achieve fully decentralized horizontal collaboration. Furthermore, it lacks a self-healing control mechanism for extreme scenarios such as central disconnection.

[0004] Therefore, how to construct a distributed hydraulic autonomous control system that can decouple complex hydraulic models in a decentralized manner and achieve self-balancing of the entire network within seconds even under extreme conditions of cloud communication interruption is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a distributed hydraulic autonomous control system and method for centralized heating networks based on multi-agent reinforcement learning. This invention solves the problems of computational dimensionality curse, large communication delay, and high risk of single-point collapse under the centralized scheduling mode of existing centralized heating systems, as well as the lack of horizontal coordination between nodes and the inability of the entire network to achieve autonomous balance when the central system is disconnected, which are problems of traditional distributed PID control.

[0006] This invention is achieved through the following technical solution: This invention discloses a distributed hydraulic autonomous control system for centralized heating networks based on multi-agent reinforcement learning, comprising: Multiple autonomous control agents are deployed on the edge controller of the user-end intelligent regulating valve; Multiple network-end collaborative topology intelligent agents are deployed at heat exchange stations or main line branch valves to coordinate the autonomous control intelligent agents at the load ends under their jurisdiction; Both the load-side autonomous control agent and the network-side collaborative topology agent constitute edge agents; The distributed peer-to-peer (P2P) communication network is configured to support point-to-point local information exchange between adjacent edge agents. A physical safety heuristic fence operator is embedded in the execution front end of each edge agent and is used to perform hydraulic safety boundary verification and mandatory constraints on the action after the reinforcement learning action is output and before the action of the execution mechanism. Each of the aforementioned edge agents is configured as follows: The entire network is divided into multiple local sub-grids according to the physical network topology. Each edge agent only interacts with the first-order neighbor nodes that are directly adjacent to it in the physical network topology and constructs a local state vector containing the hydraulic characteristics of its own node and the hydraulic characteristics of its first-order neighbor nodes on the spot. Each edge agent is equipped with an independent deep Actor neural network. Through the multi-agent proximal strategy optimization MAPPO algorithm, it independently outputs valve opening correction step actions. The reward function of the MAPPO algorithm includes a flow deviation penalty term and a neighbor heating quality balance penalty term. The neighbor heating quality balance penalty term is used to reduce the reward of this node when it takes water grabbing action, which causes the neighbor's heating quality to drop, so as to drive the edge agents of the entire network to converge to the Nash cooperative strategy. The system is also configured to perform decentralized autonomy when the central cloud is disconnected: when the edge controller determines that the central cloud is disconnected, it switches to a global decentralized hydraulic autonomy operation mode; when the source end or main line failure causes the total water supply flow or hydraulic head to decrease, the near-end edge agent transmits the attenuation characteristics through the P2P network and performs proportional collaborative step-down control, actively reducing its own valve opening to increase local resistance, and transferring the power head and surplus flow to the remote node, so that the entire network can be reconstructed into a safe hydraulic balance steady state within a few seconds.

[0007] Furthermore, the load-side autonomous control agent is deployed on the edge microprocessor of the individual household intelligent regulating valve and / or the building entrance intelligent regulating valve; the network-side collaborative topology agent is deployed on the controller of the centralized heat exchange station and / or the main line three-way electrically controlled valve. The local state vector is: ; in: , for the first Real-time transient pressure difference across the electric regulating valve at each node; The real-time mass flow rate of the ultrasonic waves flowing through this control node; This represents the current percentage of the valve's actual opening. , is a spliced ​​matrix of pressure difference and flow characteristics of first-order neighbor nodes obtained through P2P communication network; The valve opening correction step action is as follows: ; This represents the maximum safe step amplitude allowed for the valve within a single sampling period. For each intelligent agent It features a deep Actor neural network model.

[0008] Furthermore, the reward function is: ; in, This represents the penalty for traffic deviation. For nodes The rated on-demand design flow rate is a constraint that the intelligent agent must try its best to meet the basic heating needs of this node. The real-time mass flow rate of the ultrasonic waves flowing through this control node; Represents spatial equilibrium coordination terms. and These are the current node and its first-order neighbor nodes. Real-time heating quality index; To control all first-order neighbor nodes of agent i; , This is the penalty weighting coefficient.

[0009] Furthermore, the formula for predicting the pressure difference after execution by the physical security heuristic fence operator is as follows: ; in, The predicted pressure difference at this node after this action is performed; For nodes The reference resistance coefficient of the branch; For nodes Current opening The partial derivative of the lower drag coefficient with respect to the opening degree; The valve opening increment output by the intelligent agent; The density of the fluid inside the pipe; For nodes The fluid velocity in the branch; For nodes The length of the pipe section in which it is located; This is the fluid inertia term, used to simulate the transient velocity changes caused by valve action in a physical pipeline; The physical safety heuristic fence operator is also configured to: establish neighboring pipe segment pressures, calculate the predicted pressure difference of the current node and surrounding neighboring branches after the action is executed within seconds; if the predicted pressure difference is within the safety envelope, the locking is released and the action is executed; if the action is determined to be out of bounds, the AI ​​neural network control flow is forcibly cut off within microseconds, and the action is limited by the saturation operator or downgraded to local constant pressure difference PID control.

[0010] Furthermore, the proportional collaborative step-down control includes: the near-end edge agent cuts off the isolated regulation logic that originally blindly seized the flow, synchronously reduces the opening of its own electric regulating valve according to a preset dynamic ratio, and pushes the power head and limited surplus flow to the node at the far end of the pipeline by actively increasing the local resistance coefficient of the near-end control node; The determination of central cloud connection failure includes: the edge controller not receiving the central cloud clock synchronization pulse for three consecutive sampling cycles, or determining that the external communication link is physically interrupted.

[0011] Furthermore, the MAPPO algorithm adopts a centralized training and decentralized execution framework, in which each edge agent shares global information during the training phase and independently outputs actions based solely on its local state vector during the execution phase.

[0012] This invention also discloses a distributed hydraulic autonomous control method for centralized heating networks based on multi-agent reinforcement learning, comprising: Step S1: Divide the centralized heating network into multiple local sub-grids according to the physical network topology. Each edge agent collects the transient hydraulic characteristics of its own node locally and asynchronously obtains the hydraulic characteristics of its first-order neighbor nodes through a distributed peer-to-peer P2P communication network to construct a local state vector. Step S2: Each edge agent independently runs a deep Actor neural network, inputs a local state vector, outputs a valve opening correction increment, and uses a reward function that includes a flow deviation penalty and a neighbor heating quality balance penalty to drive the edge agents of the entire network to spontaneously form an altruistic Nash cooperative strategy. Step S3: Convert the opening increment output by the edge agent into the local resistance coefficient increment, and predict the pressure difference of this node and neighboring branches after execution by the physical safety heuristic fence operator; if the predicted pressure difference exceeds the pipeline explosion-proof upper limit or is lower than the cavitation cavitation limit lower limit, then limit the opening increment by the saturation operator, or cut off the AI ​​control flow in microseconds and degrade to local constant pressure difference PID control. Step S4: When the edge controller determines that the central cloud is disconnected, it switches to the global decentralized hydraulic autonomous operation mode. When the source end or main line failure causes the total water supply flow or hydraulic head to decrease, the near-end edge agent transmits the attenuation characteristics through the P2P network and executes proportional collaborative step-down control. It actively reduces its own valve opening to increase local resistance and transfers the power head and surplus flow to the remote node, so that the whole network can be reconstructed into a safe hydraulic balance steady state within a few seconds through local neighbor horizontal game.

[0013] Furthermore, in step S1, each edge agent conducts P2P communication through a distributed CAN bus or industrial wireless IoT, with a sampling period of 1 to 3 minutes, where the P2P communication period is shorter than the sampling period.

[0014] Furthermore, in step S2, the MAPPO algorithm adopts a centralized training and decentralized execution framework. During the offline training phase, the variance penalty of the neighbor's heating quality is embedded into the local reward function. During the execution phase, each edge agent outputs actions independently, relying only on its local state vector.

[0015] Furthermore, in step S4, in the proportional collaborative step-down control, the near-end edge agent synchronously reduces its own valve opening according to a preset dynamic ratio. The preset dynamic ratio is determined based on the pressure difference deviation, flow deviation, or preset fixed ratio between the near end and the far end. The central cloud disconnection determination includes not receiving the central cloud clock synchronization pulse for three consecutive sampling cycles, or the external communication link being physically interrupted.

[0016] Compared with the prior art, the present invention has the following beneficial technical effects: This invention discloses a distributed hydraulic autonomous control system for centralized heating networks based on multi-agent reinforcement learning. The entire network is divided into multiple local sub-grids according to the physical pipe network topology. Edge agents interact only with first-order neighbor nodes and construct local state vectors, eliminating the reliance on a central server to process tens of thousands of nodes across the network for global hydraulic balance calculations. This reduces the dimensionality of the state and action spaces, decreasing long-distance communication latency and cloud computing pressure. Each edge agent independently outputs valve opening correction actions using the MAPPO algorithm. The reward function includes a penalty term for balancing neighbor heating quality. When a node takes actions that cause a decrease in neighbor heating quality, its own reward decreases. This suppresses neighbor node oscillations and regulatory catch-up caused by local selfish adjustments, promoting an altruistic and collaborative Nash strategy across the entire network. When the central cloud loses connection, the system switches to a globally decentralized hydraulic autonomous operation mode. When a source or main line failure causes a decrease in total water supply flow or hydraulic head, the near-end edge agents transmit the attenuation characteristics through the P2P network and execute proportional collaborative step-down control. They actively reduce their own valve opening to increase local resistance and transfer the dynamic head and surplus flow to the remote nodes. This allows the entire network to reconstruct a safe hydraulic balance steady state within seconds without central scheduling, solving the problems of single-point collapse in centralized systems and network-wide paralysis after a network outage.

[0017] Furthermore, the study clarifies that the autonomous control agent at the load end is deployed on the edge microprocessor of the intelligent regulating valve at the entrance of each household or building, while the collaborative topology agent at the network end is deployed on the controller of the centralized heat exchange station or the main line three-way electrically controlled valve. The specific composition of the local state vector and the calculation method for valve opening correction actions are also given, making the data acquisition objects and action outputs of each edge agent feasible. The action amplitude is constrained by the maximum safe step amplitude within a single sampling period, which can avoid secondary water hammer impacts caused by drastic valve adjustments and improve the safety of the execution process.

[0018] Furthermore, the specific form of the reward function is given. The flow deviation penalty term constrains nodes to try to meet their rated on-demand design flow, while the spatial equilibrium coordination term constrains the difference in heating quality index between the current node and its first-order neighbor nodes. When a node improves its own heating quality by grabbing water, the decline in the heating quality of its neighbors will cause the spatial equilibrium coordination term to increase rapidly, and the total reward will decrease accordingly. This will suppress water grabbing behavior at the reward mechanism level, and promote all nodes in the network to take into account both their own needs and the heating quality of their neighbors, forming a collaborative operation strategy.

[0019] Furthermore, the physical safety heuristic fence operator converts the opening increment into a local resistance coefficient increment by predicting the pressure difference after execution, and incorporates the fluid inertia term, enabling it to simulate the transient velocity changes caused by valve action in the pipeline. The fence operator combines the pressure of neighboring pipe segments, calculates the predicted pressure difference for the current node and surrounding neighboring branches within seconds, and performs hard logic checks. If the predicted pressure difference exceeds the pipeline explosion-proof upper limit or falls below the cavitation-proof lower limit, the action is limited by a saturation operator or the AI ​​control flow is cut off within microseconds, and the system is downgraded to local constant pressure difference PID control. This avoids outputting extremely dangerous actions during the reinforcement learning exploration process, preventing pipeline overpressure rupture and local cavitation cavitation, and ensuring the physical safety of the heating system.

[0020] Furthermore, the specific process of proportional collaborative step-down control was clarified. The near-edge agent cuts off the previously isolated regulation logic that blindly seized traffic, and synchronously reduces the opening of its own electric regulating valve according to a preset dynamic ratio. It actively increases the local resistance coefficient of the near-end control node, pushing the power head and limited surplus traffic to the far-end node. The network outage determination is based on the absence of a central cloud clock synchronization pulse or a physical interruption of the external communication link for three consecutive sampling cycles, which improves the reliability of central connection failure detection and provides a trigger basis for self-organization and self-healing after network outage.

[0021] Furthermore, the MAPPO algorithm employs a centralized training and distributed execution framework. Each edge agent shares global information during the training phase and independently outputs actions based solely on its local state vector during the execution phase. This approach not only leverages global information to improve policy quality during training but also eliminates reliance on a central server and global communication during execution, reducing communication burden and facilitating the system's expansion in large-scale heating networks.

[0022] This invention also provides a corresponding control method for the system, which executes the steps of local state construction, MAPPO action solving, physical security fence verification, and network outage self-organization and self-healing, enabling each edge agent to operate in both normal communication and central disconnection states. This method transforms the global hydraulic balance problem into a local graph game, reducing reliance on central computing power and long-distance communication. It suppresses water grabbing through neighbor heating quality balancing penalties, ensures action safety through fence verification, and achieves network self-balancing after a network outage through proportional collaborative step reduction, thus solving the problems of insufficient centralized scheduling computing power, poor distributed control coordination, and loss of control after a network outage.

[0023] Furthermore, each edge agent communicates via a distributed CAN bus or industrial wireless IoT, with a sampling period of 1 to 3 minutes. The P2P communication period is shorter than the sampling period, which can adapt to the slow dynamic characteristics of the heating network, ensure the freshness of the neighbor's state, and avoid the bandwidth pressure caused by high-frequency communication, so that the local state vector can reflect the hydraulic changes of the neighbor in a timely manner.

[0024] Furthermore, in the ninth technical solution, the MAPPO algorithm adopts a centralized training and decentralized execution framework. During the offline training phase, the variance penalty of neighbor heating quality is embedded into the local reward function. During the execution phase, each edge agent independently outputs actions based solely on its local state vector. Thus, the training phase can leverage global information to optimize collaborative strategies, while the execution phase requires no central server involvement, ensuring the effectiveness of decentralized control and reducing online computation and communication overhead.

[0025] Furthermore, in the proportional collaborative step-down control, the near-end edge agent synchronously reduces its own valve opening according to a preset dynamic ratio. The preset dynamic ratio is determined based on the pressure difference deviation, flow deviation, or a preset fixed ratio between the near and far ends, allowing the step-down degree to be flexibly adjusted according to actual working conditions. The central cloud disconnection determination includes the absence of clock synchronization pulses for three consecutive sampling cycles or a physical interruption of the external communication link, improving the reliability of disconnection determination and supporting the entire network to reconstruct a safe hydraulic balance steady state within seconds through local neighbor lateral game theory after a network outage. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall architecture of a centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning according to the present invention. Figure 2 This is a schematic diagram of the internal control process of the network-end collaborative topology intelligent agent of the present invention; Figure 3 This is a schematic diagram of the internal control process of the autonomous control agent at the load end of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the present invention, and not all of them.

[0028] The detailed description of the embodiments of the present invention provided in the following figures is not intended to limit the scope of the claimed invention, but merely to illustrate one selected embodiment of the invention. All other embodiments obtained by those skilled in the art based on the figures and embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] For ease of description, the first in the centralized heating network will be referred to as the [missing information]. A physical node is denoted as a node. It will be deployed on the node Edge intelligent agents are denoted as intelligent agents. ,node With intelligent agents One-to-one correspondence; nodes Used to represent physical location and its hydraulic characteristics, intelligent agent The control entity used to represent the collection of execution status, action solving, and safety verification.

[0030] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0031] Example 1: Overall Architecture of Distributed Hydraulic Autonomous Control System for Centralized Heating Network like Figure 1 As shown, this embodiment provides a centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning. Its overall architecture, from bottom to top, includes a load-end autonomous control agent, a network-end collaborative topology agent, a distributed peer-to-peer (P2P) communication network, and a physical security heuristic fence operator.

[0032] like Figure 3 As shown, the load-side autonomous control agent is deployed on the edge controller of the user-side intelligent control valve. Specifically, the load-side autonomous control agent can be deployed on the edge microprocessor of the individual household intelligent control valve, or on the edge microprocessor of the building entrance intelligent control valve, or simultaneously on both types of valves. Each load-side autonomous control agent includes a local sensing and acquisition module, an edge computing module, and a P2P communication module. The local sensing and acquisition module connects to an ultrasonic flow meter, a differential pressure sensor, and a valve opening sensor to collect the transient differential pressure of the local node at a sampling period of 1 to 3 minutes. mass flow rate and valve opening The edge computing module is equipped with a lightweight deep Actor neural network for independently calculating valve opening correction actions. The P2P communication module is used for high-frequency asynchronous information exchange with directly adjacent first-order neighbor nodes in the physical pipeline network topology.

[0033] like Figure 2 As shown, the network-end collaborative topological agent is deployed at the heat exchange station or the branch valve side of the main line. Specifically, the network-end collaborative topological agent can be deployed on the centralized heat exchange station controller or on the side of the main line three-way electrically controlled valve to coordinate a group of load-end autonomous control agents under its jurisdiction. The network-end collaborative topological agent and the load-end autonomous control agent together constitute the edge agent. In addition to having similar local data acquisition, edge computing, and P2P communication functions as the load-end agents, the network-end collaborative topological agent is also responsible for maintaining the topological connectivity of its local sub-grids and acts as a local coordinating node in neighbor game theory.

[0034] The distributed peer-to-peer (P2P) communication network is implemented using a decentralized fieldbus or industrial wireless IoT, such as a distributed CAN bus or an industrial wireless open network. This network allows for high-frequency, localized point-to-point information exchange between adjacent edge agents, without relying on any central server. Each edge agent interacts only with its first-order neighbor nodes that are directly upstream, downstream, or bypassed within the physical network topology. To conduct data interaction.

[0035] The physical safety heuristic fence operator is embedded in the execution front end of each edge agent, located between the output of the deep Actor neural network and the valve motor actuator. After the reinforcement learning action is output but before the actuator moves, this fence operator performs hydraulic safety boundary checks and enforces constraints on the action. If the action is safe, it is allowed to proceed to the actuator; if the action exceeds the boundary, the action is restricted through a saturation operator, or the AI ​​control flow is cut off at the microsecond level and downgraded to local constant pressure differential PID control.

[0036] Example 2: Construction of Distributed Multi-Agent Topology State Space This embodiment details the process of constructing the distributed multi-agent topology state space.

[0037] First, the centralized heating network is abstracted into a diagram. ,in This is a set of edge computing nodes, representing individual household control valves, building entrance control valves, or branch electric valves of heat exchange stations. The network consists of pipes connecting the nodes. The entire network is divided according to the physical pipe network topology. N A geographically distributed local subgrid topology map.

[0038] For any edge agent in the entire network It consists only of the set of first-order neighbor nodes that are directly upstream, downstream, or bypassed in the physical pipeline network topology. Perform data cascading. Edge intelligent agents. At each sampling step Internally, construct a low-dimensional, compact state vector containing only the features of the current node and its neighbors: ; in: For the first Real-time transient pressure difference across the electric regulating valve at each node ( MPa) ; The real-time mass flow rate of the ultrasonic waves flowing through this control node ( t / h ); The current valve opening percentage (0-100%). It is a spliced ​​matrix of pressure difference and flow characteristics of all first-order neighbor nodes obtained through high-frequency asynchronous interaction of P2P field peer-to-peer bus networks (such as distributed CAN bus or industrial wireless open network). This indicates that the components are assembled in a preset order.

[0039] In a specific example, the sampling step size The time interval is 1 to 3 minutes. The P2P communication network uses a distributed CAN bus, and the communication cycle is shorter than the sampling step size to ensure the freshness of the neighbor's state.

[0040] Example 3: Autonomous Action Solving and Nash Coordination Reward Based on MAPPO This embodiment details the autonomous action solution process for MAPPO based on multi-agent proximal policy optimization.

[0041] Each edge agent is equipped with an independent deep Actor neural network model. (Parameters are) Edge intelligent agents The valve opening correction step action is independently output through high-frequency rolling iteration: ; in, This represents the maximum safe step amplitude allowed by the hardware valve within a single sampling period, used to prevent the valve from drastically overshooting and causing secondary water hammer shock waves in the pipeline. The tanh function limits the output to [...]. Within the interval [1,1], multiply by the maximum safe step size to obtain the actual opening correction.

[0042] To address the conflicting control oscillations among neighboring nodes caused by the blind, self-serving adjustments in traditional distributed control, this embodiment designs a Nash equilibrium coordination reward function that includes a penalty for balancing neighbor heating quality: ; in, This is a penalty for traffic deviation. Given the rated on-demand design flow required by this node, the constrained agent must try its best to meet the basic heating needs of this node. For spatial equilibrium coordination terms, and These are the real-time heating quality indices for the current node and its first-order neighbor nodes, respectively. , The penalty weighting coefficient can be adjusted according to the actual characteristics of the pipeline network, for example... Take 0.6, Take 0.4.

[0043] The heating quality index is obtained by decoupling the temperature difference and the actual heat absorption. The specific calculation formula is as follows: ; in, For nodes The temperature difference between the supply and return water For nodes The actual heat absorbed, For reference temperature difference, For nodes Reference heat absorption, , These are the weighting coefficients.

[0044] The collaborative control mechanism is as follows: when the intelligent agent When attempting selfish actions, such as forcibly opening its own valves to increase its own HQI, the hydraulic coupling characteristics of the pipe network will inevitably lead to the destruction of adjacent first-order neighbor nodes. Traffic is being monopolized, causing a sharp drop in HQI. At this point, the spatial imbalance penalty value of the consequent will surge dramatically, leading to a decrease in the agent's total reward. It has fallen to the bottom. Driven by the AI ​​back-end of maximizing the reward function, the MAPPO algorithm can force all independent agents in the network to spontaneously evolve an "altruistic and mutually supportive" Nash cooperative strategy, achieving spontaneous hydraulic balance of the entire network without centralized control.

[0045] In a specific example, the MAPPO algorithm employs a centralized training and distributed execution framework. During the offline training phase, each agent's Critic network can share global information for centralized training, while the Actor network outputs actions based on local states; the experience replay pool stores (…). , , , The quadruple set; after training, each edge agent deploys the trained Actor network parameters and no longer depends on the central server. During the execution phase, each agent only relies on its local state vector. Si ( t Independent output actions. The training cycle and execution cycle are separated, and the communication cycle is shorter than the sampling cycle to ensure the freshness of neighbor states.

[0046] Example 4: Implementation of a Physics-Inspired On-Site Safety Fence Operator This embodiment details the specific implementation of the physical security heuristic fence operator.

[0047] Pure artificial intelligence algorithms suffer from the characteristic of exploring black boxes, making them highly susceptible to outputting illegal and extreme actions during initial system operation or when encountering large-scale disturbances in the network. To ensure the uninterrupted and absolute security of the public service system, this embodiment embeds a physical security sandbox, i.e., a fence operator, based on the microgrid resistance mechanism at the hardware execution front end of each edge computing agent.

[0048] Combination Figure 2 and Figure 3 The physical safety heuristic fence operator is located between the deep Actor neural network module and the field hardware unit, and is used to perform hydraulic safety boundary verification after the reinforcement learning action output and before the actuator action.

[0049] Fence operators intercept actions generated by deep Actor neural networks It does not directly drive the motor. First, it is converted into the increase in the local resistance coefficient caused by the change in the opening degree of the current branch. Substituting the formula for predicting the pressure difference after execution using the physical safety heuristic fence operator: ; in, The predicted pressure difference at this node after this action is performed; For nodes The reference resistance coefficient of the branch; For nodes Current opening The partial derivative of the lower drag coefficient with respect to the opening degree; The valve opening increment output by the intelligent agent; The density of the fluid inside the pipe; For nodes The fluid velocity in the branch; For nodes The length of the pipe section in which it is located; This is the fluid inertia term, used to simulate the transient velocity changes caused by valve action in a physical pipeline; The fence operator, by combining the above formula with the pressure of neighboring pipe segments, can calculate the predicted pressure difference of the current node and its surrounding neighboring branches after the action is performed within seconds. System startup hardware logic check:

[0050] in This is the upper limit of the design explosion-proof value for the physical pressure resistance and impact resistance of this section of pipeline; The lower limit of the limiting dynamic water pressure head is used to prevent local cavitation and vaporization (cavitation) of the fluid due to excessively high flow velocity and low pressure.

[0051] The control response is as follows: Once the action exceeds the limit, the safety fence operator forcibly cuts off the control flow of the AI ​​neural network within microseconds, and uses the saturation operator SAT to forcibly limit the opening increment within a safe saturation boundary, or automatically degrades and reverts to local constant differential pressure physical PID control. If the predicted differential pressure is within the safe envelope, the lockout is released, and the action command is sent to the valve actuator.

[0052] Example 5: P2P peer-to-peer self-organizing reconstruction and network outage self-healing control This embodiment details the distributed self-organizing and self-healing control process of the entire network hardware control body in extreme abnormal scenarios such as interruption of the external dispatch communication network of the centralized heating system, paralysis of the central SCADA control system due to network attack, or crash of the central cloud server.

[0053] Step 1: Network Disconnection Adaptive Identification and Mode Switching. Edge controllers on hardware nodes at all levels of the network continuously monitor the clock synchronization signal from the central cloud. When an edge controller fails to receive a clock synchronization pulse from the central cloud for three consecutive sampling cycles, or determines that an external communication link has been physically interrupted, the system immediately and automatically disconnects the traditional centralized scheduling and control link, and seamlessly switches back to the "globally decentralized hydraulic autonomous operation mode" within milliseconds.

[0054] Step Two: Self-Organizing Peer-to-Peer Network State Takeover. After switching to autonomous mode, the distributed intelligent agent control kernel across the entire network completely takes over the corresponding field hardware actuators. Intelligent agents at each level no longer rely on centralized computing instructions from any central server, but instead rely entirely on a decentralized, distributed peer-to-peer (P2P) industrial network. They interact with the hydraulic characteristics of their adjacent first-order neighbor nodes connected in the physical pipeline topology at high frequency, constructing a distributed graph network local game state space at the bottom layer of the entire network.

[0055] Step 3: Proactive Self-Healing and Altruistic Collaborative Decision-Making. When a sudden hardware failure in the source or main line of the heating network causes a significant decrease in total water supply flow or total hydraulic head, the intelligent agent edge controller located near the network immediately detects the sudden drop in local dynamic water head through its onboard pressure sensor. Because its internal MAPPO action network has forcibly embedded the neighbor heating quality variance penalty into the local reward function during the offline training phase, the near-end intelligent agent instantly transmits this attenuation feature backward through the P2P network and spontaneously triggers an "altruistic collaborative" action.

[0056] Step 4: Proportional Cooperative Step-Down and New Steady-State Construction. The near-end agent actively disconnects from the previously isolated regulation logic that blindly seized flow, and autonomously executes proportional cooperative step-down control, sending control signals to proactively reduce the opening of its own electric regulating valve by a preset dynamic ratio. The preset dynamic ratio can be determined based on the pressure difference deviation, flow deviation, or a preset fixed ratio between the near and far ends.

[0057] In the proportional collaborative step-down control, the intelligent agent located at the near end of the pipeline network is denoted as the near-end intelligent agent. The near-end intelligent agent synchronously reduces its own valve opening according to a preset dynamic ratio.

[0058] In a specific example, the indentation ratio is calculated using the following formula: ; in, For near-end intelligent agents The step ratio, This is a reference value for the differential pressure at the distant target. This represents the current pressure differential at the near end. When the near-end pressure differential is significantly higher than the reference value, the step-down ratio increases, actively releasing more flow. By actively increasing the local resistance coefficient of the near-end control node, valuable dynamic head and limited surplus flow are pushed to nodes at the far end of the pipeline network.

[0059] Step 5: Decentralized Self-Balancing Closed-Loop Execution. Without any central command or dispatch response, tens of thousands of distributed intelligent valves and variable frequency pumps across the network synchronously and autonomously complete fine-tuning actions within seconds through horizontal game theory among their local neighbors at the lower levels. The amplitude of individual cycle actions is constrained within safe limits. By reshuffling the Kirchhoff pressure and flow rate of the local grid, the entire complex heating network is reconstructed locally into a new steady state of safe hydraulic balance under low-flow operation, preventing large-scale hydraulic paralysis and secondary pipe bursts and slurry failures.

[0060] Example 6 This embodiment provides a specific implementation process for a centralized heating network distributed hydraulic autonomous control method based on multi-agent reinforcement learning. This method can be applied to the centralized heating network distributed hydraulic autonomous control system described in any one of embodiments 1 to 5 above. The method includes the following steps: Step 1: Distributed Multi-Agent Topology State Space Construction. The system decomposes the complex global thermal network nonlinear hydrodynamic system into... N A geographically distributed local subgrid topology map. Edge controllers on each hardware node locally and at high frequency acquire their own transient pressure difference, mass flow rate, and valve opening characteristics. The agent, through a distributed P2P peer-to-peer fieldbus network, asynchronously acquires the hydraulic characteristics of its first-order neighbor nodes directly connected to the physical pipeline, and concatenates them into a compact low-dimensional local state vector. This transforms the complex global equilibrium problem into a local graph game.

[0061] Step 2: Autonomous Action Solving Based on MAPPO. Within each sampling cycle, each controller independently runs its built-in deep Markov decision engine. The agent, based on its own deep Actor neural network, takes its local state vector as input, performs rolling calculations, and outputs its own valve opening correction incremental action command. The algorithm sets a maximum action amplitude constraint to curb secondary water hammer impacts. During operation, it employs a coordinated reward function that includes a penalty for balancing neighbor heating quality. This forcibly embeds the neighbor's interests into the local AI reward function. If an agent selfishly takes water, causing a sharp drop in neighbor heating quality, its total reward will trigger a precipitous penalty, driving and forcing all independent agents in the network to spontaneously engage in altruistic Nash cooperative strategies. The MAPPO algorithm uses a centralized training and distributed execution framework. During offline training, the neighbor heating quality variance penalty is embedded into the local reward function. During execution, each edge agent independently outputs actions based solely on its local state vector.

[0062] Step 3: Zero-overshoot interception control of the physical heuristic field safety fence operator. The action command output by the agent is first intercepted by the physical safety fence operator embedded in the hardware front end. The operator converts the opening increment into the branch local resistance coefficient increment and substitutes it into the formula for predicting the pressure difference after execution by the physical safety heuristic fence operator, predicting the flow field pressure difference change trend caused by the action within seconds. The fence operator performs hard logic verification on the predicted pressure difference. If the prediction result is within the safety envelope formed by the upper limit of the system design explosion-proof pipe and the lower limit of the anti-cavitation hydrostatic head, the lock is released and the command is issued; if the action is determined to be out of bounds or cause overshoot, the AI ​​control flow is forcibly cut off within microseconds, and the saturation operator is used to limit the valve opening to the safe saturation region to ensure absolute hardware safety.

[0063] Step Four: P2P Peer-to-Peer Self-Organizing Reconstruction and Black-Box Anomaly Self-Healing. When the external dispatch network is interrupted, the SCADA system is attacked, or the central cloud server crashes and loses connection, the system automatically determines that the central connection is lost and seamlessly switches to a globally decentralized hydraulic autonomous operation mode within seconds. When a sudden fault at the source end of the heating network causes a significant decrease in total water supply flow or hydraulic head, the near-end building intelligent agents immediately sense the sudden drop in local pressure difference. Driven by the collaborative punishment mechanism within the MAPPO network, the near-end intelligent agents autonomously trigger proportional collaborative step-down control, spontaneously reducing their own valve openings. By actively increasing their own local resistance, the near-end intelligent agents push valuable dynamic head and limited surplus flow to nodes at the far end of the pipeline network. Without any central command and dispatch, tens of thousands of intelligent valves across the network, through the underlying local neighbor peer-to-peer game, synchronously complete fine-tuning actions within seconds, locally reconstructing the entire complex heating network into a new steady state of safe hydraulic balance under low flow operation.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning, characterized in that, include: Multiple autonomous control agents are deployed on the edge controller of the user-end intelligent regulating valve; Multiple network-end collaborative topology intelligent agents are deployed at heat exchange stations or main line branch valves to coordinate the autonomous control intelligent agents at the load ends under their jurisdiction; Both the load-side autonomous control agent and the network-side collaborative topology agent constitute edge agents; The distributed peer-to-peer (P2P) communication network is configured to support point-to-point local information exchange between adjacent edge agents. A physical safety heuristic fence operator is embedded in the execution front end of each edge agent and is used to perform hydraulic safety boundary verification and mandatory constraints on the action after the reinforcement learning action is output and before the action of the execution mechanism. Each of the aforementioned edge agents is configured as follows: The entire network is divided into multiple local sub-grids according to the physical network topology. Each edge agent only interacts with the first-order neighbor nodes that are directly adjacent to it in the physical network topology and constructs a local state vector containing the hydraulic characteristics of its own node and the hydraulic characteristics of its first-order neighbor nodes on the spot. Each edge agent is equipped with an independent deep Actor neural network. Through the multi-agent proximal strategy optimization MAPPO algorithm, it independently outputs valve opening correction step actions. The reward function of the MAPPO algorithm includes a flow deviation penalty term and a neighbor heating quality balance penalty term. The neighbor heating quality balance penalty term is used to reduce the reward of the node when the node takes a water grabbing action, which causes the neighbor's heating quality to decrease, so as to drive the edge agents of the entire network to converge to the Nash cooperative strategy. The system is also configured to perform decentralized autonomy when the central cloud is disconnected: when the edge controller determines that the central cloud is disconnected, it switches to a global decentralized hydraulic autonomy operation mode; when the source end or main line failure causes the total water supply flow or hydraulic head to decrease, the near-end edge agent transmits the attenuation characteristics through the P2P network and performs proportional collaborative step-down control, actively reducing its own valve opening to increase local resistance, and transferring the power head and surplus flow to the remote node, so that the entire network can be reconstructed into a safe hydraulic balance steady state within a few seconds.

2. The centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The load-side autonomous control agent is deployed on the edge microprocessor of the individual household intelligent regulating valve and / or the building entrance intelligent regulating valve; the network-side collaborative topology agent is deployed on the side of the centralized heat exchange station controller and / or the main line three-way electrically controlled valve. The local state vector is: ; in: , for the first Real-time transient pressure difference across the electric regulating valve at each node; The real-time mass flow rate of the ultrasonic waves flowing through this control node; This represents the current percentage of the valve's actual opening. , is a spliced ​​matrix of pressure difference and flow characteristics of first-order neighbor nodes obtained through P2P communication network; The valve opening correction step action is as follows: ; This represents the maximum safe step amplitude allowed for the valve within a single sampling period. For each intelligent agent It features a deep Actor neural network model.

3. The centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The reward function is: ; in, This represents the penalty for traffic deviation. For nodes The rated on-demand design flow rate is a constraint that the intelligent agent must try its best to meet the basic heating needs of this node. The real-time mass flow rate of the ultrasonic waves flowing through this control node; Represents spatial equilibrium coordination terms. and These are the current node and its first-order neighbor nodes. Real-time heating quality index; To control all first-order neighbor nodes of agent i; , This is the penalty weighting coefficient.

4. The centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The formula for predicting the pressure difference after execution using the physical security heuristic fence operator is as follows: ; in, The predicted pressure difference at this node after this action is performed; For nodes The reference resistance coefficient of the branch; For nodes Current opening The partial derivative of the lower drag coefficient with respect to the opening degree; The valve opening increment output by the intelligent agent; The density of the fluid inside the pipe; For nodes The fluid velocity in the branch; For nodes The length of the pipe section in which it is located; This is the fluid inertia term, used to simulate the transient velocity changes caused by valve action in physical pipelines; The physical safety heuristic fence operator is also configured to: establish neighboring pipe segment pressures, calculate the predicted pressure difference of the current node and surrounding neighboring branches after the action is executed within seconds; if the predicted pressure difference is within the safety envelope, the locking is released and the action is executed; if the action is determined to be out of bounds, the AI ​​neural network control flow is forcibly cut off within microseconds, and the action is limited by the saturation operator or downgraded to local constant pressure difference PID control.

5. A centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The proportional collaborative step-down control includes: the near-end edge agent cuts off the isolated regulation logic that originally blindly seized the flow, synchronously reduces the opening of its own electric regulating valve according to a preset dynamic ratio, and pushes the power head and limited surplus flow to the node at the far end of the pipeline by actively increasing the local resistance coefficient of the near-end control node. The determination of central cloud connection failure includes: the edge controller not receiving the central cloud clock synchronization pulse for three consecutive sampling cycles, or determining that the external communication link is physically interrupted.

6. A centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning according to claim 1, characterized in that, The MAPPO algorithm adopts a centralized training and decentralized execution framework. Each edge agent shares global information during the training phase and independently outputs actions based solely on its local state vector during the execution phase.

7. A method for distributed hydraulic autonomous control of centralized heating networks based on multi-agent reinforcement learning, characterized in that, The method is applied to the centralized heating network distributed hydraulic autonomous control system based on multi-agent reinforcement learning as described in any one of claims 1-6; the method includes: Step S1: Divide the centralized heating network into multiple local sub-grids according to the physical network topology. Each edge agent collects the transient hydraulic characteristics of its own node locally and asynchronously obtains the hydraulic characteristics of its first-order neighbor nodes through a distributed peer-to-peer P2P communication network to construct a local state vector. Step S2: Each edge agent independently runs a deep Actor neural network, inputs a local state vector, outputs a valve opening correction increment, and uses a reward function that includes a flow deviation penalty and a neighbor heating quality balance penalty to drive the edge agents of the entire network to spontaneously form an altruistic Nash cooperative strategy. Step S3: Convert the opening increment output by the edge agent into the local resistance coefficient increment, and predict the pressure difference of this node and neighboring branches after execution by the physical safety heuristic fence operator; if the predicted pressure difference exceeds the pipeline explosion-proof upper limit or is lower than the cavitation cavitation limit lower limit, then limit the opening increment by the saturation operator, or cut off the AI ​​control flow in microseconds and degrade to local constant pressure difference PID control. Step S4: When the edge controller determines that the central cloud is disconnected, it switches to the global decentralized hydraulic autonomous operation mode. When the source end or main line failure causes the total water supply flow or hydraulic head to decrease, the near-end edge agent transmits the attenuation characteristics through the P2P network and executes proportional collaborative step-down control. It actively reduces its own valve opening to increase local resistance and transfers the power head and surplus flow to the remote node, so that the whole network can be reconstructed into a safe hydraulic balance steady state within a few seconds through local neighbor horizontal game.

8. The method for distributed hydraulic autonomous control of centralized heating networks based on multi-agent reinforcement learning according to claim 7, characterized in that, In step S1, each edge agent communicates via a distributed CAN bus or industrial wireless IoT, with a sampling period of 1 to 3 minutes, where the P2P communication period is shorter than the sampling period.

9. A method for distributed hydraulic autonomous control of centralized heating networks based on multi-agent reinforcement learning according to claim 7, characterized in that, In step S2, the MAPPO algorithm adopts a centralized training and decentralized execution framework. During the offline training phase, the variance penalty of neighbor heating quality is embedded into the local reward function. During the execution phase, each edge agent outputs actions independently based solely on its local state vector.

10. A method for distributed hydraulic autonomous control of centralized heating networks based on multi-agent reinforcement learning according to claim 7, characterized in that, In step S4, in the proportional collaborative step-down control, the near-end edge agent synchronously reduces its own valve opening according to a preset dynamic ratio. The preset dynamic ratio is determined based on the pressure difference deviation, flow deviation, or preset fixed ratio between the near end and the far end. The central cloud disconnection determination includes not receiving the central cloud clock synchronization pulse for three consecutive sampling cycles, or the external communication link being physically interrupted.

Citation Information

Patent Citations

  • Water loop heat pump multi-split control method based on heating engineering

    CN122590339A