Pipe regulation method and system based on edge computing and topology-aware reinforcement learning

Abstract

This invention discloses a pipeline control method and system based on edge computing and topology-aware reinforcement learning. The method includes: constructing a directed graph of the pipeline network; constructing node feature vectors and edge action state vectors; acquiring global topology features; constructing a deep reinforcement learning network including a policy network and a value network, inputting the global topology features into the policy network to predict preliminary action vectors, and using the value network to estimate the value of the policy network's output; co-training the policy network and the value network, and finally using the trained policy network to output preliminary action vectors; setting hard constraints and soft constraints to verify the legality of the preliminary action vectors, and outputting final control commands; issuing the final control commands to the execution mechanism, and the execution mechanism executing the final control commands. The advantages of this invention are that it can adapt to changes in pipeline network topology, ensure physical operational safety, and has real-time response capabilities.

Description

Technical Field

[0001] This invention relates to the field of industrial internet and smart pipeline control technology, specifically to a pipeline regulation method and system based on edge computing and topology-aware reinforcement learning. Background Technology

[0002] In large-scale fluid transportation networks such as oil, natural gas, water supply, and district heating, the scheduling of pump stations and valves is a core element in ensuring the safe and economical operation of the system. With the expansion of pipeline networks, traditional scheduling methods are facing severe challenges.

[0003] Existing technologies mainly include the following types of solutions:

[0004] 1. Manual or PID control: This method relies on operator experience or simple single-loop PID control. It struggles with complex pipe networks involving multiple coupled nodes, exhibits slow response, and often sacrifices energy efficiency for safety, resulting in significant energy waste.

[0005] 2. Mathematical programming methods: such as mixed integer linear programming (MILP). These methods solve for the optimal solution by establishing an accurate physical model, but their computational complexity increases exponentially with the number of nodes, making it difficult to meet real-time control requirements at the second or minute level, and they are extremely sensitive to the accuracy of the model parameters.

[0006] 3. Traditional Deep Reinforcement Learning (DRL) Algorithms: Deep Q-Network (DQN) or Deep Deterministic Policy Gradient (DDPG) algorithms. For example, the water hammer control method and system for water supply networks based on multi-functional module fusion disclosed in CN120372875A uses the DQN algorithm. Although it has real-time decision-making capabilities, the input layer dimension of its neural network is usually fixed. Once the pipeline network is expanded, repaired, or a fault occurs, causing a change in the topology, the original model becomes invalid, and data must be collected and retrained, which greatly limits its engineering applicability in industrial fields. In addition, its purely data-driven model is prone to outputting dangerous actions that violate fluid dynamics laws (such as overpressure, pump surge), lacking inherent safety.

[0007] Therefore, there is an urgent need for an intelligent control method that can adapt to changes in pipeline topology, ensure physical operational safety, and provide real-time response capabilities. Summary of the Invention

[0008] The technical problem to be solved by this invention is how to provide a pipeline control method that can both adapt to changes in pipeline network topology and ensure physical operational safety, and has real-time response capability.

[0009] This invention solves the above-mentioned technical problems through the following technical means: a pipeline control method based on edge computing and topology-aware reinforcement learning, comprising: S1. Construct a directed graph of the pipeline network; S2. Collect pipeline pressure, flow rate, and valve opening to construct node feature vectors and edge action state vectors; S3. The graph attention network calculates the dynamic gating factor using the edge action state vector, uses the dynamic gating factor to multiply the attention coefficient to obtain the attention weight, uses the attention weight to perform a weighted summation of the node feature vectors of neighboring nodes, and generates the node features of the next time step through a nonlinear activation function. The node features of all nodes at all time steps are used as the global topological features. S4. Construct a deep reinforcement learning network that includes a policy network and a value network. Input global topological features into the policy network to predict the initial action vector, and use the value network to estimate the value of the policy network's output. Train the policy network and the value network together, and finally use the trained policy network to output the initial action vector. S5. Set hard and soft constraints to verify the legality of the preliminary action vector and output the final control command; S6. The final control command is sent to the actuator, and the actuator executes the final control command.

[0010] Further, S1 includes: The pipeline network is constructed as a directed graph G=(V,E), where V represents the set of nodes, including pump stations, valves, and user nodes. The static feature vector of each node in the set contains the node elevation and node type code. E represents the set of edges, representing physical pipelines and characterizing the connection relationships between nodes. The feature vector of each edge... This includes pipe length L, pipe diameter D, and Heisenberg-Williams roughness coefficient C.

[0011] Furthermore, S2 includes: S21, the nodes collected at time t pressure ,flow Updated to the node attributes of the directed graph G, together with the static feature vectors, constitute the node. eigenvectors ; S22, the valve opening at time t Update the corresponding edge attributes in the directed graph G as the edge action state vector.

[0012] Furthermore, S3 includes: Using formula Calculate the dynamic gating factor, where, , , They are nodes with neighboring nodes The pipe length, pipe diameter, and Heisenberg-Williams roughness coefficient between them; It is a constant; Use the Sigmoid activation function; Using formula Obtain the splicing features, where, The characteristic transformation matrix, Indicates feature splicing, For neighboring nodes eigenvectors; Using formula compute nodes with neighboring nodes Attention scores between, For linear mapping vectors, It is a linear activation function with leakage; Using formula Get Nodes with neighboring nodes Attention weights between them For nodes The set of neighboring nodes, Represents a node neighboring nodes , For nodes with neighboring nodes Attention scores between them; Using formula Generate node features for the next time step.

[0013] Furthermore, the process of co-training the policy network and the value network is as follows: Construct the original reward objective function:

[0014] in, Discount factor; This refers to the real-time operating power of the pumping station. Rated power; The real-time pressure of node i; Set the pressure for node i; , These are the normalized weighting coefficients. For the current moment, The time window length, As expected, The modulus of the set of nodes; Construct the physical constraint cost function:

[0015] in, , These are the upper and lower limits of the permissible safe pressure for pipeline nodes. for Activation function; Set the update target for the value network :

[0016]

[0017] in, For the batch sample size, For the trainable parameters of the value network, express time, For the actual return, calculate from the current time t to... All rewards at any time The present value of the discounted total; For value networks based on the global topological features of the input The predicted value obtained; Set the update target for the policy network. :

[0018] in, These are the trainable parameters of the policy network; For tolerance, This is the penalty coefficient; The trainable parameters of the value network and the trainable parameters of the policy network are continuously adjusted, and their update targets are calculated. Training is stopped when the update target of the value network is minimized and the update target of the policy network is maximized, thus obtaining a well-trained deep reinforcement learning network.

[0019] Furthermore, the penalty coefficient is iteratively updated using a PID controller, as expressed by the following formula:

[0020] in, This represents the physical constraint violation error value within the m-th training iteration period; , , These are the proportional coefficient, integral coefficient, and differential coefficient, respectively. Let be the penalty coefficient in the m-th training iteration period. This serves as the penalty coefficient for the policy network during the (m+1)th training iteration.

[0021] Furthermore, the hard constraint includes: For variable frequency pumps, the speed range is defined based on their HQ characteristic curve. If the rotational speed in the initial motion vector Then proceed to soft constraint verification; if the rotational speed in the initial motion vector... Then, after performing the projection operation, the soft constraint verification will proceed. The projection operation is as follows:

[0022] in, , These are the upper and lower limits of the high-efficiency operating range dynamically calculated based on the HQ characteristic curve of the variable frequency pump. This is a truncation function.

[0023] Furthermore, the soft constraint includes: A neural network proxy model is constructed, wherein the input of the neural network proxy model is the action command after hard constraint processing, and the output is the predicted pressure distribution of all network nodes. The neural network proxy model is a pre-trained multilayer perceptron or graph neural network; a safety threshold is set. ,like > If this is not the case, it is considered a soft violation, and an end-to-end gradient correction is performed. The modification process is as follows: The action instruction after soft violation processing with hard constraints is used as the current iteration action. ; Construct a penalty loss function for exceeding the pressure limit. The gradient of the loss function with respect to the input action vector is calculated through backpropagation. ; The formula for calculating the next iteration action is:

[0024] in, Indicates the gradient clipping threshold. Indicates the learning rate; Indicates the momentum factor; Will The neural network surrogate model is fed back into the model for prediction. If the new prediction pressure satisfies... Then stop the search and will As the final security directive Issued; otherwise, The step of returning the gradient calculation as the current iteration action continues to the next iteration until the constraints are satisfied.

[0025] Furthermore, the training process of the neural network proxy model is as follows: A loss function for the neural network surrogate model is constructed, and the parameters of the neural network surrogate model are continuously adjusted until the loss function of the neural network surrogate model is minimized. Training is then stopped, resulting in a trained neural network surrogate model. The loss function of the neural network surrogate model is:

[0026] in, The mean square error of the data; The physical residual at node i; Let i be the betweenness centrality index of node i; These are the global weighting coefficients for the physical constraint terms; This is the adjustment coefficient;

[0027] Where N1 is the batch sample size; The prediction pressure output by the neural network surrogate model at node i; Let i be the actual label corresponding to node i;

[0028]

[0029] in, Let i be the set of neighboring nodes of node i; The basic water demand of node i; The flow rate of the pipe segment connecting node i and its neighbor node j; , , , respectively, represent the pipe length, pipe diameter, and Hessen-Williams roughness coefficient between node i and its neighbor node j; k2 is the unit conversion factor for the Hessen-Williams formula; Pipe diameter index; For pressure gradient exponent; 1- This is a numerical stability compensation term;

[0030] in, This represents the number of paths that pass through node i and are the shortest paths. This represents the number of shortest paths connecting the starting node s and the ending node z.

[0031] This invention also provides a pipeline control system based on edge computing and topology-aware reinforcement learning, comprising: The intelligent edge computing gateway is used to run the pipeline control method based on edge computing and topology-aware reinforcement learning. Distributed sensing terminals are used to collect real-time node pressure and pipeline flow and transmit them to the intelligent edge computing gateway. The actuator is used to execute the final control commands issued by the intelligent edge computing gateway.

[0032] The advantages of this invention are: (1) This invention uses the directed graph of the pipeline network as input data. After the actuator executes the final control command, it dynamically updates the directed graph, thereby adapting to changes in the pipeline network topology. In addition, the legality of the preliminary action vector is verified by hard and soft constraints to ensure physical operation safety. At the same time, the overall scheme does not have complex mathematical programming problems, the calculation is simple, and the powerful operation capabilities of graph attention networks and deep reinforcement learning networks enable the system to have real-time response capabilities.

[0033] (2) The graph attention mechanism proposed in this invention maps the concept of hydraulic impedance in fluid mechanics to the attention bias of a neural network. Compared with the traditional pure data-driven GNN, it can automatically distinguish key nodes based on the physical properties of the pipeline (pipe diameter, roughness), and can still accurately capture the pressure transmission path under data noise interference, greatly improving the interpretability and robustness of the model. In addition, this invention constructs a motion-aware dynamic gating graph attention mechanism. Unlike the traditional GNN which only calculates attention based on static physical properties (such as pipe length, pipe diameter), this invention introduces a dynamic gating factor controlled by valve opening. This enables the neural network to perceive the changes in the strength of the pipeline topology caused by valve adjustment (such as shut-off, throttling) in real time, effectively solving the problem of prediction failure of traditional models when the flow path changes due to equipment action, and significantly improving the model's generalization ability to complex working conditions.

[0034] (3) To address the pain points of long computation time and non-differentiability in traditional hydraulic simulations (such as the Newton-Raphson method), this invention constructs a neural network proxy model based on physical information. Using this model to replace the traditional solver for safety verification not only reduces the computation time from seconds to milliseconds, but also utilizes its differentiability to achieve end-to-end action gradient correction. This gradient-based deterministic optimization mechanism completely eliminates the blindness and uncertainty of traditional random search algorithms when searching for safe actions.

[0035] (4) To address the problem of training oscillations caused by physical constraints (such as pressure red lines) in complex pipeline networks, this invention employs a PID control law fusion. The algorithm is updated. By introducing a differential term to predict violation trends and an integral term to eliminate steady-state errors, the system can dynamically and smoothly adjust the level of safety penalties, achieving an adaptive balance of boldly optimizing energy consumption in the safe region and smoothly applying constraints in the boundary region.

[0036] (5) This invention uses an edge computing gateway as the computing power carrier to realize local closed-loop control from data perception, reasoning decision-making to security verification. Combined with the graph induction capability of GNN, when the pipeline network undergoes physical expansion or topology change, it can be used plug-and-play without retraining the model, which greatly reduces the deployment and maintenance costs in industrial sites. Attached Figure Description

[0037] Figure 1 This is a hardware architecture diagram of the pipeline control method based on edge computing and topology-aware reinforcement learning disclosed in the embodiments of the present invention; Figure 2 This is a flowchart of the pipeline control method based on edge computing and topology-aware reinforcement learning disclosed in the embodiments of the present invention; Figure 3 This is a schematic diagram of the dynamic gating graph attention mechanism of the pipeline control method based on edge computing and topology-aware reinforcement learning disclosed in the embodiments of the present invention. Figure 4 This is a logic diagram for physical constraint verification and correction of the pipeline control method based on edge computing and topology-aware reinforcement learning disclosed in the embodiments of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1 like Figure 1 and Figure 2 As shown, Embodiment 1 of the present invention provides a pipeline control method based on edge computing and topology-aware reinforcement learning, the method comprising the following steps: S1: Construct a directed graph of the pipeline network; When the system starts, the intelligent edge computing gateway reads the pipeline configuration file (GeoJSON format) and abstracts the pipeline network into a directed graph G=(V,E). V represents the set of nodes, including pump stations, valves, and user nodes. The static feature vector of each node in the node set contains the node elevation and node type code. The node elevation mainly refers to the altitude of the pipeline, representing the geometric elevation of the pipeline node in physical space. It characterizes the gravitational potential energy level of the node and is an important physical parameter affecting pressure distribution and hydraulic coupling. This helps accurately determine whether there is blockage or abnormal friction inside the pipeline, preventing the model from misjudging normal climbing pressure as a fault. E represents the set of edges, representing the physical pipeline and characterizing the connection relationships between nodes. The feature vector of each edge... This includes the pipe length L, pipe diameter D, and Heisenberg-Williams roughness coefficient C.

[0040] S2: State Awareness and Data Mapping: To provide real-time physical state input for the subsequent graph neural network, this step mainly completes the mapping from raw sensor data to graph-structured data. Specifically, it collects pipeline pressure, flow rate, and valve opening, and constructs node feature vectors and edge action state vectors. S2 includes the following sub-steps: S21, The nodes collected by the sensor at time t pressure ,flow The updated node attributes in the directed graph G, together with the static feature vectors (elevation, type) from step S1, constitute the node. eigenvectors .

[0041] S22, the valve opening at time t Update the corresponding edge attributes in the directed graph G as the edge action state vector.

[0042] The above data together complete the construction of the input features for the graph attention network, preparing the data for step S3.

[0043] S3: Topological feature encoding (i.e., the dynamic gating graph attention mechanism for action perception), such as Figure 3 As shown, in order to extract the spatial coupling features of the pipeline network that include hydraulic characteristics and equipment action effects, this embodiment does not use a general GNN, but instead constructs an action-aware dynamic gated graph attention network (Action-Gated GAT). This invention introduces a dynamic gating factor as a soft switch, mapping the real-time state of the actuator to the attention gating of the neural network. The specific process of S3 is as follows: S31. Input Feature Construction: Read the graph structure data constructed in step S2, including nodes. eigenvectors (Integrating spatiotemporal attributes such as elevation and pressure) and edge action state vectors (i.e., valve opening), which serves as the basis for calculating the dynamic gating factor.

[0044] S32. Calculating Dynamic Gating Factors: In existing technologies, general graph attention networks (GAT) rely solely on data similarity in high-dimensional feature spaces when calculating weights. However, fluid transmission in industrial pipelines is strictly constrained by physical properties. To "implant" this physical law into the neural network, this embodiment uses the Hazen-Williams empirical formula, commonly used in hydraulics, as the mapping benchmark and directly reuses its core empirical constants (4.87 and 1.852). By introducing these two specific physical empirical constants, based on the physical law that flow transmission capacity is directly proportional to the 4.87th power of pipe diameter D and inversely proportional to the 1.852nd power of pipe length and roughness, the attention weight constraints of the neural network are constructed. By introducing these two specific physical empirical constants and their corresponding structures, the network possesses the ability to distinguish between hydraulic main channels and terminal branches during the initialization phase, thereby avoiding the problem of blind searching in the early training stage of a purely data-driven model. The opening and closing of valves is the decisive factor for fluid flow. This invention uses the real-time valve opening degree... As a linear truncation term, coupled with the aforementioned physical properties, the following dynamic gating factor formula is constructed:

[0045] in, , , These represent the pipe length, pipe diameter, and Hessian-Williams roughness coefficient between node i and its neighbor node j, respectively. It is a constant, a minimum value to prevent numerical overflow; This is the Sigmoid activation function. Characterizes the maximum physical transport potential of the pipeline, through A linear truncation relationship was established between the inherent geometric capabilities of the pipeline and its real-time operational state. This coupling ensures that when the valve is closed... At that time, regardless of the tube diameter D, the numerator term is suppressed to a value of 0. The defined lower bound of the value closely approximates the open circuit state in the physical world at the numerical computation level, simulating the blocking effect while avoiding the singularity of the computation matrix or the vanishing gradient problem that may be caused by directly taking zero. By placing pipe length and roughness in the denominator, an inverse attenuation relationship for information transmission is established. This aligns with the objective law in fluid mechanics that the longer the pipe or the rougher the inner wall, the greater the head loss along the pipe. This forces the neural network to reduce its attention weight on long-distance, high-resistance neighbor nodes, making it more inclined to aggregate information from short-distance, low-resistance strongly connected nodes.

[0046] To address the shortcomings of traditional GAT (Generative Attraction and Data Acquisition) which only considers data similarity and ignores physical connectivity, this step designs an aggregation mechanism with dual data and physical constraints. An attention mechanism is used to capture the correlation of operational data such as pressure and flow between nodes. A calculated dynamic gating factor serves as the physical veto power, and the two are fused through multiplicative modulation: that is, information from neighboring nodes is only allowed to be aggregated when both data similarity and physical connectivity are simultaneously satisfied.

[0047] When calculating the aggregation of neighbor node j information by node i, the attention coefficient is multiplicatively modulated using a dynamic gating factor. The calculation process is as follows: 1) Feature Linear Transformation and Concatenation: Using a learnable feature transformation matrix W, the feature vector of node i is transformed into a linear concatenation matrix. and the feature vector of neighbor node j Mapping to a high-dimensional feature space and concatenating the transformed features to construct feature pairs, also known as concatenated features:

[0048] in, Indicates feature splicing, For neighboring nodes eigenvectors; 2) Physically Gated Modulation (Calculating Attention Scores): Utilizing linear mapping vectors After linear mapping of the spliced ​​features and activation by LeakyReLU, they are directly multiplied by a dynamic gating factor. , obtain node with neighboring nodes Attention score between:

[0049] in, It is a linear activation function with leakage.

[0050] When the valve is closed or the physical resistance is extremely high, it leads to At that time, the attention score of the neighboring node It will be forcibly compressed to near 0, thus achieving physical severance at the algorithm level.

[0051] 3) The scores of all neighboring nodes are exponentially and normallyized using the Softmax function structure to obtain the final node. with neighboring nodes Attention weights between :

[0052] in, Let be the set of neighboring nodes of node i. Represents the neighboring nodes of node i , For nodes with neighboring nodes Attention score between them.

[0053] 4) Feature aggregation and state update: Utilizing the calculated attention weights The node feature vectors of neighboring nodes are weighted and summed, and then the node features for the next time step are generated using a non-linear activation function.

[0054] The resulting feature vector not only encodes the current local pressure and flow status, but also implicitly encodes complex topological semantic information such as the actual connectivity between nodes and the main path of hydraulic transmission, which significantly improves the prediction accuracy and convergence speed of the model under complex scheduling conditions.

[0055] S4: Input the node features (i.e., global topological features) of all nodes at all times obtained in step S3 into the deep reinforcement learning network. The deep reinforcement learning network includes a policy network and a value network. The policy network is used to predict the initial action vector, and the value network is used to estimate the value of the policy network's output. The policy network and value network are trained together, and finally, the trained policy network is used to output the initial action vector. Meanwhile, to address the problem of balancing safety and economy in traditional reinforcement learning with a fixed penalty coefficient (too large a coefficient leads to model inaction, too small a coefficient leads to frequent violations), this embodiment introduces an adaptive update mechanism for the penalty coefficient. The specific process of S4 is as follows: S41. Network Definition: This step constructs a deep neural network based on the Actor-Critic (Policy Network-Value Network) architecture to achieve the mapping from high-dimensional features to physical decisions. The working principle of the deep neural network is as follows: S411. The policy network receives the global topological features output in step S3, and maps the global topological features of the state to the probability distribution of actions through a multilayer perceptron (MLP) to obtain the preliminary action vector. To map the output of the neural network into valid physical control commands, the output layer of the policy network is specifically designed as follows: (1) Dimension setting: The number of neurons in the output layer is set to N = K + M, which strictly corresponds to K variable frequency pumps and M electric regulating valves in the pipeline network; (2) Physical mapping: For the dimension corresponding to the electric regulating valve, the Sigmoid activation function is used to limit the output to the (0, 1) interval, corresponding to 0%-100% opening degree; for the dimension corresponding to the variable frequency pump, the Tanh activation function is used in conjunction with linear transformation to map the output to the high-efficiency operating frequency range of the equipment.

[0056] Based on the above distribution sampling, a preliminary action vector is obtained. :

[0057] in, Let K be the normalized rotational speed of the Kth pump station. For the first The normalized opening degree of an electric regulating valve.

[0058] S412. The global topological features H (containing features of all nodes at all times) output from step S3 of the value network receiving step can also adopt an MLP architecture, but the output is designed as a single-neuron regression structure to obtain a scalar representing the state value. .

[0059] S42. Objective Function and Dual Update: This system is constructed with dynamic penalty coefficients. The optimization objective is as follows: To collaboratively train the Actor and Critic, this embodiment defines the following optimization objective: 1) Original reward objective function (maximizing energy efficiency and stability):

[0060] in, Discount factor; This refers to the real-time operating power of the pumping station. Rated power; The real-time pressure of node i; Set the pressure for node i; , These are the normalized weighting coefficients. For the current moment, The time window length, As expected, The modulus of the set of nodes; 2) Physical constraint cost function (minimizing the violation):

[0061] in, , These are the upper and lower limits of the permissible safe pressure for pipeline nodes. 3) Update objective of the value network: The value network aims to accurately predict the value of a state. Its update objective is to minimize the prediction error. As an auxiliary evaluator, it updates its trainable parameters. The aim is to minimize the prediction error of state values.

[0062]

[0063]

[0064] Where N1 is the batch sample size. For the trainable parameters of the value network, express time, For the actual return, calculate from the current time t to... All rewards at any time The present value of the discounted total; For value networks based on the global topological features of the input The predicted value obtained.

[0065] 4) The update objective of the policy network: As the decision-making agent, the policy network updates its trainable parameters. The aim is to find a method that maximizes long-term cumulative rewards while satisfying physical constraints. The action.

[0066]

[0067] in, For the trainable parameters of the policy network in deep reinforcement learning; To allow for small tolerances. This is the penalty coefficient. This penalty coefficient can be obtained through optimization using S43.

[0068] In this embodiment, the trainable parameters of the value network and the trainable parameters of the policy network are continuously adjusted, and the update targets of both are calculated. Training is stopped when the update target of the value network is minimized and the update target of the policy network is maximized, and a well-trained deep reinforcement learning network is obtained.

[0069] S43, PID multiplier update: for dynamically finding the optimal... This embodiment no longer uses simple gradient ascent, but introduces PID control principles to reduce oscillations during the training process:

[0070] in, This represents the physical constraint violation error value within the m-th training iteration period; , , These are the proportional coefficient, integral coefficient, and differential coefficient; proportional coefficient When a sudden and serious violation occurs, the price will be quickly raised. This forces the strategy to immediately return to the safe zone; integral coefficient ( ): Eliminate the cumulative error of small violations that have existed for a long time; differential coefficients ( ): Predicting trends of violations and suppressing fluctuations in advance. Let be the penalty coefficient for the m-th training iteration. Initially, a single penalty coefficient is set, and subsequent penalty coefficients are iteratively updated according to the PID control principle. This will serve as the penalty coefficient for the next round of strategy network optimization. Through this dynamic feedback mechanism, the strategy network can perceive the severity of physical constraints in real time, thereby adaptively adjusting the optimization step size to improve economic efficiency while ensuring the inherent safety of the pipeline network.

[0071] Step S5: Physical constraint verification and correction, such as... Figure 4 As shown, to ensure inherent safety in industrial settings, the system uses a physical constraint safety module to modify the initial action vector before issuing actions. Two-level processing: S51, Level 1 Hard Constraint (Projection Correction): For variable frequency pumps, based on their HQ characteristic curve, if the speed in the initial action vector... Then proceed to soft constraint verification; if the rotational speed in the initial motion vector... Then, after performing the projection operation, the soft constraint verification will proceed. The projection operation is as follows:

[0072] in, , These are the upper and lower limits of the high-efficiency operating range dynamically calculated based on the HQ characteristic curve of the variable frequency pump. The (·) function is a truncation function. The specific truncation logic is: if x < Then take If x> Then take Otherwise, keep x unchanged; x is the initial action vector output by the policy network. This corresponds to the original control command value of the actuator being verified (such as a specific water pump or valve). This value is directly generated by the neural network and has not been subject to physical boundary constraints, and may contain illegal values ​​that exceed the physical limits of the equipment (such as overclocking or negative opening). For valves, the rate of change of single-step action Δ is limited. To prevent water hammer effect.

[0073] S52, Second-order soft constraints (prediction penalty based on differentiable physical surrogate model): In order to overcome the shortcomings of traditional hydraulic simulation (such as the Newton-Raphson iterative method) in terms of computation time and non-differentiability, this embodiment innovatively constructs a physical information-based neural network surrogate model to replace the traditional solver.

[0074] S521, Neural Network Agent Model Construction: This model is a pre-trained multilayer perceptron (MLP) or graph neural network.

[0075] Input: Current network topology status and action commands after first-level hard constraint processing. ; Output: Predicted network node pressure distribution .

[0076] Training loss function (weighted based on betweenness centrality): Considering the dominant role of key nodes in the pipeline network on the global hydraulic balance, an adaptive weighted loss function based on betweenness centrality is constructed:

[0077] in, The mean square error of the data; The physical residual at node i (i.e., the degree to which the continuity equation and energy equation are violated); Let i be the betweenness centrality index of node i; These are the global weighting coefficients for the physical constraint terms; This is an adjustment coefficient. This function allows the surrogate model to preferentially fit the physical laws of the main pipeline network during the training phase, improving the prediction reliability of the model at key nodes. To quantify the above indicators, this embodiment provides a specific function definition: (1) Mean square error of data:

[0078] Where N1 is the batch sample size; The prediction pressure output by the neural network surrogate model at node i; Let i be the actual label corresponding to node i; (2) Physical residuals:

[0079]

[0080] in, Let i be the set of neighboring nodes of node i; The basic water demand of node i; The flow rate of the pipe segment connecting node i and its neighbor node j; , , , respectively, represent the pipe length, pipe diameter, and Hessen-Williams roughness coefficient between node i and its neighbor node j; k2 is the unit conversion factor for the Hessen-Williams formula; The pipe diameter index is taken as the standard value of 2.63, which is the standard pipe diameter index in the Hazen-Williams formula. The pressure gradient exponent, with a standard value of 0.54, is the reciprocal of the Haicheng-Williams head loss exponent (1.852), used to characterize the nonlinear response of flow rate to pressure drop; 1- This is a numerical stability compensation term used to ensure the mathematical analyticity of the physical residuals under negative pressure differential conditions while preserving the directionality of the pressure gradient. (3) Betweenness centrality index:

[0081] in, This represents the number of paths that pass through node i and are the shortest paths. The number of shortest paths connecting the starting node s and the ending node z; S522, Correction process based on automatic differentiation: If the neural network surrogate model predicts that the pressure at any node exceeds the safety threshold... (Right now > If a violation occurs, it is considered a soft violation, and end-to-end gradient correction is performed using the differentiability of the neural network surrogate model. (1) Initialization: Redirecting action instructions that violate soft constraints As the current iteration action ; (2) Gradient calculation: Construct a penalty loss function for exceeding the pressure limit. By utilizing the automatic differentiation mechanism of deep learning frameworks, the precise gradient of the loss function with respect to the input action vector is directly calculated through backpropagation. This gradient indicates the optimal direction for motion adjustment, and is calculated using the following formula:

[0082] in, The sign of the partial derivative; (3) Iterative Update (Introduction of Mechanical Inertia Constraint): To prevent the actuator from moving too fast and causing water hammer due to simply pursuing the mathematically optimal solution, this embodiment introduces a mechanical inertia smoothing term in the gradient update. The calculation formula is:

[0083] This step ensures that the generated correction instructions meet both hydraulic safety requirements and the mechanical operating constraints of the equipment. Among these, Indicates the gradient clipping threshold. Indicates the learning rate; This represents the momentum factor.

[0084] (4) Terminate verification: Update the action. The neural network surrogate model is input again for prediction. If the new prediction pressure satisfies... Then stop the search and will As the final security directive Issued; otherwise, The iteration continues, returning the gradient calculation step as the current iteration action, until the constraints are satisfied.

[0085] S53, Training Feedback Closed Loop: It is worth noting that the calculated soft constraint violation amount (i.e., physical residual or stress limit exceeding value) This not only serves to correct the current action, but also acts as a crucial negative feedback signal, feeding back to the deep reinforcement learning policy network in step S4, forming a dual execution-training closed loop. (1) Signal feedback: The violation error detected in S5 is transmitted back to the signal feedback system. It is synchronized in real time to the PID controller described in S4.

[0086] (2) Parameter update: The controller updates parameters based on... Dynamic updates This allows for the adjustment of penalty weights in the reinforcement learning objective function.

[0087] (3) Mechanism effect: Through this mechanism, the system not only achieves millisecond-level online security interception (execution side), but also forces the policy network to gradually learn to autonomously avoid high-risk areas during long-term training through backpropagation, thereby achieving offline policy optimization (training side). Ultimately, the system achieves dual protection by preventing risks in advance through the policy network and blocking risks in real time through the security module.

[0088] S6: Closed-loop execution and dynamic updates The intelligent edge computing gateway will verify the final instructions. Convert the signal to a 4-20mA current signal or Modbus register write instruction, and send it to the PLC for execution.

[0089] Meanwhile, if the pipeline undergoes physical changes (such as the addition of new pipeline segments), the topology maintenance module within the gateway only needs to update the node set V and edge set E in step S1. Without retraining the neural network parameters, it can directly perform inference on the new topology using the inductive bias capability of GNN, achieving plug-and-play functionality.

[0090] Through the above technical solutions, this invention analyzes the physical connection relationships of the pipeline network to construct a dynamic graph structure, mapping real-time pressure and flow data collected by the terminals into graph node features. Utilizing a dynamic gating graph attention mechanism based on action perception, a dynamic gating factor controlled by valve opening is introduced as an attention switch to extract global topological features including pipeline hydraulic coupling and equipment operating states. Through a deep reinforcement learning strategy network based on PID dual updates, the constraint penalty coefficient is dynamically adjusted using a PID controller to output preliminary control actions. These actions are verified by a safety module containing hard constraint projection and a differentiable physical proxy model. After calculating the pressure gradient using an automatic differentiation mechanism and performing backpropagation correction, they are transformed into control commands for issuance. The overall solution, through the deep integration of physical mechanisms and artificial intelligence, solves the problems of traditional methods being unable to adapt to dynamic changes in pipeline topology and lacking physical safety guarantees, achieving global energy optimization and inherent safety in complex pipeline network operation.

[0091] Example 2 Based on Embodiment 1, Embodiment 2 of the present invention provides a pipeline control system based on edge computing and topology-aware reinforcement learning. (Continue reading...) Figure 1 The system includes an intelligent edge computing gateway, a distributed sensing terminal, and an actuator; The intelligent edge computing gateway, serving as the core control unit, employs an industrial-grade embedded computer equipped with a high-performance GPU acceleration module (such as the NVIDIA Jetson series) and industrial communication interfaces (including RS485 and dual-channel gigabit Ethernet ports). The gateway runs a Linux operating system and a deep learning inference framework (such as PyTorch) to execute the method proposed in Example 1.

[0092] The distributed sensing terminals include pressure transmitters and ultrasonic flow meters. The pressure transmitters are model 3051S, deployed at various nodes in the pipeline network (pump station inlets and outlets, and before and after valves). The sampling frequency is set to 1Hz to collect real-time node pressure data. Ultrasonic flow meters are deployed in the main pipe sections to collect flow rate data. The data collected by the distributed sensing terminals is transmitted to the intelligent edge computing gateway.

[0093] The actuators include a variable frequency drive (VFD) cabinet and an electric regulating valve. The VFD cabinet connects to an intelligent edge computing gateway via the Modbus-TCP protocol, receiving frequency commands from 0-50Hz to control the centrifugal pump speed. The electric regulating valve has a 4-20mA analog feedback function, receiving 0-100% opening commands.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pipeline control method based on edge computing and topology-aware reinforcement learning, characterized in that, include: S1. Construct a directed graph of the pipeline network; S2. Collect pipeline pressure, flow rate, and valve opening to construct node feature vectors and edge action state vectors; S3. The graph attention network calculates the dynamic gating factor using the edge action state vector, uses the dynamic gating factor to multiply the attention coefficient to obtain the attention weight, uses the attention weight to perform a weighted summation of the node feature vectors of neighboring nodes, and generates the node features of the next time step through a nonlinear activation function. The node features of all nodes at all time steps are used as the global topological features. S4. Construct a deep reinforcement learning network that includes a policy network and a value network. Input global topological features into the policy network to predict the initial action vector, and use the value network to estimate the value of the policy network's output. Train the policy network and the value network together, and finally use the trained policy network to output the initial action vector. S5. Set hard and soft constraints to verify the legality of the preliminary action vector and output the final control command; S6. The final control command is sent to the actuator, and the actuator executes the final control command.

2. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 1, characterized in that, S1 includes: The pipeline network is constructed as a directed graph G=(V,E), where V represents the set of nodes, including pump stations, valves, and user nodes. The static feature vector of each node in the set contains the node elevation and node type code. E represents the set of edges, representing physical pipelines and characterizing the connection relationships between nodes. The feature vector of each edge... This includes pipe length L, pipe diameter D, and Heisenberg-Williams roughness coefficient C.

3. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 2, characterized in that, S2 includes: S21, the nodes collected at time t pressure ,flow Updated to the node attributes of the directed graph G, together with the static feature vectors, constitute the node. eigenvectors ; S22, the valve opening at time t Update the corresponding edge attributes in the directed graph G as the edge action state vector.

4. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 3, characterized in that, S3 includes: Using formula Calculate the dynamic gating factor, where, , , They are nodes with neighboring nodes The pipe length, pipe diameter, and Heisenberg-Williams roughness coefficient between them; It is a constant; Use the Sigmoid activation function; Using formula Obtain the splicing features, where, The characteristic transformation matrix, Indicates feature splicing, Neighboring nodes eigenvectors; Using formula compute nodes with neighboring nodes Attention scores between, For linear mapping vectors, It is a linear activation function with leakage; Using formula Get Nodes with neighboring nodes Attention weights between them For nodes The set of neighboring nodes, Represents a node neighboring nodes , For nodes with neighboring nodes Attention score between; Using formula Generate node features for the next time step.

5. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 1, characterized in that, The process of co-training the policy network and the value network is as follows: Construct the original reward objective function: in, Discount factor; This refers to the real-time operating power of the pumping station. Rated power; The real-time pressure of node i; Set the pressure for node i; , These are the normalized weighting coefficients. For the current moment, The time window length, As expected, The modulus of the set of nodes; Construct the physical constraint cost function: in, , These are the upper and lower limits of the permissible safe pressure for pipeline nodes. for Activation function; Set the update target for the value network : in, This represents the batch sample size. For the trainable parameters of the value network, express time, For the actual return, calculate from the current time t to... All rewards at any time The present value of the discounted total; For value networks based on the global topological features of the input The predicted value obtained; Set the update target for the policy network. : in, These are the trainable parameters of the policy network; For tolerance, This is the penalty coefficient; The trainable parameters of the value network and the trainable parameters of the policy network are continuously adjusted, and their update targets are calculated. Training is stopped when the update target of the value network is minimized and the update target of the policy network is maximized, thus obtaining a well-trained deep reinforcement learning network.

6. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 5, characterized in that, The penalty coefficient is iteratively updated using a PID controller, and the formula is expressed as follows: in, This represents the physical constraint violation error value within the m-th training iteration period; , , These are the proportional coefficient, integral coefficient, and differential coefficient, respectively. Let be the penalty coefficient in the m-th training iteration period. This serves as the penalty coefficient for the policy network during the (m+1)th training iteration.

7. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 1, characterized in that, The hard constraints include: For variable frequency pumps, the speed range is defined based on their HQ characteristic curve. If the rotational speed in the initial motion vector Then proceed to soft constraint verification; if the rotational speed in the initial motion vector... Then, after performing the projection operation, the soft constraint verification will proceed. The projection operation is as follows: in, , These are the upper and lower limits of the high-efficiency operating range dynamically calculated based on the HQ characteristic curve of the variable frequency pump. This is a truncation function.

8. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 7, characterized in that, The soft constraints include: A neural network proxy model is constructed, wherein the input of the neural network proxy model is the action command after hard constraint processing, and the output is the predicted pressure distribution of all nodes in the network. The neural network proxy model is a pre-trained multilayer perceptron or graph neural network; a safety threshold is set. ,like > If this is not the case, it is considered a soft violation, and an end-to-end gradient correction is performed. The modification process is as follows: The action instruction after soft violation processing with hard constraints is used as the current iteration action. ; Construct a penalty loss function for exceeding the pressure limit. The gradient of the loss function with respect to the input action vector is calculated through backpropagation. ; The formula for calculating the next iteration action is: in, Indicates the gradient clipping threshold. Indicates the learning rate; Indicates the momentum factor; Will The neural network surrogate model is fed back into the model for prediction. If the new prediction pressure satisfies... Then stop the search and will As the final security directive Issued; otherwise, The step of returning the gradient calculation as the current iteration action continues to the next iteration until the constraints are satisfied.

9. The pipeline control method based on edge computing and topology-aware reinforcement learning according to claim 8, characterized in that, The training process of the neural network proxy model is as follows: A loss function for the neural network surrogate model is constructed, and the parameters of the neural network surrogate model are continuously adjusted until the loss function of the neural network surrogate model is minimized. Training is then stopped, resulting in a trained neural network surrogate model. The loss function of the neural network surrogate model is: in, The mean square error of the data; The physical residual at node i; Let i be the betweenness centrality index of node i; These are the global weighting coefficients for the physical constraint terms; This is the adjustment coefficient; Where N1 is the batch sample size; The prediction pressure output by the neural network surrogate model at node i; Let i be the actual label corresponding to node i; in, Let i be the set of neighboring nodes of node i; The basic water demand of node i; The flow rate of the pipe segment connecting node i and its neighbor node j; , , , respectively, represent the pipe length, pipe diameter, and Hessen-Williams roughness coefficient between node i and its neighbor node j; k2 is the unit conversion factor for the Hessen-Williams formula; Pipe diameter index; For pressure gradient exponent; 1- This is a numerical stability compensation term; in, This represents the number of paths that pass through node i and are the shortest paths. This represents the number of shortest paths connecting the starting node s and the ending node z.

10. A pipeline control system based on edge computing and topology-aware reinforcement learning, characterized in that, include: An intelligent edge computing gateway for running the pipeline control method based on edge computing and topology-aware reinforcement learning as described in any one of claims 1-9; Distributed sensing terminals are used to collect real-time node pressure and pipeline flow and transmit them to the intelligent edge computing gateway. The actuator is used to execute the final control commands issued by the intelligent edge computing gateway.

Images