A plain river network lock group intelligent regulation and control method based on deep reinforcement learning

Abstract

The application discloses a kind of plain river network gate group intelligent regulation and control methods based on deep reinforcement learning, including the construction flow prediction module, the knowledge constraint reinforcement learning module in combination with field knowledge and experience base and multi-agent reinforcement learning module;The knowledge constraint reinforcement learning module calibrated with flow prediction module, multi-agent reinforcement learning module is integrated, and the complete plain river network intelligent regulation and control model is formed;Based on the calibrated plain river network intelligent regulation and control model, access future meteorological hydrology prediction data, and quickly output reasonable gate group scheduling scheme.The advantages are: intelligent generation scheduling scheme cluster, efficiently deal with complex, dynamic change plain river network gate group scheduling problem, improve the flexibility, timeliness and intelligent level of gate group scheduling system, solve the "non-real solution" problem that may exist in practical application of machine learning method, enhance the explainability of result and the actual operability of scheme, form "model-environment-feedback-regulation correction" adaptive closed loop.

Description

Technical Field

[0001] This invention relates to the field of plain river network gate group control technology, and in particular to an intelligent control method for plain river network gate groups based on deep reinforcement learning. Background Technology

[0002] Currently, the regulation of sluice gate groups in plain river networks mainly relies on hydrodynamic models. Fixed operating condition combinations are manually set (such as opening and closing rules based on water level thresholds or time series), and forward calculations are performed based on upper and lower boundary conditions of the hydrological data to generate a limited, discrete cluster of scheduling schemes. This is insufficient to cover the complex and ever-changing actual operating conditions. Furthermore, scheme generation depends on expert experience, has a low level of intelligence, and requires repeated model trials and verification, resulting in high computational load, low efficiency, and difficulty in dynamically adapting to changing environmental conditions, especially the complex water-rainfall-engineering conditions influenced by climate change and human activities.

[0003] With the rapid development of artificial intelligence technology, deep learning methods have gradually become a research hotspot in the field of water conservancy scheduling. Using deep learning models to learn and train on inflow / operating conditions and the water level / flow process at the target section, a mapping relationship between inflow, operating conditions, and the regulated object is established. Furthermore, relying on the deep learning model, gate control scheduling schemes can be quickly simulated and generated, greatly improving the intelligence level of scheduling decisions. However, flow prediction and river network gate group regulation based on deep learning models still have limitations. Their "black box" nature leads to a lack of interpretability in the output results. In addition, since the models typically learn patterns only from time-series data and cannot directly quantify unstructured control rules, the model output may not conform to physical laws such as hydraulic characteristics, resulting in infeasible solutions, i.e., a lack of operable scheduling schemes. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning, thereby solving the aforementioned problems existing in the prior art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for intelligent regulation of sluice gate groups in plain river networks based on deep reinforcement learning includes the following steps:

[0007] S1. Functional Module Construction: A flow prediction module is built by coupling a Transformer encoder and a deep neural network (DNN); by combining domain knowledge and an experience base, empirical rules in the water conservancy field are embedded into the reinforcement learning strategy through physical and hydrodynamic constraints. Each gate is treated as an individual agent, and the agent learns the optimal strategy through interaction with the environment, thus building a knowledge-constrained reinforcement learning module that combines domain knowledge and an experience base; by introducing global constraints into the reward function of each agent, the decisions of all agents are ensured to be adjusted towards the global optimal goal, realizing multi-agent collaboration, thus building a multi-agent reinforcement learning module.

[0008] S2. Construction of Intelligent Regulation Model for Plain River Network Gate Groups: The calibrated knowledge constraint reinforcement learning module is integrated with the flow prediction module and the multi-agent reinforcement learning module to form a complete intelligent regulation model for plain river network.

[0009] S3. Intelligent Regulation and Scheme Analysis of Sluice Gate Groups in Plain River Networks: Based on the calibrated intelligent regulation model of plain river networks, and by incorporating future meteorological and hydrological forecast data, a reasonable sluice gate group scheduling scheme can be quickly output.

[0010] Preferably, the process of building the traffic prediction module is as follows:

[0011] S111, Transformer encoder: The time-series input includes the gate opening degree at each time step, the collected rainfall data at the corresponding time step, and the upstream water inflow of the river segment. Where, x t It is a multivariate time series;

[0012] Convert time series into spatiotemporal joint embeddings

[0013] z t =Embedding(x t )+PositionalEncoding(t)+ChannelEncoding(c)

[0014] Among them, z t For the spatiotemporal joint features after embedding; x t t represents the input features at time t; c represents the variable type label, used to distinguish different hydrological feature channels; Embedding is a linear projection layer, used to map the original features to a high-dimensional space; PositionalEncoding is a temporal position encoding, used to preserve temporal information; ChannelEncoding is a feature type encoding, used to distinguish different hydrological variables;

[0015] Hydraulic constraints are embedded in the Transformer's self-attention layer, focusing on feature associations that conform to hydraulic laws to improve computation.

[0016] Q = W q Z

[0017] K = W k Z

[0018] V = W v Z

[0019]

[0020] Where ATTENTION and Sofmax are attention calculation mechanisms; Q, K, and V are the Query, Key, and Value matrices, respectively; W q W k W v Trainable weight matrices for the Query, Key, and Value matrices, respectively; Δh t K represents the difference in water levels between the upstream and downstream areas. T d is the transpose of the key matrix; k Z represents the dimension of the Key matrix; Z represents the hidden state matrix.

[0021] The Transformer encoder transforms data through residual connections and a feedforward network, ultimately outputting a comprehensive feature sequence data H.

[0022] S112. Deep Neural Network (DNN): Based on the acquired river network gate opening and key cross-section simulated flow data, the gate opening and upstream water inflow at each time step are used as input features, and the cross-sectional flow at the corresponding time step is used as the label to form a dataset Y∈R. n Where n is the number of samples;

[0023] The dataset is processed through multiple nonlinear transformations to extract features step by step. The output of the l-th layer is represented as follows:

[0024] g (l) =j (l) (K (l) g (l-1) +e (l) )

[0025] Among them, g (l) g (l-1) K represents the intermediate feature values ​​extracted from the dataset through the l-th and l-1-th hidden layers of the DNN, respectively. (l) and e (l) These are the weight matrix and the bias vector, respectively; j (l) It is a non-linear activation function;

[0026] The prediction results of the flow at the key section The flow rate is compared with the actual cross-sectional flow rate y, and L1 and L2 regularization methods are applied. Large weights are penalized by adding the sum of squared weights and the sum of absolute values ​​of the weights to the loss function; the loss function is:

[0027]

[0028] in, λ1 represents the mean squared error loss; λ2 and λ1 are the regularization coefficients of the corresponding terms. For L1 regularization terms; For L2 regularization terms; L total This represents the total loss value.

[0029] S113, Transformer-DNN Coupled Model: Couples Transformer and DNN. Transformer first processes time series data and extracts time series features. Then, these features are input into DNN for higher-level feature fusion and prediction. If the comprehensive feature sequence output by Transformer is H, DNN receives the flattened result of H and makes predictions to obtain the key section flow data.

[0030]

[0031] in, Predicted flow values ​​for key cross-sections; f DNN is the mapping function of the DNN, which takes a flattened temporal feature vector, performs multiple nonlinear transformations and feature fusion, and outputs the final predicted value; Flatten is the flattening operation, which transforms matrix H into a one-dimensional vector; H is the comprehensive feature sequence data output by the Transformer.

[0032] Preferably, after step S113, the method further includes:

[0033] S114. Input priority classification mechanism: The input data is stored in layers, and an input priority classification mechanism is set. The scheme clusters calculated by step S113 are set to low priority. Based on the scheme clusters output after optimization calculation by the Transformer-DNN coupled model, the scheme clusters that have been screened and retained are set to high priority to optimize the training results of the low priority scheme clusters.

[0034] Preferably, the process of building the knowledge-constrained reinforcement learning module that combines domain knowledge and experience base is as follows:

[0035] S121. Basic function settings: The environmental state space S is a discrete finite non-empty set, which is the situation composed of the flow and water level information of each section; the action space A is also defined as a finite set, which is the sum of actions or behaviors taken by the gate agent in the river network environment to achieve optimal scheduling of the river network, including the gate opening degree and the corresponding opening timing.

[0036] S = {s0, s1, ... s} t ...,s N}

[0037] A = {a0, a1, ... a} t ...,a M}

[0038] Where N is the size of the environmental state space; M is the size of the space for the actions taken by the agent;

[0039] At time t during the decision-making process, the gate agent is in environmental state s. t Making a decision and taking action is a t The environmental state will be determined by s t Transition to the next state s t+1 and receive a reward r t+1 The transition of its environmental state is determined by the transition function f: S×A→S. Based on the transition, if the agent transitions to a definite next state, then...

[0040] s t+1 =P(s) t a t )

[0041] In a real river network environment, the environmental state s is entered. t+1 The probability of is in the interval S, satisfying

[0042] S122. Setting constraints for the scheduling experience base: Add constraints based on domain knowledge of gate scheduling so that the reward function not only reflects the performance of the gate agent, but also its interpretability under domain knowledge constraints; adjust the agent's behavior by introducing knowledge constraints into the reward function.

[0043] r′(s t ,a t )=r(s t ,a t )+λC(s t ,q t )

[0044]

[0045] Wherein, C(s)t ,a t ) represents the domain-knowledge-based constraint function, which is a rule formulated based on water level constraints, flow rate constraints, and operational constraints; λ is a hyperparameter controlling the importance of constraints; h t h represents the water level at the current time t. max h represents the maximum allowable water level limit at the current time t. min Q represents the minimum water level limit allowed at the current time t. t Q represents the flow rate at the current time t. max Q represents the maximum allowed flow rate at the current time t; min G represents the minimum allowable flow rate at the current time t; t G represents the gate opening at time t; t-1 Let be the gate opening at time t-1; α, β, and β are the penalty weighting coefficients for exceeding the water level, flow rate, and gate opening limit, respectively;

[0046] Based on this, the constraints are gradually relaxed, allowing the gate agent to conduct more autonomous exploration and optimize scheduling strategies. Its mathematical form is:

[0047] r′(s t ,a t )=r(s t ,a t )+λ t C(s t ,a t )

[0048] Where, λ t The decay coefficient gradually decreases as time step t increases.

[0049] Preferably, in the knowledge-constrained reinforcement learning module that combines domain knowledge and experience base, a single gate is treated as an independent agent. The interaction process between the gate agent and the environment includes a state space S, an action space A, a reward function r(s,a), and a transition probability P(s′|s,a). The agent learns the optimal policy π through interaction with the environment. * (s), the objective is to maximize cumulative rewards;

[0050]

[0051] Where r(s) t ,a t ) represents state s t Take action a t Instant rewards; γ t Let be the discount factor of the agent at time t; π be the agent's policy.

[0052] Preferably, the construction process of the multi-agent reinforcement learning module is as follows: In the river network multi-agent system, the goal of each gate agent is to maximize its cumulative reward through reinforcement learning.

[0053]

[0054] in, The i-th gate agent is in state s i Take action a i State-action value function at time; It is the discount factor of the i-th gate at time t;

[0055] Each gate agent's scheduling decisions are influenced not only by its own constraints but also by coordination with other agents. Therefore, global constraints are introduced into each agent's reward function to ensure that all agents' decisions are aligned with the globally optimal goal. This is achieved by sharing constraint information among all agents, transforming the constraint reward function into...

[0056] r i ′(s i ,a i ) = r i (s i ,a i )+λC(s1,a1,…,s N ,a N )

[0057] Where, C(s1,a1,…,s) N ,a N ) is the global constraint function for the overall scheduling of the river network.

[0058] Preferably, step S2 specifically includes the following:

[0059] S21. Integration with the flow prediction module: The flow prediction module receives hydrological and meteorological forecast rainfall data and upstream water inflow data of the river section. It uses a Transformer encoder to model the input state sequence, extracts time-series features, and outputs a key section flow prediction sequence for a future period with a time step of 1 hour. The predicted key section flow sequence serves as the environmental input for the knowledge-constrained reinforcement learning module to optimize control decisions. The scheme family output by the knowledge-constrained reinforcement learning module is fed back to the flow prediction module for further correction and training.

[0060] S22. Integration with multi-agent reinforcement learning module: The single gate agent, which has been calibrated by knowledge constraint reinforcement learning, is embedded into the multi-agent deep reinforcement learning module. The multi-agent reinforcement learning framework is adopted to realize the collaborative decision-making of multiple agents, complete the dynamic optimization of river network flow regulation, and obtain a comprehensive scheduling scheme.

[0061] S23. Adaptive intelligent scheduling mechanism: Through the integration and coupling of steps S21 and S22, a time-continuous learning structure is formed to achieve an adaptive closed loop of "model-environment-feedback-regulation correction", thus forming an adaptive intelligent scheduling mechanism for the plain river network gate group and obtaining an intelligent regulation model for the plain river network.

[0062] Preferably, step S3 specifically includes the following:

[0063] S31. Intelligent regulation model training: Based on historical rainfall process, river network water level and flow, and gate scheduling process data, the data are input into the flow prediction module to drive the plain river network intelligent regulation model and construct a cluster of gate group regulation schemes.

[0064] S32. Setting of watershed objectives and constraints: Set the corresponding gate constraint parameters according to the scheduling objectives and operating conditions, so that the intelligent regulation model of the plain river network can complete reasonable scheduling under the given constraints, and achieve an adaptive closed loop of "model-environment-feedback-regulation correction".

[0065] S33. Analysis of River Network Sluice Gate Group Regulation Scheme: Based on the calibrated intelligent regulation model of plain river network, the system incorporates future meteorological and hydrological forecast data to output the water level and flow process of key sections and the cluster of sluice gate group scheduling schemes under the current upstream water inflow conditions. The time step is 1 hour to assist in and support scheduling decisions.

[0066] Preferably, the procedure before step S1 includes:

[0067] S0. Basic Data Collection and Processing: Through physical model simulation and historical data collection, obtain meteorological and hydrological time-series data, river network gate opening and key section simulated flow data required for river network scheduling, and preprocess the corresponding data.

[0068] Preferably, step S0 specifically includes the following:

[0069] S01. Acquisition of meteorological and hydrological time series data: The meteorological and hydrological measured historical data required for river network scheduling include upstream water inflow, river water level and flow data at key sections, and rainfall data obtained through manual observation and automated telemetry.

[0070] S02. Meteorological and hydrological time series data processing: A sliding window standardization process and time series alignment mechanism are adopted to unify the time step of the time series data to 1 hour, so as to achieve consistent alignment of multivariate data in the time dimension; For observation anomalies and missing points, reconstruction is completed by a dynamic repair algorithm based on spatial proximity and temporal proximity, thereby improving the temporal accuracy of the input data in subsequent location coding.

[0071] S03. Acquisition of simulated flow data for river network gate opening and key sections: Using a physical process model, different combinations of inflow processes and operating conditions are set up to conduct numerical simulation calculations of flood control under different typical flood events. Corresponding data sets of upstream inflow, gate opening and key section flow under different typical flood events and operating condition combinations are obtained, with a time step of 1 hour. A comparative sample library covering extreme inflow conditions and typical scheduling scenarios is constructed. The sample data in the sample library is encoded and labeled, and used as supervision signals for different control strategy categories during training and calculation, supporting the subsequent input channel encoding of the Transformer.

[0072] The beneficial effects of this invention are as follows: 1. The method of this invention constructs an intelligent agent-environment interaction mechanism with gate opening as the decision variable and water level and flow rate as state representations. It introduces a Transformer structure and joint embedding mechanism to extract spatiotemporal sequence features, utilizes deep reinforcement learning to automatically learn the optimal scheduling strategy, and introduces domain knowledge based on hydraulic and hydrological constraints to construct constraint functions. Compared with traditional gate scheduling methods, this method can intelligently generate a cluster of scheduling schemes, improving the flexibility, timeliness, and intelligence level of the gate group scheduling system. Compared with artificial intelligence models, this method effectively solves the problem of "unrealistic solutions" that may exist in machine learning in practical applications, forming an adaptive closed loop of "model-environment-feedback-regulation correction," enhancing the interpretability of the results and the practical operability of the scheme. 2. The method of this invention is not only applicable to the real-time joint scheduling of gate groups in complex plain river networks, but also has continuous learning capabilities. While ensuring the stability and security of regulation based on experience-based scheduling rules, it can also achieve minute-level rapid response, making it suitable for hydraulic regulation scenarios such as flood control and emergency response, providing support for the operation management and scheduling decision-making of next-generation intelligent water conservancy projects. Attached Figure Description

[0073] Figure 1 This is a flowchart of the intelligent control method in an embodiment of the present invention;

[0074] Figure 2 This is a schematic diagram illustrating the principles of each module of the intelligent control model in this embodiment of the invention;

[0075] Figure 3 This is a schematic diagram of the Markov decision-making process in an embodiment of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0077] To address the shortcomings of existing physical process models in terms of timeliness and intelligence, and the lack of interpretability and feasible solutions in artificial intelligence models, this embodiment constructs an intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning, combining Transformer encoders, deep neural networks (DNNs), reinforcement learning, and domain knowledge. This method intelligently generates a cluster of scheduling schemes, efficiently addressing the complex and dynamically changing scheduling problems of sluice gate groups in plain river networks. It improves the flexibility, timeliness, and intelligence of the sluice gate scheduling system, solves the problem of "unrealistic solutions" that may exist in the practical application of machine learning methods, enhances the interpretability of results and the practical operability of the schemes, and forms an adaptive closed loop of "model-environment-feedback-regulation correction." Figure 1 As shown, the method of the present invention specifically includes the following parts:

[0078] I. Basic Data Collection and Processing

[0079] By using physical model simulations and collecting historical data, corresponding data sets of upstream water inflow (flow rate or water level), river network gate group opening degree, and key section flow are obtained, providing a data foundation for conducting research on intelligent regulation of plain river network gate groups based on deep reinforcement learning.

[0080] 1.1 Acquisition of meteorological and hydrological time series data

[0081] The historical meteorological and hydrological data required for river network scheduling include: upstream water inflow (flow or water level), river water level and flow data at key sections, rainfall data, and other historical data, including manual observation data and automated telemetry data.

[0082] 1.2 Meteorological and Hydrological Time Series Data Processing

[0083] To meet the temporal consistency requirements of the embedded structure construction, a sliding window standardization process and a temporal alignment mechanism are adopted to unify the time step of the time series data to 1 hour, achieving consistent alignment of multi-source data in the temporal dimension. For observational anomalies and missing measurement points, a dynamic repair algorithm based on spatial and temporal proximity is used to complete the reconstruction, improving the temporal accuracy of the input data in subsequent location encoding.

[0084] 1.3 Acquisition of simulated flow data for river network gate opening and key cross-sections

[0085] Using physical process models such as MIKE, and setting different combinations of inflow processes and operating conditions, numerical simulations of flood control under various typical flood events were conducted. Corresponding data sets of upstream inflow (flow rate or water level), sluice gate opening degree, and key section flow were obtained under different typical flood events and operating condition combinations, with a time step of 1 hour. A comparative sample library covering extreme inflow conditions and typical scheduling scenarios was constructed. The sample data in the library were encoded and labeled, serving as supervisory signals for different control strategy categories during training and computation, supporting the subsequent input channel encoding of the Transformer.

[0086] II. Functional Module Setup

[0087] A flow prediction module is built by coupling a Transformer encoder and a deep neural network (DNN). A knowledge-constrained reinforcement learning module is constructed by integrating domain knowledge and an experience base, embedding empirical rules from the water conservancy field into reinforcement learning strategies through physical and hydrodynamic constraints. Each gate is treated as an independent agent, learning the optimal strategy through interaction with the environment. A knowledge-constrained reinforcement learning module is then built by incorporating global constraints into the reward function of each agent, ensuring that all agents' decisions align towards the globally optimal goal, thus achieving multi-agent collaboration and constructing a multi-agent reinforcement learning module. The principles of each functional module are described in [link to documentation]. Figure 2 .

[0088] 2.1 Traffic Prediction Module (Transformer + DNN)

[0089] (1) Transformer encoder design

[0090] The time-series input includes the gate opening degree, the rainfall information collected in step 1.1 for the corresponding time, and the upstream water inflow (flow rate or water level). Where x t It is a multivariate time series.

[0091] Convert time series data into spatiotemporal joint embeddings:

[0092] z t =Embedding(x t )+PositionalEncoding(t)+ChannelEncoding(c)

[0093] Among them, z t For the spatiotemporal joint features after embedding; x tt represents the input features at time t; c represents the variable type label (such as rainfall, flow rate, water level, gate opening, etc.) used to distinguish different hydrological feature channels; Embedding is a linear projection layer that maps the original features to a high-dimensional space; PositionalEncoding is a temporal position encoding that preserves temporal information; ChannelEncoding is a feature type encoding that distinguishes different hydrological variables.

[0094] Hydraulic constraints are embedded in the Transformer's self-attention layer, focusing on feature associations that conform to hydraulic laws, to improve computation:

[0095] Q = W q Z

[0096] K = W k Z

[0097] V = W v Z

[0098]

[0099] Where ATTENTION and Softmax are attention calculation mechanisms; Q, K, and V are Query, Key, and Value matrices; W q W k E v Trainable weight matrices for the Query, Key, and Value matrices, respectively; Δh t K represents the water level difference between upstream and downstream (m). T d is the transpose of the key matrix; k Z is the dimension of the Key matrix; Z is the hidden state matrix.

[0100] The Transformer encoder transforms data through residual connections and a feedforward network, ultimately outputting a comprehensive feature sequence data H.

[0101] (2) Design of Deep Neural Networks (DNNs)

[0102] Based on the dataset obtained in step 1.3, the gate opening and upstream water inflow (flow rate or water level) at each time step are used as input features, and the cross-sectional flow rate at the corresponding time step is used as the label to form the dataset Y∈R. n , where n represents the number of samples. The dataset is progressively extracted using multiple nonlinear transformations; the output of the l-th layer can be expressed as:

[0103] g (l) =j (l) (K (l) g (l-1) +e (l) )

[0104] Among them, g (l) g (l-1) These are the intermediate feature values ​​extracted from the dataset after passing through the l-th and l-1-th hidden layers of the DNN; K (l) and e (l) These are the weight matrix and the bias vector, respectively; j (l) It is a non-linear activation function.

[0105] The prediction results of the key section flow are as follows With the actual cross-sectional flow rate y(m 3 A comparison is made using / s). Simultaneously, L1 and L2 regularization methods are incorporated. By adding the sum of squared weights and the sum of absolute values ​​of the weights to the loss function, large weights are penalized, thereby limiting the growth of model parameters and keeping the weights relatively small, thus reducing the risk of overfitting. The loss function can be expressed as:

[0106]

[0107] in, λ represents the mean squared error loss (MSE); λ1 and λ2 are regularization coefficients, which control the weight of the regularization term in the total loss function; This represents the L1 regularization term, which facilitates feature selection; L represents the L2 regularization term, used to suppress excessively large weights and control model complexity; L total The total loss value represents the weighted sum of the model prediction error and the model complexity.

[0108] (3) Transformer-DNN Coupled Model

[0109] Coupled Transformer and DNN: The Transformer first processes the time series data, extracting temporal features, and then inputs these features into the DNN for higher-level feature fusion and prediction. If the comprehensive feature sequence output by the Transformer is H, the DNN receives the flattened result of H and performs prediction to obtain the key section flow data.

[0110]

[0111] in, Predicted flow rate (m³) for key cross-sections 3 / s); f DNN is the mapping function of a deep neural network (DNN), which receives a flattened temporal feature vector, performs multiple nonlinear transformations and feature fusion, and outputs the final predicted value; Flatten is the flattening operation, which transforms matrix H into a one-dimensional vector; H is the comprehensive feature sequence data output by the Transformer.

[0112] (4) Input priority classification mechanism

[0113] To improve the efficiency and quality of model training under multi-source data, the input data is stored in layers, and an input priority classification mechanism is set. The scheme clusters calculated in the previous step (3) are set to low priority; based on the scheme clusters output after model optimization calculation, the scheme clusters that are screened, checked and retained are set to high priority, which are used to optimize the training results of the low priority scheme clusters.

[0114] 2.2 Knowledge Constraint Reinforcement Learning Module Combining Domain Knowledge and Experience Base

[0115] Knowledge-constrained reinforcement learning (Constraint RL) combines domain knowledge with an experience base, embedding empirical rules from the water conservancy field (such as gate opening and discharge capacity, flow limit, etc.) into the reinforcement learning strategy through physical and hydrodynamic constraints. It treats a single gate as an independent agent, and the interaction process between the gate agent and the environment is as follows: Figure 3 As shown, the diagram includes the state space S, action space A, reward function r(s,a), and transition probability P(s′|s,a). The agent learns the optimal policy π through interaction with the environment. * (s), the objective is to maximize the cumulative reward:

[0116]

[0117] Where r(s) t ,a t ) is state s t Take action a t Instant rewards, γ t π is the discount factor of the agent at time t, and π is the agent's policy.

[0118] (1) Basic function settings

[0119] The environmental state space S is a discrete, finite, non-empty set, which is composed of information such as flow rate and water level at each cross section.

[0120] S = {s0, s1, ... s} t ...,s N}

[0121] Where N is the size of the environment state space.

[0122] Action space A is also defined as a finite set, which is the sum of actions or behaviors taken by the gate agent in the river network environment to achieve optimal river network scheduling, including the gate's opening degree and corresponding opening timing.

[0123] A = {a0, a1, ... a t...,a M}

[0124] Where M is the spatial size of the actions taken by the agent.

[0125] At time t during the decision-making process, the gate agent is in environmental state s. t Making a decision and taking action is a t The environmental state will be determined by s t Transition to the next state s t+1 and receive a reward r t+1 The transition of its environmental state is determined by the transition function f: S×A→S. Based on the transition, if the agent transitions to a definite next state, then:

[0126] s t+1 =P(s) t a t )

[0127] However, in actual river network environments, the next state that the gate agent transitions to is generally an uncertain random variable, entering the environmental state s. t+1 The probability lies in the interval S, therefore it needs to satisfy...

[0128] (2) Setting constraints for scheduling experience base

[0129] Adding domain knowledge constraints to gate scheduling ensures that the reward function reflects not only the performance of the gate agent but also its interpretability under domain knowledge constraints. The agent's behavior can be adjusted by incorporating knowledge constraints into the reward function.

[0130] r′(s t ,a t )=r(s t ,a t )+λC(s t ,a t )

[0131] Wherein, C(s) t ,a t ) is a domain-knowledge-based constraint function, where λ is a hyperparameter controlling the importance of the constraint. The constraint function C(s) t ,a t The rules are based on water level constraints, flow constraints, and operational constraints.

[0132]

[0133] Among them, h t Let h be the water level (m) at the current time t. max h represents the maximum allowable water level limit (m) at the current time t.min Q represents the minimum allowable water level limit (m) at the current time t. t Let Q be the flow rate (m³ / s) at the current time t. max Q represents the maximum allowable flow rate limit (m³ / s) at the current time t. min G represents the minimum allowable flow rate limit (m³ / s) at the current time t. t Let G be the gate opening (m) at the current time t. t-1 Let be the gate opening (m) at time t-1, and α, β, and γ be the penalty weighting coefficients for water level, flow rate, and gate opening exceeding the limit, respectively.

[0134] Based on this, the constraints are gradually relaxed, allowing the gate agent to conduct more autonomous exploration and optimize the scheduling strategy. Its mathematical form can be expressed as:

[0135] r′(s t ,a t )=r(s t ,a t )+λ t C(s t ,a t )

[0136] In the formula, λ t It is the decay coefficient that gradually decreases as the time step t increases.

[0137] 2.3 Multi-agent reinforcement learning module

[0138] Since river network scheduling involves multiple interconnected gates, a single agent often struggles to optimize global scheduling decisions. Therefore, collaboration among multiple agents can better address dynamically changing water flows and adjust and optimize water allocation in real time.

[0139] In a multi-agent river network system, the goal of each gate agent is to maximize its cumulative reward through reinforcement learning:

[0140]

[0141] In the formula, The i-th gate agent is in state s i Take action a i State-action value function at time It is the discount factor of the i-th gate at time t.

[0142] Each gate agent's scheduling decisions are influenced not only by its own constraints but also by coordination with other agents. To address this, global constraints are incorporated into each agent's reward function to ensure that all agents' decisions align towards the globally optimal goal. This can be achieved by sharing constraint information among all agents, resulting in the constraint reward function becoming:

[0143] r i ′(s i ,a i ) = r i (s i ,a i )+λC(s1,a1,…,s N ,a N )

[0144] Where, C(s1,a1,…,s) N ,a N ) is the global constraint function for the overall scheduling of the river network, which ensures the global coordination of the system.

[0145] III. Construction of Intelligent Regulation Model for Sluice Gate Groups in Plain River Networks

[0146] The calibrated knowledge-constrained reinforcement learning module is integrated with the flow prediction module and the multi-agent reinforcement learning module to form a complete intelligent regulation model for plain river networks. The model's functional modules are as follows: Figure 2 As shown.

[0147] 3.1 Integration with traffic prediction module

[0148] The flow prediction module receives rainfall data from hydrological and meteorological forecasts and upstream water inflow (flow rate or water level) data for a river section. It uses a Transformer structure to model the input state sequence, extracts temporal features, and outputs a flow prediction sequence for key cross-sections over a future period, with a time step of 1 hour. This predicted flow sequence serves as the environmental input for a knowledge-constrained reinforcement learning module, optimizing control decisions. The module then feeds back the solution family output by the knowledge-constrained reinforcement learning module to the flow prediction module for further refinement and training.

[0149] 3.2 Integration with multi-agent reinforcement learning modules

[0150] By embedding a single gate agent, which has been calibrated through knowledge-constrained reinforcement learning, into a multi-agent deep reinforcement learning module, and using a multi-agent reinforcement learning framework, collaborative decision-making among multiple agents is achieved, dynamic optimization of river network flow regulation is completed, and a comprehensive scheduling scheme is obtained.

[0151] 3.3 Adaptive Intelligent Scheduling Mechanism

[0152] Through the integration and coupling of 3.1 and 3.2, a time-continuous learning structure is formed, achieving an adaptive closed loop of "model-environment-feedback-regulation correction", thus constituting an adaptive intelligent scheduling mechanism for the plain river network gate group.

[0153] IV. Intelligent Regulation and Scheme Analysis of Sluice Gate Groups in Plain River Networks

[0154] Based on the calibrated intelligent regulation model of the plain river network, and by incorporating future meteorological and hydrological forecast data, a reasonable gate group scheduling scheme can be quickly output.

[0155] 4.1 Training of Intelligent Control Model

[0156] Based on historical rainfall data, river network water level and flow rate data, and gate scheduling data, the data are input into the flow prediction module to drive the intelligent control model and construct a cluster of gate group control schemes.

[0157] 4.2 Setting of Watershed Objectives and Constraints

[0158] Based on the scheduling objectives and operating conditions, corresponding gate constraint parameters are set. These parameters take into account the gate scheduling rules and historical experience, including gate opening and closing limits and allowable water level and flow ranges. This enables the control model to complete reasonable scheduling under the given constraints, achieving an adaptive closed loop of "model-environment-feedback-control correction".

[0159] 4.3 Analysis of River Network Sluice Gate Group Regulation Scheme

[0160] Based on the calibrated intelligent control model of the plain river network gate group, and by incorporating future meteorological and hydrological forecast data, the model outputs the water level and flow process of key sections and the gate group scheduling schemes under the current upstream water inflow conditions, with a time step of 1 hour, to assist in supporting scheduling decisions.

[0161] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0162] This invention provides an intelligent control method for sluice gate groups in plain river networks based on deep reinforcement learning. This method constructs an intelligent agent-environment interaction mechanism with gate opening as the decision variable and water level and flow rate as state representations. It introduces a Transformer structure and joint embedding mechanism to extract spatiotemporal sequence features, utilizes deep reinforcement learning to automatically learn the optimal scheduling strategy, and incorporates domain knowledge based on hydraulic and hydrological constraints to construct constraint functions. Compared to traditional gate scheduling methods, this method can intelligently generate a cluster of scheduling schemes, improving the flexibility, timeliness, and intelligence level of the sluice gate group scheduling system. Compared to artificial intelligence models, this method effectively solves the problem of "unrealistic solutions" that may exist in machine learning applications, forming an adaptive closed loop of "model-environment-feedback-control correction," enhancing the interpretability of the results and the practical operability of the scheme. This method is not only applicable to the real-time joint scheduling of sluice gate groups in complex plain river networks, but also possesses continuous learning capabilities. While ensuring the stability and security of regulation based on experience-based scheduling rules, it can also achieve minute-level rapid response, making it suitable for hydraulic control scenarios such as flood control and emergency response, providing support for the operation management and scheduling decision-making of next-generation intelligent water conservancy projects.

[0163] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent regulation of sluice gate groups in plain river networks based on deep reinforcement learning, characterized in that: Includes the following steps, S1. Functional Module Construction: A flow prediction module is built by coupling a Transformer encoder and a deep neural network (DNN); by combining domain knowledge and an experience base, empirical rules in the water conservancy field are embedded into the reinforcement learning strategy through physical and hydrodynamic constraints. Each gate is treated as an individual agent, and the agent learns the optimal strategy through interaction with the environment, thus building a knowledge-constrained reinforcement learning module that combines domain knowledge and an experience base; by introducing global constraints into the reward function of each agent, the decisions of all agents are ensured to be adjusted towards the global optimal goal, realizing multi-agent collaboration, thus building a multi-agent reinforcement learning module. S2. Construction of Intelligent Regulation Model for Plain River Network Gate Groups: The calibrated knowledge constraint reinforcement learning module is integrated with the flow prediction module and the multi-agent reinforcement learning module to form a complete intelligent regulation model for plain river network. S3. Intelligent regulation and scheme analysis of gate groups in plain river network: Based on the calibrated intelligent regulation model of plain river network, the system integrates future meteorological and hydrological forecast data to quickly output a reasonable gate group scheduling scheme. The process of building the traffic prediction module is as follows: S111, Transformer encoder: The time-series input includes the gate opening degree at each time step, the collected rainfall data at the corresponding time step, and the upstream water inflow of the river segment. ,in, It is a multivariate time series; Convert time series into spatiotemporal joint embeddings ;in, For embedded spatiotemporal joint features; for Input characteristics at any given time; These are variable type labels used to distinguish different hydrological characteristic channels; This is a linear projection layer used to map the original features to a high-dimensional space; It is used for timing position encoding to preserve timing information; Encoding the feature type is used to distinguish different hydrological variables; Hydraulic constraints are embedded in the Transformer's self-attention layer, focusing on feature associations that conform to hydraulic laws to improve computation. ； ； ； ； in, and For attention calculation mechanisms; , , These are the Query, Key, and Value matrices, respectively. , , These are the trainable weight matrices for the Query, Key, and Value matrices, respectively. The difference in water levels between upstream and downstream; This is the transpose of the Key matrix; The dimension of the key matrix; The hidden state matrix; The Transformer encoder transforms data through residual connections and a feedforward network, ultimately outputting a comprehensive feature sequence. ; S112. Deep Neural Network (DNN): Based on the acquired river network gate opening and key cross-section simulated flow data, the gate opening and upstream water inflow at each time step are used as input features, and the cross-sectional flow at the corresponding time step is used as the label to form a dataset. ,in, The number of samples; The dataset is progressively extracted using multiple nonlinear transformations, the first of which... The output of the layer is represented as follows: ； in, These are the first and second hidden layers of the dataset after passing through the DNN. , Intermediate feature values ​​extracted from the layer; and These are the weight matrix and the bias vector, respectively; It is a non-linear activation function; The prediction results of the flow at the key section Compared with the actual cross-sectional flow For comparison, L1 and L2 regularization methods are added, penalizing large weights by adding the sum of squared weights and the sum of absolute values ​​of the weights to the loss function; the loss function is: ； in, This is the mean squared error loss; , is the regularization coefficient for the corresponding term; For L1 regularization terms; For L2 regularization terms; This represents the total loss value. S113, Transformer-DNN Coupled Model: This model couples the Transformer and DNN. The Transformer first processes the time series data, extracting temporal features, and then inputs these features into the DNN for higher-level feature fusion and prediction. If the comprehensive feature sequence output by the Transformer is... Then the DNN receives The flattened result is used to predict the flow data of key sections; ； in, Predict flow rates for key cross-sections; It is the mapping function of DNN, which takes a flattened temporal feature vector, performs multiple nonlinear transformations and feature fusion, and outputs the final predicted value. For the flattening operation, the matrix is... Convert it into a one-dimensional vector; The comprehensive feature sequence data output by the Transformer; The process of building the knowledge-constrained reinforcement learning module that combines domain knowledge and experience base is as follows: S121. Basic Function Settings: Environment State Space It is a discrete, finite, non-empty set, consisting of the situational information of flow and water level at each cross-section; action space. It is also defined as a finite set, which is the sum of actions or behaviors taken by the gate agent in the river network environment in order to achieve optimal scheduling of the river network, including the gate opening degree and the corresponding opening timing. ； ； in, Size of the environmental state space; The size of the space for the actions taken by the intelligent agent; In the decision-making process During this time, the gate agent is in the environmental state Making decisions and taking actions The environmental condition will be changed from Transition to the next state and receive a reward The transition of its environmental state is determined by the migration function. Given that, based on the migration pattern, if the agent migrates to a given next state, then... ； In a real river network environment, it enters an environmental state. The probability is in the interval In the middle, satisfy ; S122. Setting constraints for the scheduling experience base: Add constraints based on domain knowledge of gate scheduling so that the reward function not only reflects the performance of the gate agent, but also its interpretability under domain knowledge constraints; adjust the agent's behavior by introducing knowledge constraints into the reward function. ； ； in, This is a domain-knowledge-based constraint function, which consists of rules formulated based on water level constraints, flow rate constraints, and operational constraints. Hyperparameters used to control the importance of constraints; For the current moment Water level; For the current moment Maximum permissible water level limit; For the current moment Minimum permissible water level limit; For the current moment Traffic; For the current moment Maximum allowed traffic limit; For the current moment Minimum allowed traffic limit; For the current moment The gate opening degree; For a moment The gate opening degree; , , These are the penalty weighting coefficients for exceeding the limits of water level, flow rate, and gate opening, respectively. Based on this, the constraints are gradually relaxed, allowing the gate agent to conduct more autonomous exploration and optimize scheduling strategies. Its mathematical form is: ； in, As time goes by The attenuation coefficient increases and then gradually decreases; The construction process of the multi-agent reinforcement learning module is as follows: In the river network multi-agent system, the goal of each gate agent is to maximize its cumulative reward through reinforcement learning. ； in, It is the first The gate agent is in state Take action State-action value function at time; It is the first Each gate is in Discount factor of time; Each gate agent's scheduling decisions are influenced not only by its own constraints but also by coordination with other agents. Therefore, global constraints are introduced into each agent's reward function to ensure that all agents' decisions are adjusted towards the globally optimal goal. This is achieved by sharing constraint information among all agents, and the constraint reward function becomes... ； in, This is the global constraint function for the overall scheduling of the river network.

2. The intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning according to claim 1, characterized in that: Step S113 is followed by, S114. Input priority classification mechanism: Store input data in a hierarchical manner and set an input priority classification mechanism; The scheme clusters calculated in step S113 are set to low priority. Based on the scheme clusters output after optimization calculation by the Transformer-DNN coupled model, the scheme clusters that have been screened and retained are set to high priority to optimize the training results of the low priority scheme clusters.

3. The intelligent regulation method for plain river network sluice gate groups based on deep reinforcement learning according to claim 2, characterized in that: In the knowledge-constrained reinforcement learning module that combines domain knowledge and experience base, each gate is treated as an independent intelligent agent. The interaction process between the gate agent and the environment includes state space... Action space Reward function and transition probability The agent learns the optimal strategy through interaction with the environment. The goal is to maximize accumulated rewards; ； in, For state Take action Instant rewards; For intelligent agents in Discount factor of time; The strategy for the agent.

4. The intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning according to claim 1, characterized in that: Step S2 specifically includes the following: S21. Integration with the flow prediction module: The flow prediction module receives hydrological and meteorological forecast rainfall data and upstream water inflow data of the river section. It uses a Transformer encoder to model the input state sequence, extracts time-series features, and outputs a key section flow prediction sequence for a future period with a time step of 1 hour. The predicted key section flow sequence serves as the environmental input for the knowledge-constrained reinforcement learning module to optimize control decisions. The scheme family output by the knowledge-constrained reinforcement learning module is fed back to the flow prediction module for further correction and training. S22. Integration with multi-agent reinforcement learning module: The single gate agent, which has been calibrated by knowledge constraint reinforcement learning, is embedded into the multi-agent deep reinforcement learning module. The multi-agent reinforcement learning framework is adopted to realize the collaborative decision-making of multiple agents, complete the dynamic optimization of river network flow regulation, and obtain a comprehensive scheduling scheme. S23. Adaptive intelligent scheduling mechanism: Through the integration and coupling of steps S21 and S22, a time-continuous learning structure is formed to achieve an adaptive closed loop of "model-environment-feedback-regulation correction", thus forming an adaptive intelligent scheduling mechanism for the plain river network gate group and obtaining an intelligent regulation model for the plain river network.

5. The intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning according to claim 1, characterized in that: Step S3 specifically includes the following: S31. Intelligent regulation model training: Based on historical rainfall process, river network water level and flow, and gate scheduling process data, the data are input into the flow prediction module to drive the plain river network intelligent regulation model and construct a cluster of gate group regulation schemes. S32. Setting of watershed objectives and constraints: Set the corresponding gate constraint parameters according to the scheduling objectives and operating conditions, so that the intelligent regulation model of the plain river network can complete reasonable scheduling under the given constraints, and achieve an adaptive closed loop of "model-environment-feedback-regulation correction". S33. Analysis of River Network Sluice Gate Group Regulation Scheme: Based on the calibrated intelligent regulation model of plain river network, the system incorporates future meteorological and hydrological forecast data to output the water level and flow process of key sections and the cluster of sluice gate group scheduling schemes under the current upstream water inflow conditions. The time step is 1 hour to assist in and support scheduling decisions.

6. The intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning according to any one of claims 1 to 5, characterized in that: Step S1 is preceded by, S0. Basic Data Collection and Processing: Through physical model simulation and historical data collection, obtain meteorological and hydrological time-series data, river network gate opening and key section simulated flow data required for river network scheduling, and preprocess the corresponding data.

7. The intelligent regulation method for sluice gate groups in plain river networks based on deep reinforcement learning according to claim 6, characterized in that: Step S0 specifically includes the following: S01. Acquisition of meteorological and hydrological time series data: The meteorological and hydrological measured historical data required for river network scheduling include upstream water inflow, river water level and flow data at key sections, and rainfall data obtained through manual observation and automated telemetry. S02. Meteorological and hydrological time series data processing: A sliding window standardization processing and time series alignment mechanism are adopted to unify the time step of time series data to 1 hour, so as to achieve consistent alignment of multi-dimensional data in the time dimension. For observational anomalies and missing measurement points, a dynamic repair algorithm based on spatial and temporal proximity is used to reconstruct the data, thereby improving the temporal accuracy of the input data in subsequent location encoding. S03. Data acquisition of river network gate opening and key section simulated flow: Using a physical process model, different inflow processes and working condition combinations are set up to carry out flood control numerical simulation calculations for different typical flood events, and obtain corresponding data sets of upstream inflow, gate group opening and key section flow under different typical flood events and working condition combinations, with a time step of 1 hour. A comparative sample library covering extreme water inflow conditions and typical scheduling scenarios is constructed. The sample data in the sample library is encoded and labeled, and used as supervision signals for different control strategy categories during training and computation, supporting the encoding of the input channels of the subsequent Transformer.

Images