Plain river gatekeeper group intelligent regulation and control method based on deep reinforcement learning

Abstract

The invention discloses a plain river gatekeeper group intelligent regulation and control method based on deep reinforcement learning. The method comprises the steps that a flow prediction module, a knowledge constraint reinforcement learning module combining domain knowledge and an experience library and a multi-agent reinforcement learning module are built; integrating the calibrated knowledge constraint reinforcement learning module with a flow prediction module and a multi-agent reinforcement learning module to form a complete plain river network intelligent regulation and control model; and on the basis of the calibrated plain river network intelligent regulation and control model, future meteorological and hydrological prediction data are accessed, and a reasonable gate group scheduling scheme is rapidly output. The method has the advantages that the scheduling scheme cluster is intelligently generated, the complex and dynamically changing plain river gatekeeper group scheduling problem is efficiently solved, the flexibility, timeliness and intelligent level of a gatekeeper group scheduling system are improved, the possible'non-realistic solution 'problem of a machine learning method in practical application is solved, the interpretability of a result and the practical operability of a scheme are enhanced, and the method is suitable for large-scale popularization and application. And a self-adaptive closed loop of model-environment-feedback-regulation and control correction is formed.

Description

Technical Field

[0001] The present invention relates to the technical field of plain river network gate group control, and in particular to a plain river network gate group intelligent control method based on deep reinforcement learning. Background Art

[0002] Currently, the regulation of plain river sluice systems primarily relies on hydrodynamic models. These models manually set fixed operating condition combinations (such as opening and closing rules based on water level thresholds or time series) and perform forward calculations based on upper and lower boundary conditions. This generates a limited, discrete cluster of scheduling plans, which are difficult to cover the complex and changing actual operating conditions. Furthermore, plan generation relies on expert experience, has a low level of intelligence, and requires repeated model calculations and verification, resulting in high computational complexity and low efficiency. This makes it difficult to dynamically adapt to changing environmental conditions, especially the complex water-rain-operating conditions influenced by climate change and human activities.

[0003] With the rapid development of artificial intelligence (AI) technology, deep learning methods have gradually become a research hotspot in the field of water conservancy scheduling. Deep learning models are used to train the relationship between inflow / operating conditions and the water level / flow process at target sections, establishing a mapping relationship between inflow and operating conditions and the target of control. Deep learning models can then be used to rapidly simulate and generate gate control scheduling plans, significantly improving the intelligence of scheduling decisions. However, flow prediction and gate control based on deep learning models still have limitations. Their "black box" nature makes the output lack interpretability. Furthermore, because models typically learn patterns only from time series data and are unable to directly quantify and learn unstructured control rules, the model output may not conform to physical laws such as hydraulic properties, resulting in infeasible solutions and, in other words, unworkable scheduling plans. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent control method for plain river network gate groups based on deep reinforcement learning, so as to solve the above-mentioned problems existing in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for intelligent control of plain river network gate groups based on deep reinforcement learning, comprising the following steps:

[0007] S1, function module building: a traffic prediction module is built by coupling a designed Transformer encoder and a deep neural network DNN; a knowledge constraint reinforcement learning module combining domain knowledge and experience library is built by embedding experience rules in the water conservancy field into the reinforcement learning strategy through physical constraints and hydrodynamic constraints, regarding a single gate as a single agent, and having the agent learn an optimal strategy in interaction with the environment; a multi-agent reinforcement learning module is built by introducing global constraints into the reward function of each agent to ensure that the decisions of all agents are adjusted towards the global optimal goal and realize multi-agent collaboration;

[0008] S2, plain river network gate group intelligent regulation and control model construction: the knowledge constraint reinforcement learning module calibrated is integrated with the traffic prediction module and the multi-agent reinforcement learning module to form a complete plain river network intelligent regulation and control model;

[0009] S3, plain river network gate group intelligent regulation and control and scheme analysis: based on the calibrated plain river network intelligent regulation and control model, future meteorological and hydrological prediction data are input to quickly output a reasonable gate group regulation scheme.

[0010] Preferably, the building process of the traffic prediction module is,

[0011] S111, Transformer encoder: the gate group opening degree at each time and the collected rainfall data and upstream inflow at the corresponding time are input as a time series wherein, x t is a multivariate time series;

[0012] the time series is converted into a spatio-temporal joint embedding,

[0013] z t = Embedding(x t )+PositionalEncoding(t)+ChannelEncoding(c)

[0014] wherein, z t is the embedded spatio-temporal joint feature; x t is the input feature at time t; c is a variable type label for distinguishing different hydrological feature channels; Embedding is a linear projection layer for mapping the original feature to a high-dimensional space; PositionalEncoding is a time sequence position encoding for preserving time sequence information; ChannelEncoding is a feature type encoding for distinguishing different hydrological variables;

[0015] In the Transformer self-attention layer, the hydraulic constraint is embedded, the feature correlation conforming to the hydraulic law is focused on, and the calculation is improved,

[0016] Q = W q Z

[0017] K = W k Z

[0018] V = W v Z

[0019]

[0020] wherein, ATTENTION and Sofmax are attention computing mechanisms; Q, K, V are Query, Key, Value matrices respectively; W q , W k , W v are trainable weight matrices of Query, Key, Value matrices respectively; Δh t is the upstream and downstream water level difference; K T is the transpose matrix of Key matrix; d k is the dimension of Key matrix; Z is the hidden state matrix;

[0021] The Transformer encoder is converted through a residual connection and a feedforward network, and finally outputs a comprehensive feature sequence data H;

[0022] S112, deep neural network DNN: based on the obtained river network gate opening and key section simulation flow data, the hourly gate opening and upstream inflow of the river section are taken as input features, and the corresponding section flow at the moment is taken as a label to form a data set Y ∈ R n , wherein n is the number of samples;

[0023] The data set is gradually extracted features through multi-layer nonlinear transformation, and the output of the lth layer is represented as,

[0024] g (l) = j (l) (K (l) g (l-1) + e (l) )

[0025] wherein, g (l) , g (l-1) are intermediate feature values extracted by the lth and l-1th layers of the DNN hidden layer respectively; K (l) and e (l) are weight matrix and bias vector respectively; j (l) is a nonlinear activation function;

[0026] The prediction result of the key section flow is Compared with the actual cross-sectional flow y, L1 and L2 regularization methods are added at the same time, and large weights are penalized by adding the sum of squares and absolute values ​​of weights to the loss function; the loss function is:

[0027]

[0028] in, is the mean square error loss; λ1 and λ2 are the regularization coefficients of the corresponding terms; is the L1 regularization term; is the L2 regularization term; L total is the total loss value;

[0029] S113, Transformer-DNN coupling model: The Transformer and DNN are coupled. The Transformer first processes the time series data and extracts temporal features. These features are then input into the DNN for higher-level feature fusion and prediction. If the comprehensive feature sequence output by the Transformer is H, the DNN receives the flattened result of H and predicts the flow data of key sections.

[0030]

[0031] in, Predict flow values ​​for key sections; f DNN is the mapping function of DNN, which receives the flattened time series feature vector and outputs the final prediction value after multiple layers of nonlinear transformation and feature fusion. Flatten is a flattening operation that converts the matrix H into a one-dimensional vector. H is the comprehensive feature sequence data output by Transformer.

[0032] Preferably, after step S113, the method further includes:

[0033] S114. Input priority classification mechanism: The input data is stored in layers and an input priority classification mechanism is set; the solution cluster calculated in step S113 is set to a low priority, and based on the solution cluster output after the Transformer-DNN coupling model optimization calculation, the solution cluster that has passed the screening and verification and is retained is set to a high priority to optimize the training results of the low-priority solution cluster.

[0034] Preferably, the process of building the knowledge constraint reinforcement learning module combining domain knowledge and experience base is as follows:

[0035] S121. Basic function settings: The environmental state space S is a discrete, finite, non-empty set consisting of the flow and water level information at each section. The action space A is also defined as a finite set, which is the sum of the actions or behaviors taken by the gate agent to achieve optimal river network operation under the river network environmental state, including the gate opening and the corresponding opening timing.

[0036] S={s0,s1,…s t ...,s N}

[0037] A={a0,a1,…a t ...,a M}

[0038] Among them, N is the size of the environment state space; M is the size of the space of actions taken by the agent;

[0039] At time t in the decision-making process, the gate agent is in the environment state s t The next decision is to take action a t , the environmental state will be determined by s t Transition to the next state s t+1 , and get reward r t+1 ; The transfer of its environmental state is determined by the migration function f:S×A→S. According to the migration situation, if the agent migrates to the next state, then,

[0040] s t+1 =P(s t , a t )

[0041] In the actual river network environment, enter the environmental state s t+1 The probability of being in interval S satisfies

[0042] S122, Scheduling Experience Library Constraint Setting: Add gate scheduling domain knowledge constraints so that the reward function reflects not only the performance of the gate agent but also its interpretability under domain knowledge constraints. Adjust the agent's behavior by introducing knowledge constraints into the reward function.

[0043] r′(s t ,a t )=r(s t ,a t )+λC(s t ,q t )

[0044]

[0045] Among them, C(st ,a t ) is a constraint function based on domain knowledge, which is a rule formulated according to water level constraints, flow constraints, and operation constraints; λ is a hyperparameter that controls the importance of constraints; h t is the water level at the current time t; h max is the maximum water level limit allowed at the current time t; h min is the minimum water level limit allowed at the current time t; Q t is the flow rate at the current time t; Q max is the maximum flow limit allowed at the current time t; Q min is the minimum flow limit allowed at the current time t; G t is the gate opening at the current time t; G t-1 is the gate opening at time t-1; α, β, and β are the penalty weight coefficients for exceeding the limit of water level, flow rate, and gate opening respectively;

[0046] On this basis, the influence of constraints is gradually relaxed, allowing the gate agent to conduct more autonomous exploration and optimize the scheduling strategy. Its mathematical form is:

[0047] r′(s t ,a t )=r(s t ,a t )+λ t C(s t ,a t )

[0048] Among them, λ t is the attenuation coefficient that gradually decreases as the time step t increases.

[0049] Preferably, in the knowledge-constrained reinforcement learning module that combines domain knowledge with the experience base, a single gate is regarded as a separate agent. The interaction process between the gate agent and the environment includes the state space S, the action space A, the reward function r(s,a) and the transition probability P(s′|s,a); the agent learns the optimal strategy π in the interaction with the environment * (s), the goal is to maximize the cumulative reward;

[0050]

[0051] Among them, r(s t ,a t ) is state s t Take action a t Immediate reward when γ t is the agent’s discount factor at time t; π is the agent’s strategy.

[0052] Preferably, the process of building the multi-agent reinforcement learning module is as follows: in the river network multi-agent system, the goal of each gate agent is to maximize its own cumulative reward through reinforcement learning.

[0053]

[0054] in, is the i-th gate agent in state s i Take action a i The state-action value function when ; is the discount factor of the i-th gate at time t;

[0055] The scheduling decision of each gate agent is not only affected by its own constraints, but also needs to consider the coordination with other agents; therefore, the global constraint is introduced into the reward function of each agent to ensure that the decisions of all agents are adjusted towards the global optimal goal. This is achieved by sharing the constraint information among all agents. The constraint reward function becomes,

[0056] r i ′(s i ,a i )=r i (s i ,a i )+λC(s1,a1,…,s N ,a N )

[0057] Among them, C(s1,a1,…,s N ,a N ) is the global constraint function for the overall regulation of the river network.

[0058] Preferably, step S2 specifically includes the following contents:

[0059] S21. Integration with the flow prediction module: The flow prediction module receives hydrological and meteorological forecast rainfall data and upstream water flow data, uses the Transformer encoder to model the input state sequence, extracts time series features, and outputs a key section flow prediction sequence for the future period, with a time step of 1 hour. The predicted key section flow sequence serves as the environmental input of the knowledge-constrained reinforcement learning module to optimize the control decision-making. The solution cluster output by the knowledge-constrained reinforcement learning module is fed back to the flow prediction module for further correction training.

[0060] S22. Integration with the Multi-Agent Reinforcement Learning Module: The single gate agent, calibrated by knowledge-constrained reinforcement learning, is embedded into the multi-agent deep reinforcement learning module. Using the multi-agent reinforcement learning framework, collaborative decision-making among multiple agents is achieved, dynamic optimization of river network flow regulation is achieved, and a comprehensive scheduling solution is obtained.

[0061] S23. Adaptive intelligent scheduling mechanism: Through the integrated coupling of steps S21 and S22, a time-continuous learning structure is formed to achieve an adaptive closed loop of "model-environment-feedback-control correction", forming an adaptive intelligent scheduling mechanism for the plain river network gate group and obtaining a plain river network intelligent control model.

[0062] Preferably, step S3 specifically includes the following contents:

[0063] S31. Intelligent Control Model Training: Based on historical rainfall data, river network water level and flow, and gate scheduling process data, these data are input into the flow prediction module to drive the plain river network intelligent control model and construct a gate group control plan cluster;

[0064] S32. Basin target and constraint setting: Set corresponding gate constraint parameters according to the scheduling target and operating conditions, so that the plain river network intelligent regulation model can complete reasonable scheduling under the given constraints, and achieve an adaptive closed loop of "model-environment-feedback-regulation correction";

[0065] S33. Analysis of river network and sluice group control schemes: Based on the calibrated plain river network intelligent control model, access to future meteorological and hydrological forecast data is used to output the water level and flow processes of key sections and the sluice group control scheme cluster under the current upstream water inflow conditions of the river section, with a time step of 1 hour, to assist in supporting control decisions.

[0066] Preferably, before step S1, the method further includes:

[0067] S0. Basic data collection and processing: Through physical model simulation and historical data collection, obtain the meteorological and hydrological time series data, river network gate opening and key section simulated flow data required for river network scheduling, and pre-process the corresponding data.

[0068] Preferably, step S0 specifically includes the following contents:

[0069] S01. Acquisition of meteorological and hydrological time-series data: The meteorological and hydrological measured historical data required for river network operation include upstream water flow, river level and flow data at key sections, and rainfall data obtained through manual observation and automated telemetry.

[0070] S02. Meteorological and hydrological time series data processing: Using sliding window normalization and time series alignment mechanisms, the time step of time series data is unified to 1 hour, achieving consistent alignment of multivariate data in the temporal dimension. For observation anomalies and missing points, dynamic repair algorithms based on spatial and temporal proximity are used to complete reconstruction, improving the temporal accuracy of input data in subsequent position encoding.

[0071] S03. Acquisition of simulated flow data for river network gate openings and key sections: Using a physical process model, different water inflow processes and operating condition combinations are set to carry out numerical simulation calculations of flood control for different typical flood events. The corresponding data sets of upstream water inflow, gate group openings, and key section flows under different typical flood events and operating condition combinations are obtained, with a time step of 1 hour. A control sample library covering extreme water inflow conditions and typical scheduling scenarios is constructed. The sample data in the sample library uses a coding label mechanism and serves as a supervisory signal for different control strategy categories during the training calculation process to support the input channel encoding of the subsequent Transformer.

[0072] The present invention has the following beneficial effects: 1. By constructing an agent-environment interaction mechanism with gate opening as the decision variable and water level and flow as state representations, the present method introduces a Transformer architecture and a joint embedding mechanism to extract spatiotemporal sequence features, automatically learns the optimal scheduling strategy using deep reinforcement learning, and constructs constraint functions based on domain knowledge of hydraulic and hydrological constraints. Compared to traditional gate scheduling methods, this method can intelligently generate scheduling solution clusters, improving the flexibility, timeliness, and intelligence of the gate group scheduling system. Compared to artificial intelligence models, this method effectively addresses the "unrealistic solution" problem that may exist in practical machine learning applications, forming an adaptive closed loop of "model-environment-feedback-control and correction", enhancing the interpretability of results and the practical operability of solutions. 2. The present method is not only suitable for real-time joint scheduling of gate groups in complex plain river networks, but also possesses continuous learning capabilities. While ensuring the stability and safety of control based on empirical scheduling rules, it can also achieve rapid response within minutes. It is suitable for hydraulic control scenarios such as flood prevention and emergency response, providing support for the operation management and scheduling decision-making of the next generation of intelligent water conservancy projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a flow chart of the intelligent control method in an embodiment of the present invention;

[0074] Figure 2 This is a schematic diagram of the principles of each module of the intelligent control model in an embodiment of the present invention;

[0075] Figure 3 Schematic diagram of the Markov decision process in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] In response to the problems of insufficient timeliness and intelligence of existing physical process models, and insufficient interpretability and feasible solutions of artificial intelligence models, this embodiment combines Transformer encoders, deep neural networks DNN, reinforcement learning and domain knowledge to construct an intelligent control method for plain river network gate groups based on deep reinforcement learning, intelligently generate scheduling plan clusters, efficiently respond to complex and dynamically changing plain river network gate group scheduling problems, improve the flexibility, timeliness and intelligence level of the gate group scheduling system, solve the "unrealistic solution" problem that may exist in the actual application of machine learning methods, enhance the interpretability of the results and the actual operability of the plan, and form an adaptive closed loop of "model-environment-feedback-control correction". Figure 1 As shown, the method of the present invention specifically includes the following parts:

[0078] 1. Basic Data Collection and Processing

[0079] Through physical model simulation, historical data collection and other means, we obtain corresponding data sets of upstream water inflow (flow or water level), river network gate group opening and key section flow, providing a data basis for conducting research on intelligent regulation of plain river network gate groups based on deep reinforcement learning.

[0080] 1.1. Acquisition of meteorological and hydrological time series data

[0081] The meteorological and hydrological measured historical data required for river network scheduling include: upstream water inflow (flow or water level), river water level and flow data at key sections, rainfall data and other measured historical data, including manual observation data and automated telemetry data.

[0082] 1.2 Data processing of meteorological and hydrological time series data

[0083] To meet the temporal consistency requirements of the embedded structure, a sliding window normalization process and time series alignment mechanism are used to unify the time step of the time series data to one hour, achieving consistent alignment of multi-source data in the temporal dimension. For observation anomalies and missing points, a dynamic repair algorithm based on spatial and temporal proximity is used to reconstruct them, improving the temporal accuracy of the input data in subsequent position encoding.

[0084] 1.3. Acquisition of river network gate opening and simulated flow data at key sections

[0085] Using physical process models such as MIKE, and setting different combinations of water inflow processes and operating conditions, numerical simulations of flood control for various typical flood events were conducted. Data sets corresponding to upstream water inflow (flow or water level), gate group openings, and flow at key sections were obtained for each of these typical flood events and operating conditions, with a time step of one hour. A control sample library covering extreme water inflow conditions and typical scheduling scenarios was constructed. The sample data in the sample library was encoded using a labeling mechanism. This served as supervisory signals for different control strategy categories during the training process, supporting the subsequent input channel encoding of the Transformer.

[0086] 2. Functional Module Construction

[0087] By coupling the designed Transformer encoder and deep neural network (DNN), a flow prediction module is built. By combining domain knowledge and experience libraries, empirical rules in the water conservancy field are embedded into reinforcement learning strategies through physical and hydrodynamic constraints. A single gate is considered as a separate intelligent agent, which learns the optimal strategy through interaction with the environment to build a knowledge-constrained reinforcement learning module that combines domain knowledge and experience libraries. By introducing global constraints into the reward function of each intelligent agent, the decisions of all intelligent agents are ensured to be adjusted towards the global optimal goal, multi-agent collaboration is achieved, and a multi-agent reinforcement learning module is built. The principles of each functional module can be found in [1]. Figure 2 .

[0088] 2.1 Traffic Prediction Module (Transformer+DNN)

[0089] (1) Transformer encoder design

[0090] The moment-by-moment gate opening, the corresponding rainfall information collected in step 1.1, and the upstream water flow (flow or water level) of the river section are used as time series input. where x t is a multivariate time series.

[0091] Convert a time series into a joint spatiotemporal embedding:

[0092] z t =Embedding(x t )+PositionalEncoding(t)+ChannelEncoding(c)

[0093] Among them, z t is the spatiotemporal joint feature after embedding; x tis the input feature at time t; c is the variable type label (such as rainfall, flow, water level, gate opening, etc.), which is used to distinguish different hydrological feature channels; Embedding is the linear projection layer, which maps the original features to a high-dimensional space; PositionalEncoding is the time series position encoding, which retains the time series information; ChannelEncoding is the feature type encoding, which distinguishes different hydrological variables.

[0094] Embedding hydraulic constraints in the Transformer self-attention layer focuses on feature associations that conform to hydraulic laws and performs improved calculations:

[0095] Q=W q Z

[0096] K=W k Z

[0097] V=W v Z

[0098]

[0099] Among them, ATTENTION and Softmax are attention calculation mechanisms; Q, K, V are Query, Key, and Value matrices; W q 、W k 、E v are the trainable weight matrices of Query, Key, and Value matrices respectively; Δh t is the upstream and downstream water level difference (m); K T is the transposed matrix of the Key matrix; d k is the dimension of the Key matrix; Z is the hidden state matrix.

[0100] The Transformer encoder is converted through residual connections and feedforward networks, and finally outputs comprehensive feature sequence data H.

[0101] (2) Deep Neural Network (DNN) Design

[0102] Based on the data set obtained in step 1.3, the gate opening and upstream water flow (flow or water level) at each moment are used as input features, and the cross-sectional flow at the corresponding moment is used as the label to form the data set Y∈R n , where n represents the number of samples. The dataset is gradually extracted through multiple layers of nonlinear transformations, and the output of the lth layer can be expressed as:

[0103] g (l) =j (l) (K (l) g (l-1) +e (l) )

[0104] Among them, g (l) 、g (l-1) It is the intermediate feature value extracted from the dataset after passing through the lth and l-1th layers of the DNN hidden layer; K (l) and e (l) are the weight matrix and bias vector respectively; j (l) is a non-linear activation function.

[0105] The predicted result of the key section flow is Compared with the actual cross-sectional flow y(m 3 / s) for comparison. L1 and L2 regularization methods are also added to the loss function to penalize large weights by adding the sum of squares and absolute values ​​of weights. This limits the growth of model parameters and keeps the weights small, thus reducing the risk of overfitting. The loss function can be expressed as:

[0106]

[0107] in, represents the mean square error loss (MSE); λ1 and λ2 are regularization coefficients that control the weight of the regularization term in the total loss function; Represents the L1 regularization term, which facilitates feature selection; represents the L2 regularization term, which is used to suppress excessive weights and control model complexity; L total Represents the total loss value, which is the weighted sum of the model prediction error and the model complexity.

[0108] (3) Transformer-DNN coupling model

[0109] Coupling Transformer and DNN: Transformer first processes time series data and extracts temporal features, then inputs these features into DNN for higher-level feature fusion and prediction. If the comprehensive feature sequence output by Transformer is H, DNN receives the flattened result of H and predicts the key section flow data:

[0110]

[0111] in, Predicted flow value for key sections (m 3 / s);f DNN It is the mapping function of the deep neural network (DNN), which receives the flattened time series feature vector and outputs the final prediction value after multiple layers of nonlinear transformation and feature fusion. Flatten is a flattening operation that converts the matrix H into a one-dimensional vector. H is the comprehensive feature sequence data output by Transformer.

[0112] (4) Input priority classification mechanism

[0113] Driven by multivariate data, in order to improve the efficiency and quality of model training, the input data is stored in layers and an input priority classification mechanism is set. The solution cluster calculated in the previous step (3) is set with a low priority; based on the solution cluster output after model optimization calculation, the solution clusters that are screened, verified and retained are set with a high priority to optimize the training results of the low-priority solution clusters.

[0114] 2.2. Knowledge-constrained reinforcement learning module combining domain knowledge and experience base

[0115] Constraint RL combines domain knowledge with experience base, embeds empirical rules in the water conservancy field (such as opening discharge, flow limit, etc.) into the reinforcement learning strategy through physical constraints and hydrodynamic constraints, and regards a single gate as a separate intelligent agent. The interaction process between the gate intelligent agent and the environment is as follows: Figure 3 As shown in Figure 1, it includes the state space S, action space A, reward function r(s,a) and transition probability P(s′|s,a). The agent learns the optimal strategy π in the interaction with the environment. * (s), the goal is to maximize the cumulative reward:

[0116]

[0117] Among them, r(s t ,a t ) is the state s t Take action a t The immediate reward when γ t is the agent’s discount factor at time t, and π is the agent’s strategy.

[0118] (1) Basic function settings

[0119] The environmental state space S is a discrete finite non-empty set, which is composed of information such as flow rate and water level at each section.

[0120] S={s0,s1,…s t ...,s N}

[0121] Where N is the size of the environment state space.

[0122] The action space A is also defined as a finite set, which is the sum of the actions or behaviors taken by the gate agent in the river network environment to achieve the optimal scheduling of the river network, including the gate opening and the corresponding opening timing.

[0123] A={a0,a1,...a t...,a M}

[0124] Among them, M is the size of the space of actions taken by the agent.

[0125] At time t in the decision-making process, the gate agent is in the environment state s t The next decision is to take action a t , the environmental state will be determined by s t Transition to the next state s t+1 , and get reward r t+1 The transition of its environment state is determined by the migration function f:S×A→S. According to the migration situation, if the agent migrates to a certain next state, then:

[0126] s t+1 =P(s t , a t )

[0127] However, in the actual river network environment, the next state that the gate agent migrates to is generally an uncertain random variable. t+1 The probability is in the interval S, so it needs to satisfy

[0128] (2) Scheduling experience database constraint setting

[0129] By adding domain knowledge constraints to gate scheduling, the reward function not only reflects the performance of the gate agent, but also reflects its interpretability under domain knowledge constraints. The behavior of the agent can be adjusted by introducing knowledge constraints into the reward function:

[0130] r′(s t ,a t )=r(s t ,a t )+λC(s t ,a t )

[0131] Among them, C(s t ,a t ) is a constraint function based on domain knowledge, and λ is a hyperparameter that controls the importance of the constraint. t ,a t ) is a rule formulated based on water level constraints, flow constraints, and operation constraints.

[0132]

[0133] Among them, h t is the water level at the current time t (m), h max is the maximum water level limit allowed at the current time t (m), hmin is the minimum water level limit allowed at the current time t (m), Q t is the flow rate at the current time t (m3 / s), Q max is the maximum flow rate limit allowed at the current time t (m3 / s), Q min is the minimum flow rate limit allowed at the current time t (m3 / s), G t is the gate opening at the current time t (m), G t-1 is the gate opening at time t-1 (m), α, β, and γ are the penalty weight coefficients for exceeding the limit of water level, flow rate, and gate opening, respectively.

[0134] On this basis, the influence of constraints is gradually relaxed, allowing the gate agent to conduct more autonomous exploration and optimize the scheduling strategy. Its mathematical form can be expressed as:

[0135] r′(s t ,a t )=r(s t ,a t )+λ t C(s t ,a t )

[0136] Where λ t is the attenuation coefficient that gradually decreases as the time step t increases.

[0137] 2.3 Multi-agent reinforcement learning module

[0138] Since river network scheduling problems involve multiple interconnected gates, it is often difficult for a single intelligent agent to optimize global scheduling decisions. Therefore, through the collaboration of multiple intelligent agents, we can better cope with dynamically changing water flows and adjust and optimize water distribution in real time.

[0139] In the river network multi-agent system, the goal of each gate agent is to maximize its own cumulative reward through reinforcement learning:

[0140]

[0141] Where, is the i-th gate agent in state s i Take action a i The state-action value function when is the discount factor of the i-th gate at time t.

[0142] The scheduling decision of each gate agent is not only affected by its own constraints, but also needs to consider coordination with other agents. To this end, global constraints are introduced into the reward function of each agent to ensure that the decisions of all agents are adjusted towards the global optimal goal. This can be achieved by sharing constraint information among all agents. The constraint reward function becomes:

[0143] r i ′(s i ,a i )=r i (s i ,a i )+λC(s1,a1,…,s N ,a N )

[0144] Among them, C(s1,a1,…,s N ,a N ) is the global constraint function of the overall river network operation, which ensures the global coordination of the system.

[0145] 3. Construction of Intelligent Control Model for Plain River Network Sluice Groups

[0146] The knowledge constraint reinforcement learning module with good calibration is integrated with the flow prediction module and the multi-agent reinforcement learning module to form a complete plain river network intelligent regulation model. The functional modules of the model are as follows: Figure 2 shown.

[0147] 3.1. Integration with traffic prediction module

[0148] The flow prediction module receives hydrometeorological rainfall forecast data and upstream inflow (flow or water level) data. Using a Transformer architecture, it models the input state sequence, extracts temporal features, and outputs a sequence of flow forecasts for key sections over the next period, with a one-hour time step. This predicted flow sequence serves as the environmental input for the knowledge-constrained reinforcement learning module, which optimizes control decisions. The resulting solution cluster is then fed back to the flow prediction module for further training and correction.

[0149] 3.2. Integration with Multi-Agent Reinforcement Learning Module

[0150] A single gate agent calibrated by knowledge-constrained reinforcement learning is embedded in a multi-agent deep reinforcement learning module. A multi-agent reinforcement learning framework is used to implement collaborative decision-making among multiple agents, complete dynamic optimization of river network flow control, and obtain a comprehensive scheduling plan.

[0151] 3.3 Adaptive Intelligent Scheduling Mechanism

[0152] Through the integrated coupling of 3.1 and 3.2, a time-continuous learning structure is formed, achieving an adaptive closed loop of "model-environment-feedback-regulation and correction", and forming an adaptive intelligent scheduling mechanism for plain river network gate groups.

[0153] 4. Intelligent Control and Scheme Analysis of Plain River Network Lock Groups

[0154] Based on the calibrated plain river network intelligent regulation model, future meteorological and hydrological forecast data are accessed to quickly output reasonable sluice group scheduling plans.

[0155] 4.1 Intelligent Control Model Training

[0156] Based on data such as historical rainfall processes, river network water level and flow, and gate scheduling processes, they are input into the flow prediction module to drive the intelligent control model and construct a cluster of gate group control solutions.

[0157] 4.2. Basin objectives and constraints setting

[0158] The corresponding gate constraint parameters are set according to the scheduling objectives and operating conditions. The scheduling rules and historical experience of the gate are taken into consideration, including the gate opening and closing amplitude limit, the allowable water level and flow range, etc., so that the control model can complete reasonable scheduling under the established constraints and achieve an adaptive closed loop of "model-environment-feedback-control correction".

[0159] 4.3 Analysis of river network and sluice gate control schemes

[0160] Based on the calibrated intelligent control model of the plain river network sluice group, and access to future meteorological and hydrological forecast data, the model outputs the water level and flow process of key sections and the sluice group scheduling plan cluster under the current upstream water inflow conditions of the river section, with a time step of 1 hour, to assist in supporting scheduling decisions.

[0161] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0162] This invention provides a method for intelligently controlling gate groups in plain river networks based on deep reinforcement learning. This method constructs an agent-environment interaction mechanism with gate opening as the decision variable and water level and flow as state representations. It introduces a Transformer architecture and a joint embedding mechanism to extract spatiotemporal sequence features. Deep reinforcement learning is then used to automatically learn optimal scheduling strategies. Furthermore, domain knowledge based on hydraulic and hydrological constraints is incorporated to construct constraint functions. Compared to traditional gate scheduling methods, this method can intelligently generate scheduling solution clusters, improving the flexibility, timeliness, and intelligence of the gate group scheduling system. Compared to artificial intelligence models, this method effectively addresses the "unrealistic solution" problem that can exist in practical machine learning applications, forming an adaptive closed loop of "model-environment-feedback-control correction," enhancing the interpretability of results and the practical operability of the solutions. This method is not only suitable for real-time joint scheduling of gate groups in complex plain river networks, but also possesses continuous learning capabilities. While maintaining the stability and safety of control based on empirical scheduling rules, it can also achieve minute-level rapid response. This method is suitable for hydraulic control scenarios such as flood prevention and emergency response, providing support for the operation, management, and scheduling decision-making of a new generation of intelligent water conservancy projects.

[0163] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligently controlling a plain river network sluice group based on deep reinforcement learning, characterized by: The following steps are included: S1. Functional module construction: Build a flow prediction module by coupling the designed Transformer encoder and deep neural network (DNN). By combining domain knowledge and experience libraries, the empirical rules in the water conservancy field are embedded into the reinforcement learning strategy through physical and hydrodynamic constraints. A single gate is regarded as a separate intelligent agent, and the intelligent agent learns the optimal strategy through interaction with the environment to build a knowledge-constrained reinforcement learning module that combines domain knowledge and experience libraries. By introducing global constraints into the reward function of each intelligent agent, the decision-making of all intelligent agents is ensured to be adjusted towards the global optimal goal, multi-agent collaboration is achieved, and a multi-agent reinforcement learning module is built. S2. Construction of an intelligent regulation model for plain river network sluice groups: Integrate the calibrated knowledge-constrained reinforcement learning module with the flow prediction module and the multi-agent reinforcement learning module to form a complete intelligent regulation model for plain river networks; S3. Intelligent control and solution analysis of plain river network sluice groups: Based on the calibrated plain river network intelligent control model, access to future meteorological and hydrological forecast data, and quickly output reasonable sluice group scheduling solutions.

2. The method for intelligent control of plain river network gate groups based on deep reinforcement learning according to claim 1 is characterized by: The construction process of the traffic prediction module is as follows: S111, Transformer encoder: takes the gate group opening at each moment and the collected rainfall data at the corresponding moment and the water flow in the upstream of the river as time series input Among them, x t is a multivariate time series; Convert time series into joint spatiotemporal embeddings, z t =Embedding(x t )+PositionalEncoding(t)+ChannelEncoding(c)where, z t is the spatiotemporal joint feature after embedding; x t is the input feature at time t; c is the variable type label, which is used to distinguish different hydrological feature channels; Embedding is the linear projection layer, which is used to map the original features to a high-dimensional space; PositionalEncoding is the time series position encoding, which is used to retain the time series information; ChannelEncoding is the feature type encoding, which is used to distinguish different hydrological variables; Embed hydraulic constraints in the Transformer self-attention layer, focus on feature associations that conform to hydraulic laws, and perform improved calculations. Q=W q Z K=W k WITH V=W v Z Among them, ATTENTION and Softmax are attention calculation mechanisms; Q, K, and V are Query, Key, and Value matrices respectively; W q 、W k 、W v are the trainable weight matrices of Query, Key, and Value matrices respectively; Δh t is the upstream and downstream water level difference; K T is the transposed matrix of the Key matrix; d k is the dimension of the Key matrix; Z is the hidden state matrix; The Transformer encoder converts through residual connections and feedforward networks, and finally outputs comprehensive feature sequence data H; S112, Deep Neural Network DNN: Based on the acquired river network gate opening and key section simulated flow data, the gate opening and upstream water flow of the river section at each moment are used as input features, and the cross-section flow at the corresponding moment is used as a label to form a data set Y∈R n , where n is the number of samples; The dataset is gradually extracted through multiple layers of nonlinear transformation, and the output of the lth layer is expressed as, g (l) =j (l) (K (l) g (l-1) +e (l) ) Among them, g (l) 、g (l-1) are the intermediate feature values ​​extracted from the dataset through the lth and l-1th layers of the DNN hidden layer; K (l) and e (l) are weight matrix and bias vector respectively; j (l) is a nonlinear activation function; The prediction results of key section flow Compared with the actual cross-sectional flow y, L1 and L2 regularization methods are added at the same time, and large weights are penalized by adding the sum of squares and absolute values ​​of weights to the loss function; the loss function is: in, is the mean square error loss; λ1 and λ2 are the regularization coefficients of the corresponding terms; is the L1 regularization term; is the L2 regularization term; L total is the total loss value; S113, Transformer-DNN coupling model: The Transformer and DNN are coupled. The Transformer first processes the time series data and extracts temporal features. These features are then input into the DNN for higher-level feature fusion and prediction. If the comprehensive feature sequence output by the Transformer is H, the DNN receives the flattened result of H and predicts the flow data of key sections. in, Predict flow values ​​for key sections; f DNN is the mapping function of DNN, which receives the flattened time series feature vector and outputs the final prediction value after multiple layers of nonlinear transformation and feature fusion. Flatten is a flattening operation that converts the matrix H into a one-dimensional vector. H is the comprehensive feature sequence data output by Transformer.

3. The method for intelligent control of plain river network gate groups based on deep reinforcement learning according to claim 2 is characterized by: After step S113, the following steps are also included: S114, input priority classification mechanism: hierarchical storage of input data, setting up an input priority classification mechanism; The solution cluster calculated by step S113 is set to a low priority. Based on the solution cluster output after the Transformer-DNN coupling model optimization calculation, the solution cluster that has passed the screening and verification and is retained is set to a high priority to optimize the training results of the low-priority solution cluster.

4. The method for intelligent control of plain river network gate groups based on deep reinforcement learning according to claim 1 is characterized by: The process of building the knowledge-constrained reinforcement learning module that combines domain knowledge and experience base is as follows: S121. Basic function setting: The environmental state space S is a discrete finite non-empty set consisting of the flow and water level information of each section; The action space A is also defined as a finite set, which is the sum of the actions or behaviors taken by the gate agent in the river network environment to achieve the optimal scheduling of the river network, including the gate opening and the corresponding opening timing. S={s0,s1,...s t ...,s N } A={a0,a1,...a t ...,a M } Among them, N is the size of the environment state space; M is the size of the space of actions taken by the agent; At time t in the decision-making process, the gate agent is in the environment state s t The next decision is to take action a t , the environmental state will be determined by s t Transition to the next state s t+1 , and get reward r t+1 ; The transfer of its environmental state is determined by the migration function f:S×A→S. According to the migration situation, if the agent migrates to the next state, then, s t+1 =P(s t ,a t ) In the actual river network environment, enter the environmental state s t+1 The probability of being in interval S satisfies S122, Scheduling Experience Library Constraint Setting: Add gate scheduling domain knowledge constraints so that the reward function reflects not only the performance of the gate agent but also its interpretability under domain knowledge constraints. Adjust the agent's behavior by introducing knowledge constraints into the reward function. r′(s t ,a t )=r(s t ,a t )+λC(s t ,a t ) Among them, C(s t ,a t ) is a constraint function based on domain knowledge, which is a rule formulated according to water level constraints, flow constraints, and operation constraints; λ is a hyperparameter that controls the importance of constraints; h t is the water level at the current time t; h max is the maximum water level limit allowed at the current time t; h min is the minimum water level limit allowed at the current time t; Q t is the flow rate at the current time t; Q max is the maximum flow limit allowed at the current time t; Q min is the minimum flow limit allowed at the current time t; G t is the gate opening at the current time t; G t-1 is the gate opening at time t-1; α, β, and γ are the penalty weight coefficients for exceeding the limit of water level, flow rate, and gate opening respectively; On this basis, the influence of constraints is gradually relaxed, allowing the gate agent to conduct more autonomous exploration and optimize the scheduling strategy. Its mathematical form is: r′(s t ,a t )=r(s t ,a t )+λ t C(s t ,a t ) Among them, λ t is the attenuation coefficient that gradually decreases as the time step t increases.

5. The method for intelligent control of plain river network gate groups based on deep reinforcement learning according to claim 4 is characterized by: In the knowledge-constrained reinforcement learning module that combines domain knowledge with the experience base, a single gate is considered as a separate agent. The interaction process between the gate agent and the environment includes the state space S, action space A, reward function r(s,a) and transition probability P(s′|s,a); the agent learns the optimal strategy π in the interaction with the environment * (s), the goal is to maximize the cumulative reward; Among them, r(s t ,a t ) is state s t Take action a t Immediate reward when γ t is the agent’s discount factor at time t; π is the agent’s strategy.

6. The method for intelligent control of plain river network gate groups based on deep reinforcement learning according to claim 5 is characterized by: The construction process of the multi-agent reinforcement learning module is as follows: in the river network multi-agent system, the goal of each gate agent is to maximize its own cumulative reward through reinforcement learning. in, is the i-th gate agent in state s i Take action a i The state-action value function when r i t is the discount factor of the i-th gate at time t; The scheduling decision of each gate agent is not only affected by its own constraints, but also needs to consider the coordination with other agents; therefore, the global constraint is introduced into the reward function of each agent to ensure that the decisions of all agents are adjusted towards the global optimal goal. This is achieved by sharing the constraint information among all agents. The constraint reward function becomes, r i ′(s i ,a i )=r i (s i ,a i )+λC(s1,a1,…,s N ,a N ) Among them, C(s1,a1,…,s N ,a N ) is the global constraint function for the overall regulation of the river network.

7. The method for intelligently controlling a plain river network gate group based on deep reinforcement learning according to claim 1 is characterized in that: Step S2 specifically includes the following contents: S21. Integration with the flow prediction module: The flow prediction module receives hydrological and meteorological forecast rainfall data and upstream water flow data, uses the Transformer encoder to model the input state sequence, extracts time series features, and outputs a key section flow prediction sequence for the future period, with a time step of 1 hour. The predicted key section flow sequence serves as the environmental input of the knowledge-constrained reinforcement learning module to optimize the control decision-making. The solution cluster output by the knowledge-constrained reinforcement learning module is fed back to the flow prediction module for further correction training. S22. Integration with the Multi-Agent Reinforcement Learning Module: The single gate agent, calibrated by knowledge-constrained reinforcement learning, is embedded into the multi-agent deep reinforcement learning module. Using the multi-agent reinforcement learning framework, collaborative decision-making among multiple agents is achieved, dynamic optimization of river network flow regulation is achieved, and a comprehensive scheduling solution is obtained. S23. Adaptive intelligent scheduling mechanism: Through the integrated coupling of steps S21 and S22, a time-continuous learning structure is formed to achieve an adaptive closed loop of "model-environment-feedback-control and correction", forming an adaptive intelligent scheduling mechanism for the plain river network gate group and obtaining a plain river network intelligent control model.

8. The method for intelligently controlling a plain river network gate group based on deep reinforcement learning according to claim 1 is characterized by: Step S3 specifically includes the following contents: S31. Intelligent Control Model Training: Based on historical rainfall data, river network water level and flow, and gate scheduling process data, these data are input into the flow prediction module to drive the plain river network intelligent control model and construct a gate group control plan cluster; S32. Basin target and constraint setting: Set corresponding gate constraint parameters based on the scheduling target and operating conditions, so that the plain river network intelligent regulation model can complete reasonable scheduling under the established constraints, achieving an adaptive closed loop of "model-environment-feedback-regulation correction"; S33. Analysis of river network and sluice group control schemes: Based on the calibrated plain river network intelligent control model, access to future meteorological and hydrological forecast data is used to output the water level and flow processes of key sections and the sluice group control scheme cluster under the current upstream water inflow conditions of the river section, with a time step of 1 hour, to assist in supporting control decisions.

9. The method for intelligently controlling a plain river network sluice group based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: Before step S1, the method further includes: S0. Basic data collection and processing: Through physical model simulation and historical data collection, obtain the meteorological and hydrological time series data, river network gate opening and key section simulated flow data required for river network scheduling, and pre-process the corresponding data.

10. The method for intelligent control of plain river network gate groups based on deep reinforcement learning according to claim 9 is characterized in that: Step S0 specifically includes the following contents: S01. Acquisition of meteorological and hydrological time-series data: The meteorological and hydrological measured historical data required for river network operation include upstream water flow, river level and flow data at key sections, and rainfall data obtained through manual observation and automated telemetry. S02. Meteorological and hydrological time series data processing: Using sliding window normalization and time series alignment mechanisms, the time step of time series data is unified to 1 hour, achieving consistent alignment of multivariate data in the time dimension; For observation anomalies and missing points, reconstruction is completed through a dynamic repair algorithm based on spatial proximity and temporal proximity, improving the temporal accuracy of the input data in subsequent position encoding; S03. Acquisition of simulated flow data for river network gate openings and key sections: Using a physical process model, different water inflow processes and operating condition combinations were set to conduct numerical simulations of flood control for different typical flood events. The corresponding data sets for upstream river inflow, gate group openings, and key section flows under different typical flood events and operating condition combinations were obtained, with a time step of 1 hour. A control sample library covering extreme water inflow conditions and typical scheduling scenarios is constructed. The sample data in the sample library is encoded with a label mechanism and serves as a supervisory signal for different control strategy categories during the training calculation process, supporting the input channel encoding of the subsequent Transformer.

Images