A river channel management strategy intelligent optimization system based on reinforcement learning

Abstract

This invention discloses an intelligent optimization system for river management strategies based on reinforcement learning, comprising: a multi-source data modeling module, which acquires multi-source monitoring data, constructs a directed graph structure of the river network, and forms a river time-series data matrix; a spatiotemporal propagation feature extraction module, which calculates a time-varying propagation coefficient matrix based on an improved ASTGCN joint processing; a reinforcement learning decision generation module, which constructs a set of state vectors and management actions to generate candidate management actions; a causal bias correction module, which constructs a propagation contribution matrix, calculates causal bias, and corrects rewards; a reachability safety determination module, which constructs a multi-stage safety determiner and outputs corrected management actions; and a strategy iteration update module, which updates strategy parameters based on corrected rewards and corrected actions, and outputs a management scheduling strategy. This invention achieves adaptive optimization scheduling and red-line constraint execution of river management strategies by improving ASTGCN, causal bias correction, and multi-stage safety determination.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to an intelligent optimization system for river management strategies based on reinforcement learning. Background Technology

[0002] With the development of water environment management and smart water technology, river management is gradually shifting from manual experience-based decision-making to data-driven decision-making. Existing technologies typically construct river water environment assessment models based on water quality monitoring data, hydrological data, and pollution discharge data, and control gate scheduling, reservoir discharge flow, and pollution source reduction ratios through rule-based control methods, optimized scheduling algorithms, or traditional machine learning methods. Some solutions introduce reinforcement learning algorithms to optimize river scheduling strategies, achieving dynamic regulation of river water quality indicators by constructing state and action spaces. However, these methods often use static or simplified river network structures as environmental models, making it difficult to accurately reflect the complex water quality propagation relationships between upstream and downstream areas.

[0003] In existing reinforcement learning-based river management schemes, water quality changes are typically used directly as reward signals, lacking differentiation of the actual causal contribution of management actions. Due to external disturbances such as rainfall variations, upstream water flow fluctuations, and sudden pollution events, actual observed water quality changes often include components not caused by management actions. This leads to reward misguidance in reinforcement learning strategies during training, resulting in unstable strategies or strategies deviating from the true management objectives. Furthermore, most existing methods fail to structurally model the propagation paths of actions within the river network or decompose the propagation contributions of action impacts, making it difficult to achieve refined attribution of management effectiveness.

[0004] On the other hand, existing intelligent river management technologies typically use fixed thresholds as constraints on water quality or ecological flow, making compliance judgments only at the current moment and lacking accessibility analysis for multiple future time steps. When governance actions may lead to water quality exceeding standards or insufficient ecological flow during future propagation, existing technologies struggle to identify and correct this risk in a timely manner, easily resulting in strategies exceeding ecological red lines or creating delayed risks.

[0005] Therefore, how to provide an intelligent optimization system for river management strategies based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an intelligent optimization system for river management strategies based on reinforcement learning. This invention comprehensively incorporates an improved ASTGCN spatiotemporal feature extraction technique, a propagation contribution decomposition causal bias correction mechanism, and a multi-stage reachability safety determiner. It dynamically models the upstream and downstream water quality propagation relationship, structurally decomposes the actual propagation contribution of management actions, and predicts and corrects the safety reachability of states at multiple future time steps, thereby achieving adaptive optimization and full-process safety control of river management strategies. This invention can accurately characterize the time-varying propagation characteristics of river networks, reduce the misleading effect of external disturbances on reward signals, improve the accuracy of causal attribution of management actions, and effectively prevent strategies from crossing ecological red lines. It has the advantages of high decision-making stability, strong causal consistency, and high safety reliability.

[0007] An intelligent optimization system for river management strategies based on reinforcement learning according to an embodiment of the present invention includes: The multi-source data modeling module acquires multi-source monitoring data related to river management, defines each monitoring section as a node according to the river topology, constructs a directed graph structure of the river network, and forms a corresponding river time series data matrix. The spatiotemporal propagation feature extraction module, based on the improved ASTGCN model, performs joint processing on the directed graph structure of the river network and the river channel time series data matrix, extracts the spatiotemporal feature vector of each node at the current time, and calculates the time-varying propagation coefficient matrix between nodes. The reinforcement learning decision generation module constructs the state vector and governance action set of the reinforcement learning environment, initializes the reinforcement learning policy parameters, and generates corresponding candidate governance actions from the governance action set based on the reinforcement learning policy parameters and state vector, and calculates the immediate reward. The causal bias correction module constructs a propagation contribution matrix of candidate governance actions based on the current state vector and the time-varying propagation coefficient matrix, decomposes the contribution of actual observed state changes, calculates the causal bias value based on the consistency between the contribution decomposition result and the propagation contribution matrix, and corrects the immediate reward based on the causal bias value. The accessibility safety determination module constructs a multi-stage accessibility safety determiner, which generates a future state sequence based on the current state vector, time-varying propagation coefficient matrix and candidate governance actions, and performs safety determination on the future state sequence according to preset ecological flow threshold, water quality threshold and water level threshold, and outputs corrective governance actions that meet safety constraints. The strategy iteration and update module iteratively updates the reinforcement learning strategy parameters based on the corrected immediate rewards and corrected governance actions, and outputs the updated river governance strategy for river scheduling execution.

[0008] Optionally, the multi-source monitoring data includes water quality data, hydrological data, meteorological data, pollution source emission data, and water conservancy project scheduling data from upstream and downstream monitoring sections of the river.

[0009] Optionally, the construction of the directed graph structure of the river network to form the corresponding river channel time-series data matrix includes: Each monitoring section is defined as a graph node, and the upstream and downstream river segments with water flow connectivity are defined as directed edges, with the direction of the edges pointing from upstream to downstream, forming a directed graph structure of the river network. A river network adjacency matrix is ​​generated based on the spatial location relationship of the river channel and the water flow connectivity relationship. Multi-source monitoring data are time-aligned and synchronized according to a preset time step to form the data sequence corresponding to each node at each time step. The water quality data, hydrological data, meteorological data, and pollution discharge data of each node in a continuous time step are arranged in node order to construct a node feature matrix. The node feature matrix is ​​then stacked in time dimension to form a river channel time series data matrix.

[0010] Optionally, the spatiotemporal propagation feature extraction module includes: Align the adjacency relationship of the directed graph of the river network with the river channel time series data matrix, and organize the water quality data, hydrological data and meteorological data of each river node at each time step according to the node dimension and the time dimension to form three-dimensional node time series feature data. An improved ASTGCN model is constructed, comprising a joint topological hydrology coding layer, an upstream-downstream partitioned spatial attention map convolutional layer, and a multi-scale temporal attention convolutional layer. The topological hydrological joint coding layer is based on the adjacency relationship of the directed graph of the river network and the three-dimensional node temporal feature data. It introduces the river section flow direction, flow rate, river level and the intensity of hydraulic connection between upstream and downstream, and weights and filters the original adjacency relationship to generate an enhanced spatial correlation weight matrix. The upstream and downstream partition spatial attention map convolutional layer takes the enhanced spatial association weight matrix as input, calculates the spatial attention weight for each target node for the upstream node set and the downstream node set respectively, performs weighted aggregation on the neighbor features in the upstream and downstream directions respectively, and fuses the upstream aggregation result with the downstream aggregation result to obtain a spatial feature representation containing upstream and downstream directional information; The multi-scale temporal attention convolutional layer takes spatial feature representation as input, sets up multiple temporal convolutional branches according to short time scale, medium time scale and long time scale respectively, performs convolutional extraction on node features at each time scale, and performs weighted fusion of the outputs at each time scale through the temporal attention mechanism to obtain the multi-scale temporal feature representation of each node. Multi-scale time-series feature representations are concatenated with flow rate, velocity, and rainfall data at corresponding time steps. Through feature mapping and normalization, a time-varying propagation coefficient matrix is ​​generated that characterizes the intensity of each node's influence on water quality at each time step.

[0011] Optionally, the reinforcement learning decision generation module includes: At each time step, the water quality index values, water level values, flow rate values, pollutant discharge values, ecological flow constraint indicators, and time-varying propagation coefficient matrix of each river node are combined to form the state vector of the reinforcement learning environment. A reinforcement learning strategy based on the Actor-Critic structure is adopted to initialize the policy network and the value network. The policy network takes the state vector as input and outputs the selection probability corresponding to each candidate action in the governance action set. The value network takes the state vector as input and outputs the value evaluation value of the state. A set of governance actions is pre-defined, which includes gate opening adjustment instructions, reservoir discharge flow adjustment instructions, pollution source reduction ratio instructions, and ecological restoration measure execution instructions. At the current time step, the state vector is input into the strategy network, and candidate governance actions are selected from the set of governance actions based on the output of the strategy network. After executing a candidate governance action, the actual observed state vector of the next time step is obtained. Based on the current time step state vector and the time-varying propagation coefficient matrix, a counterfactual state vector is generated. The counterfactual state vector is compared with the actual observed state vector. The instant reward value is calculated based on the difference in water quality indicators, the ecological flow constraint results, and the action magnitude of the candidate governance action. The current state vector, the next state vector, and the instant reward value are input into the value network to obtain the advantage index of the candidate governance action. The parameters in the strategy network and value network are adjusted according to the advantage index to form a continuous state transition process between the state vector, candidate governance actions and the next state vector.

[0012] Optionally, the causal bias correction module includes: At the current time step, obtain the state vector composed of the water quality, hydrology, pollution discharge, ecological flow constraints and time-varying propagation coefficient matrix of each river node, as well as the candidate governance actions. Based on the gate opening adjustment, reservoir discharge flow adjustment, pollution source reduction ratio and ecological restoration measure implementation intensity among the candidate governance actions, determine the local action impact of each river node. Based on the time-varying propagation coefficient matrix and the local action impact, the propagation impact coefficient of the candidate governance action on each upstream-downstream river connection is calculated, and the propagation impact coefficients on each connection are arranged in the order of the river network topology to construct the initial propagation contribution matrix. In the initial propagation contribution matrix, the propagation influence coefficients from different action occurrence nodes in each column are normalized to obtain the normalized propagation contribution matrix, with each affected node as the column unit. After executing the candidate governance action, the actual observation state vector of the next time step is obtained. The differences in water quality index values, hydrological index values, and ecological flow constraint changes between the next time step and the current time step are combined in the order of river nodes to form the actual observation state change vector. The actual observation state change vector is decomposed in the dimension of the action occurrence node according to the normalized propagation contribution matrix to obtain the action contribution change vector attributable to the candidate governance action and the disturbance residual amount that cannot be explained by the propagation contribution. Based on the difference between the action contribution change vector and the theoretical propagation change derived from the time-varying propagation coefficient matrix and the local action influence quantity, as well as the magnitude of the disturbance residual quantity, the causal deviation value is calculated, and the instant reward value is adjusted by increasing or decreasing based on the causal deviation value to obtain the corrected instant reward.

[0013] Optionally, the reachability security determination module includes: A multi-stage reachability safety determiner is constructed, which includes a multi-stage reachability set generation unit, a redline safety envelope construction unit, and a minimum deviation action correction unit. The prediction steps are determined by the multi-stage reachable set generation unit based on the time-varying amplitude of the time-varying propagation coefficient matrix, the ecological flow constraint state in the state vector, and the action amplitude of the candidate governance actions. The river state at multiple future time steps is recursively generated according to the determined prediction steps to obtain the future state sequence. The red-line safety envelope construction unit establishes a safety envelope for each node corresponding to the prediction time step. The safety envelope is composed of ecological flow threshold, water quality threshold and water level threshold. When constructing the safety envelope, a time-varying safety margin is introduced for each threshold, and each threshold is expanded or contracted. The time-varying safety margin is determined according to the fluctuation degree of the time-varying propagation coefficient matrix of the corresponding prediction time step and the action amplitude of the candidate governance action. The red-line safety envelope construction unit performs consistency determination between the future state sequence and the corresponding safety envelope step by step and node by node, records the predicted time step, node number and excess risk that triggers the excess risk, and generates a constraint indication sequence based on the recorded excess risk. The minimum deviation action correction unit adjusts the candidate governance actions by component level under the constraint instruction sequence. After each adjustment, the multi-stage reachable set generation unit is called again to generate an updated future state sequence and complete the consistency determination until a corrected governance action that does not trigger the risk of exceeding the limit is obtained in all prediction time steps. The corrected governance action is then output as a governance action that satisfies the safety constraints.

[0014] Optionally, the strategy iteration update module includes: At time step t, obtain the current state vector, the correction and governance action output by the multi-stage reachability safety determiner, and the immediate reward after correction. After executing the correction and governance action, obtain the state vector for the next time step. In the reinforcement learning strategy based on the Actor-Critic structure, the current state vector and the next time step state vector are input into the value network to obtain the value evaluation results of the current state and the value evaluation results of the next state. The time difference error is calculated based on the corrected immediate reward, the value assessment result of the current state, and the value assessment result of the next state. The time difference error is obtained by subtracting the value assessment result of the current state from the weighted sum of the corrected immediate reward and the value assessment result of the next state. The parameters of the value network are updated based on the time difference error, and the selection probability of the candidate governance action output by the policy network is adjusted based on the time difference error, so that the selection probability of the corresponding governance action increases when the time difference error is positive and decreases when the time difference error is negative. After updating the parameters of the value network and policy network, new reinforcement learning policy parameters are formed, and the river management strategy is continuously iteratively optimized.

[0015] The beneficial effects of this invention are: This invention maps river topology and multi-source monitoring data into a directed river network graph and a river temporal data matrix. Based on an improved ASTGCN model, it extracts the spatiotemporal characteristics of the river network and calculates the time-varying propagation coefficient matrix between nodes. This allows river management decisions to characterize the propagation relationship between upstream and downstream areas as conditions change within a unified spatiotemporal framework. Compared to existing technologies that use static relationships or simplified propagation assumptions, this invention can generate a propagation representation that matches the directionality and time-varying nature of the river network before the management strategy is generated. This improves the consistency and usability of state representation and enhances the adaptability of the strategy under different hydrological scenarios.

[0016] This invention further constructs a propagation contribution matrix for candidate governance actions based on a time-varying propagation coefficient matrix, and decomposes the contribution of actual observed state changes. The causal bias value is calculated by comparing the consistency between the contribution decomposition result and the propagation contribution matrix, and the immediate reward is corrected accordingly. This suppresses reward shifts introduced by external disturbances such as rainfall fluctuations, upstream water inflow changes, and sudden pollution during reinforcement learning training. Compared with existing technologies that directly use water quality changes as rewards and struggle to distinguish the true contribution of governance actions, this invention makes the reward signal more consistent with the action propagation mechanism, reducing the risk of misjudgment and misoptimization, and improving the stability and effectiveness of iterative updates to governance strategies.

[0017] This invention constructs a multi-stage reachability safety determiner, which predicts and determines the safety of future state sequences after candidate governance actions are generated, and outputs corrective governance actions that meet safety constraints when there is a risk of exceeding limits. This achieves forward-looking constraint control of ecological flow thresholds, water quality thresholds, and water level thresholds. Compared with existing technologies that mostly use single-moment threshold verification or ex-post correction schemes, this invention can complete multi-time-step risk identification and action correction before strategy execution, avoiding crossing ecological red lines or causing delayed exceedance risks, and improving the safety, reliability, and engineering feasibility of river management and scheduling. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an intelligent optimization system for river management strategies based on reinforcement learning proposed in this invention. Figure 2 This is a schematic diagram of the improved ASTGCN model for extracting spatiotemporal feature vectors and calculating time-varying propagation coefficient matrices in an intelligent optimization system for river management strategies based on reinforcement learning proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figure 1 and Figure 2 A smart optimization system for river management strategies based on reinforcement learning, comprising: The multi-source data modeling module acquires multi-source monitoring data related to river management, defines each monitoring section as a node according to the river topology, constructs a directed graph structure of the river network, and forms a corresponding river time series data matrix. The spatiotemporal propagation feature extraction module, based on the improved ASTGCN model, performs joint processing on the directed graph structure of the river network and the river channel time series data matrix, extracts the spatiotemporal feature vector of each node at the current time, and calculates the time-varying propagation coefficient matrix between nodes. The reinforcement learning decision generation module constructs the state vector and governance action set of the reinforcement learning environment, initializes the reinforcement learning policy parameters, and generates corresponding candidate governance actions from the governance action set based on the reinforcement learning policy parameters and state vector, and calculates the immediate reward. The causal bias correction module constructs a propagation contribution matrix of candidate governance actions based on the current state vector and the time-varying propagation coefficient matrix, decomposes the contribution of actual observed state changes, calculates the causal bias value based on the consistency between the contribution decomposition result and the propagation contribution matrix, and corrects the immediate reward based on the causal bias value. The accessibility safety determination module constructs a multi-stage accessibility safety determiner, which generates a future state sequence based on the current state vector, time-varying propagation coefficient matrix and candidate governance actions, and performs safety determination on the future state sequence according to preset ecological flow threshold, water quality threshold and water level threshold, and outputs corrective governance actions that meet safety constraints. The strategy iteration and update module iteratively updates the reinforcement learning strategy parameters based on the corrected immediate rewards and corrected governance actions, and outputs the updated river governance strategy for river scheduling execution.

[0021] In this embodiment, the multi-source monitoring data includes water quality data, hydrological data, meteorological data, pollution source emission data, and water conservancy project scheduling data from upstream and downstream monitoring sections of the river.

[0022] In this embodiment, the construction of the directed graph structure of the river network to form the corresponding river channel time-series data matrix includes: Each monitoring section is defined as a graph node, and the upstream and downstream river segments with water flow connectivity are defined as directed edges, with the direction of the edges pointing from upstream to downstream, forming a directed graph structure of the river network. A river network adjacency matrix is ​​generated based on the spatial location relationship of the river channel and the water flow connectivity relationship. Multi-source monitoring data are time-aligned and synchronized according to a preset time step to form the data sequence corresponding to each node at each time step, where the preset time step is 1 hour. The water quality data, hydrological data, meteorological data, and pollution discharge data of each node in a continuous time step are arranged in node order to construct a node feature matrix. The node feature matrix is ​​then stacked in time dimension to form a river channel time series data matrix.

[0023] In this embodiment, the spatiotemporal propagation feature extraction module includes: Align the adjacency relationship of the directed graph of the river network with the river channel time series data matrix, and organize the water quality data, hydrological data and meteorological data of each river node at each time step according to the node dimension and the time dimension to form three-dimensional node time series feature data. An improved ASTGCN model is constructed, comprising a joint topological hydrology coding layer, an upstream-downstream partitioned spatial attention map convolutional layer, and a multi-scale temporal attention convolutional layer. The topological hydrological joint coding layer is based on the adjacency relationship of the directed graph of the river network and the three-dimensional node temporal feature data. It introduces the river section flow direction, flow rate, river level and the intensity of hydraulic connection between upstream and downstream, and weights and filters the original adjacency relationship to generate an enhanced spatial correlation weight matrix. The upstream and downstream spatial attention graph convolutional layer takes the enhanced spatial association weight matrix as input. For each target node, it calculates spatial attention weights for both the upstream and downstream node sets. It then performs weighted aggregation of neighbor features in the upstream and downstream directions, and merges the upstream and downstream aggregation results to obtain a spatial feature representation containing upstream and downstream directional information. Specifically, the spatial attention weights for each target node are calculated for both the upstream and downstream node sets as follows: For the target node, the upstream and downstream adjacent subsets are first extracted from the enhanced spatial association weight matrix according to the direction of the directed edge of the river network, forming the upstream association weight vector and the downstream association weight vector respectively. The target node features and the corresponding neighbor node features are taken as the input feature pairs for attention calculation. For each upstream neighbor node in the upstream adjacent subset, the target node feature and the upstream neighbor node feature are concatenated and then input into the shared attention scoring map to obtain the upstream attention score value. The upstream attention score value is multiplied by the corresponding upstream association weight to obtain the upstream unnormalized attention weight. For each downstream neighbor node in the downstream adjacent subset, the downstream attention score value is obtained in the same way. The downstream attention score value is multiplied by the corresponding downstream association weight to obtain the downstream unnormalized attention weight. The upstream unnormalized attention weights are normalized within the upstream adjacent subset to obtain the upstream spatial attention weights, and the downstream unnormalized attention weights are normalized within the downstream adjacent subset to obtain the downstream spatial attention weights, thus obtaining the spatial attention weights for the upstream node set and the downstream node set. The multi-scale temporal attention convolutional layer takes spatial feature representation as input, sets up multiple temporal convolutional branches according to short time scale, medium time scale and long time scale respectively, performs convolutional extraction on node features at each time scale, and performs weighted fusion of the outputs at each time scale through the temporal attention mechanism to obtain the multi-scale temporal feature representation of each node. Multi-scale time-series feature representations are concatenated with flow rate, velocity, and rainfall data at corresponding time steps. Through feature mapping and normalization, a time-varying propagation coefficient matrix is ​​generated that characterizes the intensity of each node's influence on water quality at each time step.

[0024] In this embodiment, the reinforcement learning decision generation module includes: At each time step, the water quality index values, water level values, flow rate values, pollutant discharge values, ecological flow constraint indicators, and time-varying propagation coefficient matrix of each river node are combined to form the state vector of the reinforcement learning environment. A reinforcement learning strategy based on the Actor-Critic structure is adopted to initialize the policy network and the value network. The policy network takes the state vector as input and outputs the selection probability corresponding to each candidate action in the governance action set. The value network takes the state vector as input and outputs the value evaluation value of the state, where the output is the selection probability corresponding to each candidate action in the governance action set. Specifically: The policy network performs forward computation on the state vector to obtain a set of unnormalized action scores. The unnormalized action scores correspond one-to-one with each candidate action in the governance action set. The numerical range of the unnormalized action scores is not subject to probability constraints. Perform exponential mapping and normalization on the unnormalized action scores. Divide the exponential mapping result of each candidate action by the sum of the exponential mapping results of all candidate actions to obtain the selection probability of each candidate action, so that all selection probabilities are positive and the sum is 1. When the set of governance actions contains multiple action components, the unnormalized action scores are grouped according to the action components and subjected to exponential mapping and normalization processing respectively to obtain the selection probability distribution of each action component. The selection probability distribution of each action component is then combined to form the selection probability output of the candidate governance actions. The value assessment of the output state is as follows: After the state vector is input into the input layer of the value network, it is transformed through two fully connected layers in sequence. After each layer of mapping, nonlinear activation is performed to obtain the hidden feature vector. The hidden feature vectors are input into the value output layer of the value network. The value output layer linearly combines the hidden feature vectors and adds a bias term, outputting a single scalar as the value assessment value, so that the value assessment value is independent of the set of governance actions and corresponds only to the current state vector. The value assessment values ​​are constrained within a preset range, determined by the upper and lower bounds of the reward values, to obtain the state value assessment values. A set of governance actions is pre-defined, which includes gate opening adjustment instructions, reservoir discharge flow adjustment instructions, pollution source reduction ratio instructions, and ecological restoration measure execution instructions. At the current time step, the state vector is input into the strategy network, and candidate governance actions are selected from the set of governance actions based on the output of the strategy network. After executing a candidate governance action, the actual observed state vector for the next time step is obtained. A counterfactual state vector is generated based on the current time step state vector and the time-varying propagation coefficient matrix. The counterfactual state vector is compared with the actual observed state vector. An immediate reward value is calculated based on the difference in water quality indicators, ecological flow constraints, and the magnitude of the candidate governance action. The current state vector, the next state vector, and the immediate reward value are input into the value network to obtain an advantage index for the candidate governance action, where: The counterfactual state vector is generated based on the current time step state vector and the time-varying propagation coefficient matrix, specifically as follows: Determine the baseline conditions for generating counterfactual events, and use the water quality status, hydrological status, pollution discharge status and ecological flow constraint status in the current time step state vector as the baseline input. Set the gate opening adjustment, reservoir discharge flow adjustment, pollution source reduction ratio and ecological restoration measure execution intensity in the candidate governance actions to zero actions or keep the current execution value unchanged as the counterfactual action configuration. Based on the time-varying propagation coefficient matrix, the propagation influence weight of each node on its neighboring nodes is extracted. Combined with the water quality and hydrological status at the current time step, the water quality indicators of each node are recursively updated in the directed topological order of the river network to obtain the predicted values ​​of the water quality indicators of each node at the next time step without the execution of candidate governance actions. At the same time, the update rules of the ecological flow constraint state are consistent under the counterfactual action configuration. The predicted water quality indicators of each node in the next time step of the counterfactual process are combined with the water level, flow rate and ecological flow constraint states updated under the counterfactual action configuration in the form of a state vector to form the counterfactual state vector of the next time step. The immediate reward value is calculated based on the difference in water quality indicators, the ecological flow constraint results, and the magnitude of candidate treatment actions. Obtain the corresponding values ​​of water quality indicators at each node in the counterfactual state vector and the actual observed state vector. Calculate the differences in chemical oxygen demand, ammonia nitrogen, total phosphorus, and dissolved oxygen in the order of the nodes. Then, linearly combine the differences of each indicator according to the preset indicator weight coefficients to obtain the water quality difference score. Obtain the ecological flow constraint results in the actual observed state vector, determine whether each node meets the preset ecological flow threshold in the next time step, and take the preset positive constant for the ecological flow constraint score when all nodes meet the threshold, and take the preset negative constant for the ecological flow constraint score when any node does not meet the threshold. The action range of candidate governance actions is obtained, including changes in gate opening, changes in reservoir discharge flow, changes in pollution source reduction ratio, and changes in the intensity of ecological restoration measures. The action ranges are linearly combined according to preset cost weight coefficients to obtain action cost scores. The water quality difference score, ecological flow constraint score, and action cost score are combined according to preset weighting relationship to form an instant reward value. The indicators for determining the dominance of candidate governance actions are as follows: Input the current state vector and the next time step state vector into the value network to obtain the value assessment value of the current state and the value assessment value of the next state. Then, perform a weighted sum of the instant reward value and the value assessment value of the next state to obtain the target value value corresponding to the current state. The difference between the target value and the current value assessment is used as an indicator of the advantage of candidate governance actions. The parameters in the strategy network and value network are adjusted according to the advantage index to form a continuous state transition process between the state vector, candidate governance actions and the next state vector.

[0025] In this embodiment, the causal deviation correction module includes: At the current time step, obtain the state vector composed of the water quality, hydrology, pollution discharge, ecological flow constraints and time-varying propagation coefficient matrix of each river node, as well as the candidate governance actions. Based on the gate opening adjustment, reservoir discharge flow adjustment, pollution source reduction ratio and ecological restoration measure implementation intensity among the candidate governance actions, determine the local action impact of each river node. Based on the time-varying propagation coefficient matrix and the local action impact, the propagation impact coefficient of the candidate governance action on each upstream-downstream river connection is calculated. The propagation impact coefficients on each connection are arranged in the order of the river network topology to construct an initial propagation contribution matrix. Specifically, the propagation impact coefficient of the candidate governance action on each upstream-downstream river connection is calculated as follows: For each directed connection from an upstream node to a downstream node in the river network, the connection weight in the time-varying propagation coefficient matrix is ​​read, the local action influence of the upstream node is read, and the connection weight and the local action influence are combined to obtain the basic propagation influence value of the connection at the current time step. When a candidate governance action contains multiple action components, the basic propagation impact value corresponding to each component is calculated for gate opening adjustment, reservoir discharge flow adjustment, pollution source reduction ratio and ecological restoration measure execution intensity. The basic propagation impact values ​​of each component are linearly synthesized according to the preset component weights to obtain the connected comprehensive propagation impact value as the propagation impact coefficient. The comprehensive propagation impact value is subjected to sign consistency verification and range pruning to ensure that the comprehensive propagation impact value is consistent with the river flow direction and falls within a preset value range. The preset value range is jointly determined by the value range of the propagation coefficient matrix and the value range of the local action impact amount, thus obtaining the propagation impact coefficient of the candidate governance action on the connection. In the initial propagation contribution matrix, the propagation influence coefficients from different action occurrence nodes in each column are normalized to obtain the normalized propagation contribution matrix, with each affected node as the column unit. After executing the candidate governance action, the actual observation state vector of the next time step is obtained. The differences in water quality index values, hydrological index values, and ecological flow constraint changes between the next time step and the current time step are combined in the order of river nodes to form the actual observation state change vector. The actual observation state change vector is decomposed in the dimension of the action occurrence node according to the normalized propagation contribution matrix to obtain the action contribution change vector attributable to the candidate governance action and the disturbance residual amount that cannot be explained by the propagation contribution. Based on the difference between the action contribution change vector and the theoretical propagation change derived from the time-varying propagation coefficient matrix and the local action influence, as well as the magnitude of the perturbation residual, a causal bias value is calculated. The instant reward value is then adjusted by increasing or decreasing based on this causal bias value to obtain the corrected instant reward. Specifically, the calculation of the causal bias value involves: Obtain the action contribution change vector of each node, generate the theoretical propagation change of each node based on the time-varying propagation coefficient matrix and the local action influence, align the two one by one according to the node order and calculate the node difference. The node difference is the absolute difference between the action contribution change and the theoretical propagation change of the corresponding node. Obtain the magnitude of the disturbance residual, normalize the disturbance residual, and obtain the disturbance intensity value; The causal deviation value is obtained by summing up all the node differences and linearly combining them with the disturbance intensity value according to a preset weight.

[0026] In this embodiment, the reachability security determination module includes: A multi-stage reachability safety determiner is constructed, which includes a multi-stage reachability set generation unit, a redline safety envelope construction unit, and a minimum deviation action correction unit. The prediction steps are determined by the multi-stage reachable set generation unit based on the time-varying amplitude of the time-varying propagation coefficient matrix, the ecological flow constraint state in the state vector, and the action amplitude of the candidate governance actions. The river state at multiple future time steps is recursively generated according to the determined prediction steps to obtain the future state sequence. The red-line safety envelope construction unit establishes a safety envelope for each node corresponding to the prediction time step. The safety envelope is composed of ecological flow threshold, water quality threshold and water level threshold. When constructing the safety envelope, a time-varying safety margin is introduced for each threshold, and each threshold is expanded or contracted. The time-varying safety margin is determined according to the fluctuation degree of the time-varying propagation coefficient matrix of the corresponding prediction time step and the action amplitude of the candidate governance action. The red-line safety envelope construction unit performs consistency checks on the future state sequence step-by-step and node-by-node with the corresponding safety envelope, records the predicted time step, node number, and excess risk type, and generates a constraint indication sequence based on the recorded excess risk. Specifically, the consistency checks on the future state sequence step-by-step and node-by-node with the corresponding safety envelope are performed as follows: For each predicted time step in the future state sequence, the water quality index values, water level values, and flow rate values ​​of each node are read, and the safety envelope boundary corresponding to the predicted time step and node is read. The safety envelope boundary includes the water quality threshold boundary, water level threshold boundary, and ecological flow threshold boundary of the node. For each node, a threshold consistency check is performed. The node's water quality index value is compared with the corresponding water quality threshold boundary, the node's water level value is compared with the corresponding water level threshold boundary, and the node's flow value is compared with the corresponding ecological flow threshold boundary. When any comparison result exceeds the corresponding threshold boundary, the node is determined to have exceeded the limit at the prediction time step. Nodes and predicted time steps that are determined to exceed limits are marked, and the types of exceeding limits are determined according to the source of exceeding limits, such as water quality exceeding limits, water level exceeding limits, or insufficient ecological flow; nodes that have not exceeded limits are determined to be consistent; when any node exceeds the limit in any predicted time step in the future state sequence, the candidate governance action is determined to have the risk of exceeding limits. The minimum deviation action correction unit adjusts the candidate governance actions by component level under the constraint instruction sequence. After each adjustment, the multi-stage reachable set generation unit is called again to generate an updated future state sequence and complete the consistency determination until a corrected governance action that does not trigger the risk of exceeding the limit is obtained in all prediction time steps. The corrected governance action is then output as a governance action that satisfies the safety constraints.

[0027] In this embodiment, the strategy iteration update module includes: At time step t, obtain the current state vector, the correction and governance action output by the multi-stage reachability safety determiner, and the immediate reward after correction. After executing the correction and governance action, obtain the state vector for the next time step. In the reinforcement learning strategy based on the Actor-Critic structure, the current state vector and the next time step state vector are input into the value network to obtain the value evaluation results of the current state and the value evaluation results of the next state. The time difference error is calculated based on the corrected immediate reward, the value assessment result of the current state, and the value assessment result of the next state. The time difference error is obtained by subtracting the value assessment result of the current state from the weighted sum of the corrected immediate reward and the value assessment result of the next state. The parameters of the value network are updated based on the time difference error, and the selection probability of the candidate governance action output by the policy network is adjusted based on the time difference error, so that the selection probability of the corresponding governance action increases when the time difference error is positive and decreases when the time difference error is negative. After updating the parameters of the value network and policy network, new reinforcement learning policy parameters are formed, and the river management strategy is continuously iteratively optimized. Example 1:

[0028] To verify the feasibility of this invention in practice, it was applied to a river in a main urban area. This river is approximately 14.3 kilometers long and has nine automatic water quality monitoring sections, two regulating gates, and one small water replenishment pumping station. The upstream section is a suburban agricultural area, while the downstream section enters an urban landscape section. For a long time, this river section has suffered from insufficient ecological flow and reduced self-purification capacity during the dry season. Ammonia nitrogen and total phosphorus levels tend to approach the upper limit of the Class IV surface water standard during the high-temperature summer period. The original scheduling method relied on fixed threshold control and manual experience-based judgment, increasing the water replenishment flow when water quality approached the limit. However, due to a lack of precise characterization of the upstream-downstream transmission relationship, scheduling was often delayed. Furthermore, during periods of heavy rainfall, the source of water quality improvement was easily misjudged, with the dilution effect of rainfall mistaken for the scheduling effect, leading to unstable strategies.

[0029] In this scenario, hourly monitoring data for six months from May to October were first collected, including chemical oxygen demand (COD), ammonia nitrogen, total phosphorus, dissolved oxygen, water level, flow rate, rainfall, and outfall discharge data. Nine cross-sections were defined as nodes, and a directed graph structure of the river network was constructed based on the river connectivity, forming a river time-series data matrix with nine nodes and a time step of one hour. Subsequently, the directed graph structure of the river network and the river time-series data matrix were jointly processed based on the improved ASTGCN model to extract the spatiotemporal feature vectors of each node and calculate the time-varying propagation coefficient matrix between nodes. Field analysis shows that during the flood season from July to September, the propagation intensity of upstream cross-sections to downstream cross-sections is on average 15% to 20% higher than during the dry season in May, indicating that the model can reflect the impact of changes in hydrodynamic conditions on pollutant migration paths.

[0030] Constraints related to water quality, hydrology, and ecological flow, along with a time-varying propagation coefficient matrix, are used to construct a reinforcement learning state vector. This vector is used to build a set of governance actions, including gate opening regulation, reservoir replenishment flow regulation, and discharge reduction ratio. A reinforcement learning strategy based on an Actor-Critic structure is then used to generate candidate governance actions. After executing a candidate governance action, an immediate reward is calculated based on water quality changes. Subsequently, a propagation contribution matrix for the candidate governance actions is constructed based on the time-varying propagation coefficient matrix. The contribution of actual observed state changes is decomposed, and the causal bias value is calculated based on the consistency between the contribution decomposition result and the propagation contribution matrix to correct the immediate reward. Taking a moderate-intensity rainfall event lasting 6 hours in August as an example, traditional methods assume that the 0.28 mg / L decrease in ammonia nitrogen is entirely due to gate scheduling. However, this invention, through contribution decomposition, identifies approximately 0.19 mg / L as originating from rainfall runoff dilution, and only 0.09 mg / L as originating from scheduling behavior. This reduces the current reward value and prevents the strategy from amplifying the scheduling intensity in subsequent training.

[0031] A multi-stage reachability safety arbiter was constructed to predict the status for the next 24 hours and make safety determinations based on ecological flow thresholds, water quality thresholds, and water level thresholds. During a test in September's dry season, the system predicted that if the original candidate scheduling actions were maintained, dissolved oxygen at two cross-sections might drop below 3.0 mg / L within the next 18 hours. After adjustment by the safety arbiter, the water replenishment flow rate was increased by approximately 2 cubic meters per second, maintaining the predicted minimum dissolved oxygen above 3.2 mg / L. Finally, the reinforcement learning strategy parameters were iteratively updated based on the corrected immediate rewards and corrected governance actions to achieve continuous optimization.

[0032] After six months of comparative operation, the method of this invention has achieved improved water quality stability without significantly increasing the frequency of scheduling.

[0033] Table 1. Comparison of River Operation Indicators Before and After Implementation (May to October) According to the data in Table 1 above, in terms of water quality indicators, after implementing the method of this invention, ammonia nitrogen decreased from 1.92 mg / L to 1.65 mg / L, and total phosphorus decreased from 0.27 mg / L to 0.23 mg / L, with a reduction of approximately 14% in both cases. This represents a relatively stable and achievable level of improvement in urban river management. The minimum dissolved oxygen level during the dry season increased from 2.8 mg / L to 3.2 mg / L, keeping it consistently above the ecological warning value of 3.0 mg / L. This indicates that the introduction of time-varying propagation modeling and multi-stage safety assessment has made the control of hypoxia risk more effective, and improved the self-purification conditions of the water body.

[0034] Regarding ecological flow and scheduling efficiency, the number of hours with ecological flow below the threshold decreased from 46 hours to 18 hours, significantly reducing the risk of cross-sectional imbalance during the dry season. Scheduling response time shortened from 3.6 hours to 2.1 hours, indicating that reinforcement learning strategies can generate governance actions more quickly. The number of gate openings and closings increased only from 22 times per month to 24 times per month, with the increase controlled within 10%, indicating that while water quality improved, there was no excessively frequent scheduling, and the operational intensity remained within an acceptable range for the project.

[0035] From the perspective of strategy stability, the standard deviation of reward fluctuation decreased from 0.34 to 0.21, a decrease of approximately 38%, indicating that the propagation contribution decomposition and causal bias correction mechanisms weakened the impact of external disturbances such as rainfall on the reward signal, making strategy updates more stable. Overall, the data show that this invention achieves coordinated improvements in water quality, ecological security, and decision-making stability, with a reasonable improvement range that aligns with actual river operation characteristics.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A smart optimization system for river management strategies based on reinforcement learning, characterized in that, include: The multi-source data modeling module acquires multi-source monitoring data related to river management, defines each monitoring section as a node according to the river topology, constructs a directed graph structure of the river network, and forms a corresponding river time series data matrix. The spatiotemporal propagation feature extraction module, based on the improved ASTGCN model, performs joint processing on the directed graph structure of the river network and the river channel time series data matrix, extracts the spatiotemporal feature vector of each node at the current time, and calculates the time-varying propagation coefficient matrix between nodes. The reinforcement learning decision generation module constructs the state vector and governance action set of the reinforcement learning environment, initializes the reinforcement learning policy parameters, and generates corresponding candidate governance actions from the governance action set based on the reinforcement learning policy parameters and state vector, and calculates the immediate reward. The causal bias correction module constructs a propagation contribution matrix of candidate governance actions based on the current state vector and the time-varying propagation coefficient matrix, decomposes the contribution of actual observed state changes, calculates the causal bias value based on the consistency between the contribution decomposition result and the propagation contribution matrix, and corrects the immediate reward based on the causal bias value. The accessibility safety determination module constructs a multi-stage accessibility safety determiner, which generates a future state sequence based on the current state vector, time-varying propagation coefficient matrix and candidate governance actions, and performs safety determination on the future state sequence according to preset ecological flow threshold, water quality threshold and water level threshold, and outputs corrective governance actions that meet safety constraints. The strategy iteration and update module iteratively updates the reinforcement learning strategy parameters based on the corrected immediate rewards and corrected governance actions, and outputs the updated river governance strategy for river scheduling execution.

2. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The multi-source monitoring data includes water quality data, hydrological data, meteorological data, pollution source emission data, and water conservancy project scheduling data from upstream and downstream monitoring sections of the river.

3. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The construction of the directed graph structure of the river network, forming the corresponding river channel time-series data matrix, includes: Each monitoring section is defined as a graph node, and the upstream and downstream river segments with water flow connectivity are defined as directed edges, with the direction of the edges pointing from upstream to downstream, forming a directed graph structure of the river network. A river network adjacency matrix is ​​generated based on the spatial location relationship of the river channel and the water flow connectivity relationship. Multi-source monitoring data are time-aligned and synchronized according to a preset time step to form the data sequence corresponding to each node at each time step. The water quality data, hydrological data, meteorological data, and pollution discharge data of each node in a continuous time step are arranged in node order to construct a node feature matrix. The node feature matrix is ​​then stacked in time dimension to form a river channel time series data matrix.

4. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The spatiotemporal propagation feature extraction module includes: Align the adjacency relationship of the directed graph of the river network with the river channel time series data matrix, and organize the water quality data, hydrological data and meteorological data of each river node at each time step according to the node dimension and the time dimension to form three-dimensional node time series feature data. An improved ASTGCN model is constructed, comprising a joint topological hydrology coding layer, an upstream-downstream partitioned spatial attention map convolutional layer, and a multi-scale temporal attention convolutional layer. The topological hydrological joint coding layer is based on the adjacency relationship of the directed graph of the river network and the three-dimensional node temporal feature data. It introduces the river section flow direction, flow rate, river level and the intensity of hydraulic connection between upstream and downstream, and weights and filters the original adjacency relationship to generate an enhanced spatial correlation weight matrix. The upstream and downstream partition spatial attention map convolutional layer takes the enhanced spatial association weight matrix as input, calculates the spatial attention weight for each target node for the upstream node set and the downstream node set respectively, performs weighted aggregation on the neighbor features in the upstream and downstream directions respectively, and fuses the upstream aggregation result with the downstream aggregation result to obtain a spatial feature representation containing upstream and downstream directional information; The multi-scale temporal attention convolutional layer takes spatial feature representation as input, sets up multiple temporal convolutional branches according to short time scale, medium time scale and long time scale respectively, performs convolutional extraction on node features at each time scale, and performs weighted fusion of the outputs at each time scale through the temporal attention mechanism to obtain the multi-scale temporal feature representation of each node. Multi-scale time-series feature representations are concatenated with flow rate, velocity, and rainfall data at corresponding time steps. Through feature mapping and normalization, a time-varying propagation coefficient matrix is ​​generated that characterizes the intensity of each node's influence on water quality at each time step.

5. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning decision generation module includes: At each time step, the water quality index values, water level values, flow rate values, pollutant discharge values, ecological flow constraint indicators, and time-varying propagation coefficient matrix of each river node are combined to form the state vector of the reinforcement learning environment. A reinforcement learning strategy based on the Actor-Critic structure is adopted to initialize the policy network and the value network. The policy network takes the state vector as input and outputs the selection probability corresponding to each candidate action in the governance action set. The value network takes the state vector as input and outputs the value evaluation value of the state. A set of governance actions is pre-defined, which includes gate opening adjustment instructions, reservoir discharge flow adjustment instructions, pollution source reduction ratio instructions, and ecological restoration measure execution instructions. At the current time step, the state vector is input into the strategy network, and candidate governance actions are selected from the set of governance actions based on the output of the strategy network. After executing a candidate governance action, the actual observed state vector of the next time step is obtained. Based on the current time step state vector and the time-varying propagation coefficient matrix, a counterfactual state vector is generated. The counterfactual state vector is compared with the actual observed state vector. The instant reward value is calculated based on the difference in water quality indicators, the ecological flow constraint results, and the action magnitude of the candidate governance action. The current state vector, the next state vector, and the instant reward value are input into the value network to obtain the advantage index of the candidate governance action. The parameters in the strategy network and value network are adjusted according to the advantage index to form a continuous state transition process between the state vector, candidate governance actions and the next state vector.

6. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The causal bias correction module includes: At the current time step, obtain the state vector composed of the water quality, hydrology, pollution discharge, ecological flow constraints and time-varying propagation coefficient matrix of each river node, as well as the candidate governance actions. Based on the gate opening adjustment, reservoir discharge flow adjustment, pollution source reduction ratio and ecological restoration measure implementation intensity among the candidate governance actions, determine the local action impact of each river node. Based on the time-varying propagation coefficient matrix and the local action impact, the propagation impact coefficient of the candidate governance action on each upstream-downstream river connection is calculated, and the propagation impact coefficients on each connection are arranged in the order of the river network topology to construct the initial propagation contribution matrix. In the initial propagation contribution matrix, the propagation influence coefficients from different action occurrence nodes in each column are normalized to obtain the normalized propagation contribution matrix, with each affected node as the column unit. After executing the candidate governance action, the actual observation state vector of the next time step is obtained. The differences in water quality index values, hydrological index values, and ecological flow constraint changes between the next time step and the current time step are combined in the order of river nodes to form the actual observation state change vector. The actual observation state change vector is decomposed in the dimension of the action occurrence node according to the normalized propagation contribution matrix to obtain the action contribution change vector attributable to the candidate governance action and the disturbance residual amount that cannot be explained by the propagation contribution. Based on the difference between the action contribution change vector and the theoretical propagation change derived from the time-varying propagation coefficient matrix and the local action influence quantity, as well as the magnitude of the disturbance residual quantity, the causal deviation value is calculated, and the instant reward value is adjusted by increasing or decreasing based on the causal deviation value to obtain the corrected instant reward.

7. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The reachability security determination module includes: A multi-stage reachability safety determiner is constructed, which includes a multi-stage reachability set generation unit, a redline safety envelope construction unit, and a minimum deviation action correction unit. The prediction steps are determined by the multi-stage reachable set generation unit based on the time-varying amplitude of the time-varying propagation coefficient matrix, the ecological flow constraint state in the state vector, and the action amplitude of the candidate governance actions. The river state at multiple future time steps is recursively generated according to the determined prediction steps to obtain the future state sequence. The red-line safety envelope construction unit establishes a safety envelope for each node corresponding to the prediction time step. The safety envelope is composed of ecological flow threshold, water quality threshold and water level threshold. When constructing the safety envelope, a time-varying safety margin is introduced for each threshold, and each threshold is expanded or contracted. The time-varying safety margin is determined according to the fluctuation degree of the time-varying propagation coefficient matrix of the corresponding prediction time step and the action amplitude of the candidate governance action. The red-line safety envelope construction unit performs consistency determination between the future state sequence and the corresponding safety envelope step by step and node by node, records the predicted time step, node number and excess risk that triggers the excess risk, and generates a constraint indication sequence based on the recorded excess risk. The minimum deviation action correction unit adjusts the candidate governance actions by component level under the constraint instruction sequence. After each adjustment, the multi-stage reachable set generation unit is called again to generate an updated future state sequence and complete the consistency determination until a corrected governance action that does not trigger the risk of exceeding the limit is obtained in all prediction time steps. The corrected governance action is then output as a governance action that satisfies the safety constraints.

8. The intelligent optimization system for river management strategies based on reinforcement learning according to claim 1, characterized in that, The strategy iteration update module includes: At time step t, obtain the current state vector, the correction and governance action output by the multi-stage reachability safety determiner, and the immediate reward after correction. After executing the correction and governance action, obtain the state vector for the next time step. In the reinforcement learning strategy based on the Actor-Critic structure, the current state vector and the next time step state vector are input into the value network to obtain the value evaluation results of the current state and the value evaluation results of the next state. The time difference error is calculated based on the corrected immediate reward, the value assessment result of the current state, and the value assessment result of the next state. The time difference error is obtained by subtracting the value assessment result of the current state from the weighted sum of the corrected immediate reward and the value assessment result of the next state. The parameters of the value network are updated based on the time difference error, and the selection probability of the candidate governance action output by the policy network is adjusted based on the time difference error, so that the selection probability of the corresponding governance action increases when the time difference error is positive and decreases when the time difference error is negative. After updating the parameters of the value network and policy network, new reinforcement learning policy parameters are formed, and the river management strategy is continuously iteratively optimized.

Images