Graph neural network and reinforcement learning-based cascade reservoir optimal scheduling system and method

Abstract

The invention discloses a cascade reservoir optimal scheduling system and method based on a graph neural network and reinforcement learning, and overcomes the limitation of a traditional algorithm by constructing a cascade reservoir optimal scheduling model based on the graph neural network and reinforcement learning. Specifically, a graph neural network architecture is adopted, a message passing neural network MPNN is utilized to extract a local relationship between reservoirs, and a global relationship is captured through a self-attention mechanism of a Transform encoder, so that a complex space-time dependency relationship between cascade reservoirs is comprehensively modeled; in addition, in combination with a PPO reinforcement learning model, a scheduling strategy can be autonomously learned in a dynamic environment, and a globally optimal solution is gradually approached. Compared with a traditional method, the method has remarkable advantages in calculation efficiency, global optimality and capability of adapting to complex scheduling scenes, and an innovative solution is provided for solving the optimal scheduling problem of the cascade reservoirs.

Description

Technical Field

[0001] The present invention belongs to the field of water resource optimization and scheduling, and in particular relates to a cascade reservoir optimization and scheduling system and method based on graph neural network and reinforcement learning. Background Art

[0002] Cascade reservoir operation is of great significance for water resource utilization, energy development, and ecological protection. Appropriate operation strategies can optimize water resource allocation, improve energy efficiency, mitigate the impact of floods, and safeguard downstream water supply and the ecological environment. Especially in the context of increasing electricity demand, the optimal operation of cascade reservoirs not only maximizes the benefits of hydropower generation but also plays a key role in the stable operation of the power grid. However, due to the complexity and multi-objective nature of cascade reservoir operation, developing an efficient operation plan remains a challenging task.

[0003] Traditional methods for optimizing the operation of cascade reservoirs primarily rely on dynamic programming algorithms. However, these algorithms face the "curse of dimensionality" problem, which is characterized by exponentially increasing computational complexity in practical applications. As the number of reservoirs and the number of operation stages increases, computational time and storage requirements explode, making the algorithms unsuitable for large-scale cascade reservoir operation. While improved algorithms based on dynamic programming, such as decomposition coordination methods and genetic algorithms, can reduce computational complexity to a certain extent, these methods often rely on simplified assumptions about the problem, are prone to local optimal solutions, and fail to fully account for interrelationships between reservoirs and complex nonlinear constraints, limiting the global optimality of the operation scheme.

[0004] Therefore, based on the above-mentioned widespread technical problems, it is necessary to propose a cascade reservoir optimization scheduling system and method based on graph neural network and reinforcement learning to solve the above problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a cascade reservoir optimization scheduling system and method based on graph neural network and reinforcement learning. By constructing a cascade reservoir optimization scheduling model based on graph neural network and reinforcement learning, the limitations of traditional algorithms are overcome. Compared with traditional methods, it has significant advantages in computational efficiency, global optimality and the ability to adapt to complex scheduling scenarios, providing an innovative solution to the problem of cascade reservoir optimization scheduling.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning, including: The cascade reservoir graph data modeling module is used to construct a directed graph model with reservoirs as nodes and water flow relationships as edges. Node features include water level, flow, power generation, water abandonment, and storage capacity, while edge features include flow, flow velocity, and water level difference. The graph neural network analysis module uses a message passing neural network (MPNN) to aggregate neighborhood node information and update node features through multiple rounds of message passing. The global dependency modeling module captures the global spatiotemporal dependencies between nodes based on the self-attention mechanism of the Transformer encoder; The reinforcement learning optimization module uses a proximal strategy to optimize the PPO algorithm to train the scheduling strategy. The input is the reservoir status, including water level, inflow, and time period. The output is the final water level action of each reservoir. The objective function is to maximize the comprehensive benefits of power generation, minimize the amount of water abandoned, and optimize the remaining storage capacity. Real-time scheduling decision module generates scheduling instructions based on trained strategies to control reservoir water level and discharge flow Preferably, the node feature vector in the cascade reservoir graph data model is: ; The edge eigenvector is: ; Where, Reservoir node The eigenvector of Reservoir The water level characteristics, Reservoir The flow characteristics of Reservoir The power generation characteristics, Reservoir The characteristics of the water discarded Reservoir The storage capacity characteristics, Represents a connected reservoir node and The eigenvectors of the edges, Indicates that from the reservoir Flow to reservoir The flow characteristics of Reservoir To the reservoir The flow velocity characteristics of Reservoir and reservoirs The water level difference characteristics between them.

[0007] Preferably, the node update rule of the MPNN is: ; Where, and are the learned weight and bias parameters, represents the concatenation operation of feature vectors, is the activation function; Representation node In the Update characteristics after round message passing, Representation node In the Information aggregated from neighboring nodes during rounds of message passing.

[0008] Preferably, the node update rule of the MPNN is: ; Where, is the node feature output by MPNN, and N is the total number of reservoirs; Representation node The feature representation of contains global dependencies.

[0009] Preferably, the objective function of the PPO algorithm is: ; in, For the comprehensive benefits of cascade reservoirs, for Stage reservoir of power generation, for Stage reservoir The amount of discarded water, for Stage reservoir The remaining storage capacity, weight 、 、 Dynamically adjust according to scheduling needs.

[0010] Preferably, the environment state of the reinforcement learning optimization module Including the initial water level of each reservoir, the inflow flow of the leading reservoir and the number of time periods, action The final water level of each reservoir .

[0011] Preferably, the above-mentioned method for optimizing the scheduling system of cascade reservoirs based on graph neural networks and reinforcement learning comprises the following steps: S1, build the cascade reservoir graph data model and define node and edge features; S2, uses MPNN to extract local node relationships and combines it with the Transformer encoder to capture global dependencies; S3, trains a reinforcement learning model based on the PPO algorithm and generates a scheduling strategy through an interactive environment; S4, applies the training strategy in real time, outputs the reservoir final water level action and verifies the constraints.

[0012] Preferably, the feedback value of the PPO algorithm in step S3 is Calculated as: ; When a constraint is violated Set as , and terminate the current scheduling process.

[0013] Preferably, the neural network of the scheduling strategy includes two fully connected layers and a ReLU activation function, the input is the environmental state, and the output is the final water level action of the reservoir.

[0014] Preferably, the method supports dynamic adjustment of weights 、 、 , to adapt to the multi-objective priorities in different scheduling scenarios.

[0015] The beneficial effects of the present invention are as follows: This method overcomes the limitations of traditional algorithms by constructing a cascade reservoir optimization scheduling model based on graph neural networks and reinforcement learning. Specifically, it employs a graph neural network architecture, utilizes a message passing neural network (MPNN) to extract local relationships between reservoirs, and captures global relationships through the self-attention mechanism of the Transformer encoder, thereby comprehensively modeling the complex spatiotemporal dependencies between cascade reservoirs. Furthermore, combined with the PPO reinforcement learning model, it can autonomously learn scheduling strategies in a dynamic environment and gradually approach the global optimal solution. Compared with traditional methods, this method offers significant advantages in computational efficiency, global optimality, and adaptability to complex scheduling scenarios, providing an innovative solution to the problem of cascade reservoir optimization scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of a flow chart of the present invention; Figure 2 Schematic diagram of the strategy neural network construction method of the Agent in an embodiment of the present invention; Figure 3 This is a flowchart for modeling and solving the optimal scheduling of cascade reservoirs based on proximal strategy optimization in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Example 1: like Figure 1 As shown in the figure, a cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning includes: The cascade reservoir graph data modeling module is used to construct a directed graph model with reservoirs as nodes and water flow relationships as edges. Node features include water level, flow, power generation, water abandonment, and storage capacity, while edge features include flow, flow velocity, and water level difference. The graph neural network analysis module uses a message passing neural network (MPNN) to aggregate neighborhood node information and update node features through multiple rounds of message passing. The global dependency modeling module captures the global spatiotemporal dependencies between nodes based on the self-attention mechanism of the Transformer encoder; The reinforcement learning optimization module uses a proximal strategy to optimize the PPO algorithm to train the scheduling strategy. The input is the reservoir status, including water level, inflow, and time period. The output is the final water level action of each reservoir. The objective function is to maximize the comprehensive benefits of power generation, minimize the amount of water abandoned, and optimize the remaining storage capacity. Real-time scheduling decision module generates scheduling instructions based on trained strategies to control reservoir water level and discharge flow Preferably, the node feature vector in the cascade reservoir graph data model is: ; The edge eigenvector is: ; Where, Reservoir node The eigenvector of Reservoir The water level characteristics, Reservoir The flow characteristics of Reservoir The power generation characteristics, Reservoir The characteristics of the water discarded Reservoir The storage capacity characteristics, Represents a connected reservoir node and The eigenvectors of the edges, Indicates that from the reservoir Flow to reservoir The flow characteristics of Reservoir To the reservoir The flow velocity characteristics of Reservoir and reservoirs The water level difference characteristics between them.

[0018] Preferably, the node update rule of the MPNN is: ; Where, and are the learned weight and bias parameters, represents the concatenation operation of feature vectors, is the activation function; Representation node In the Update characteristics after round message passing, Representation node In the Information aggregated from neighboring nodes during rounds of message passing.

[0019] Preferably, the node update rule of the MPNN is: ; Where, is the node feature output by MPNN, and N is the total number of reservoirs; Representation node The feature representation of contains global dependencies.

[0020] Preferably, the objective function of the PPO algorithm is: ; in, For the comprehensive benefits of cascade reservoirs, for Stage reservoir of power generation, for Stage reservoir The amount of discarded water, for Stage reservoir The remaining storage capacity, weight 、 、 Dynamically adjust according to scheduling needs.

[0021] Preferably, the environment state of the reinforcement learning optimization module Including the initial water level of each reservoir, the inflow flow of the leading reservoir and the number of time periods, action The final water level of each reservoir .

[0022] Preferably, the above-mentioned method for optimizing the scheduling system of cascade reservoirs based on graph neural networks and reinforcement learning comprises the following steps: S1, build the cascade reservoir graph data model and define node and edge features; S2, uses MPNN to extract local node relationships and combines it with the Transformer encoder to capture global dependencies; S3, trains a reinforcement learning model based on the PPO algorithm and generates a scheduling strategy through an interactive environment; S4, applies the training strategy in real time, outputs the reservoir final water level action and verifies the constraints.

[0023] Preferably, the feedback value of the PPO algorithm in step S3 is Calculated as: ; When a constraint is violated Set as , and terminate the current scheduling process.

[0024] Preferably, the neural network of the scheduling strategy includes two fully connected layers and a ReLU activation function, the input is the environmental state, and the output is the final water level action of the reservoir.

[0025] Preferably, the method supports dynamic adjustment of weights 、 、 , to adapt to the multi-objective priorities in different scheduling scenarios.

[0026] Example 2: In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] The method comprises the following steps: S1, Construction of neural network architecture for cascade reservoir graph in watershed; S2, training the optimal operation strategy of cascade reservoirs based on reinforcement learning model.

[0028] The step S1 of the invention further comprises: S11, Construction of watershed cascade reservoir map data model: The data model of the cascade reservoir diagram in the watershed can be constructed by a directed graph Represents that the node set Represents the reservoirs in the basin, the edge set Indicates the relationship between water flow directions between reservoirs; Each node Representing a reservoir in a watershed, the node's characteristics describe the state and important parameters of the reservoir. These characteristics may include but are not limited to: : Indicates the current water level of the reservoir; : Indicates the inflow or outflow of the reservoir, reflecting the water flow situation of the reservoir; : represents the power generation of the reservoir; : Indicates the amount of water discharged from the reservoir; : Indicates the reserved storage capacity of the reservoir.

[0029] These features can be combined into a vector: ; side Connecting Nodes and nodes , indicating a reservoir To the reservoir The relationship between the transmission flow; each edge can have different characteristics, describing the properties of the flow, including: :Indicates from the reservoir Flow to reservoir water flow; : Indicates the speed of water flow; : represents the water level difference between the two reservoirs; These features can be represented as a vector: ; S12, Graph Neural Network Analysis of Local Information of Graph Model Based on Message Passing Neural Network (MPNN): The core of Graph Neural Network (GNN) is to update the features of nodes through message passing mechanism. The state of can integrate the information of its neighboring nodes. The specific node update rules are: ; in, Representation node At the k-th layer, is a node The set of neighboring nodes, It is an aggregation operation of neighborhood node information. It is the node feature update operation.

[0030] In each round of message passing, nodes From its neighbor nodes The information of each neighbor node is weighted by the edge feature and the information of the neighbor nodes is aggregated. Aggregation comes in the following forms: Sum: ; average: ; Maximum value: ; Once a node receives a message from its neighbors, the next step is to update its feature representation. Feature updates are usually done through a nonlinear function, and the common method is to use a neural network (such as a fully connected layer) to achieve this; nodes The feature update formula at the kth layer is: ; in, and are the learned weight and bias parameters, represents the concatenation operation of feature vectors, is an activation function (such as ReLU) used to introduce nonlinearity.

[0031] After multiple rounds of message passing, the feature vector of each node (i.e., each reservoir) in the graph is The information of the node and its neighboring nodes is fused; these feature vectors will be passed as input to the subsequent processing modules.

[0032] S13, capturing global dependencies of graph models based on Transformer: In this paper, the Transformer encoder is used to further enhance the expressive power of graph neural network (GNN) models, particularly in capturing the global dependencies between nodes in reservoir operation. Unlike traditional GNNs, which update node representations based on local information from neighboring nodes, the Transformer encoder leverages its powerful self-attention mechanism to model global dependencies between nodes in the graph, thereby improving graph representation capabilities.

[0033] The workflow of the Transformer encoder is as follows: ; in, is a node The initial features of (embedding vectors from the output of the graph neural network), is the characteristic of all nodes, is the final node embedding after being processed by the Transformer encoder.

[0034] Through the Transformer's self-attention mechanism, nodes can not only pay attention to their own status, but also adjust their own representation according to the status of other nodes, thereby providing more comprehensive contextual information for subsequent reservoir operation optimization.

[0035] The step S2 of the invention further comprises: S21, Construction of Cascade Reservoir Optimal Operation Model: Based on the actual dispatching needs of engineering projects and integrated dispatchers, the dispatching model selects power generation, water abandonment, and remaining storage capacity as its optimization objectives. Using a weighting approach, a multi-objective joint optimization dispatching model for cascade reservoirs is established, maximizing the overall benefits of the cascade reservoirs. Weights are used in the calculation to adjust the focus on different dispatching objectives, eliminating the need to normalize the indicators.

[0036] The objective function is: ; in, For the comprehensive benefits of cascade reservoirs, for Stage reservoir The default unit of power generation is 100 million kWh. is the weight value of power generation, for Stage reservoir The default unit of abandoned water is 100 million m³. is the weight value of the abandoned water volume, for Stage reservoir The default unit of remaining storage capacity is 100 million m³. is the weight value of the remaining storage capacity index. is the number of scheduling periods, is the number of reservoirs. If the indicator dimensions are different, they must be converted to the default unit before calculation.

[0037] Boundary and constraints: Water balance: ; in, and for Stage reservoir The initial and final storage capacity, for Stage reservoir If the inflow If it is 1, it is the leading reservoir of the cascade. Obey probability distribution ,like If not 1, is the outflow from the upstream reservoir, for Stage reservoir Outbound traffic, is the length of the period.

[0038] Water level constraint: ; in, and They are Stage reservoir Minimum and maximum water level limits, Reservoir The maximum water level fluctuation allowed within a time period.

[0039] Traffic constraints: ; Where, For reservoirs exist The minimum downstream flow rate during the period, For reservoirs exist The maximum downstream flow rate during the period.

[0040] Output constraints: ; in, For reservoirs exist Minimum output constraint for each period, For reservoirs exist The period water head, the reservoir water head expected output curve can be obtained exist Expected output during the period .

[0041] Boundary Constraints: ; in, For reservoirs The starting water level, For reservoirs Controlled water level at the end of the dispatch period.

[0042] The solution to the cascade reservoir optimization operation model is to obtain the operation process or operation rules of each cascade reservoir under the above-mentioned boundary and constraint conditions, so as to maximize the overall comprehensive benefits of the cascade reservoirs.

[0043] S22, Modeling and solving of cascade reservoir optimal operation based on proximal strategy optimization algorithm: First, the scheduling environment The input of the stage is the agent's action: ; in, is the number of reservoirs, for Stage reservoir The output of the scheduling environment is divided into three parts. The first part is the next stage, i.e. The state of the stage environment (State): ; in, is the number of stages, for The inflow of the stage-by-stage cascade tap reservoirs, Obtained using sampling method.

[0044] The second part of the scheduling environment output is the feedback value (Reward) of using the Action environment, that is, By checking the water level and storage capacity curve of each reservoir in this stage and the water level at the end of the stage given by Action, the initial and final storage capacity of the period can be obtained. and ; According to the water balance formula: ; The outbound flow rate during this period can be calculated , and the outbound flow must meet the outbound flow constraints.

[0045] Water level downstream of the reservoir By outbound traffic Query the outflow tail water level curve to get the water head of each reservoir period. ; in, For reservoirs The head loss of each reservoir is

[0046] In the formula is the output coefficient of the power station, For power generation flow, the output of each reservoir during each period should meet the constraints.

[0047] reservoir exist The power generation during the period is: ; The amount of discarded water is: ; The remaining storage capacity is: ; in, For reservoirs The total storage capacity of the system is used to calculate the feedback value of this stage based on the power generation, water abandonment and remaining storage capacity: ; The third part of the scheduling environment output is the calculation stop sign When a constraint violation occurs in the scheduling calculation at this stage, Assigned to , indicating that the scheduling process violates the constraint and no longer continues to the next stage of calculation, the scheduling environment is reset, and the feedback value of this stage Assigned to When this stage is the last stage of the scheduling period, that is, , then Assigned to Indicates that the entire scheduling is completed. If there is no violation of constraints in the scheduling process at this stage and it is not the last scheduling period of the scheduling period, then Assigned to , continue with the next stage of scheduling calculation.

[0048] The process of building the cascade reservoir optimization operation model environment is as follows: Figure 1 shown.

[0049] The strategy neural network construction method of the Agent in the present invention is as follows Figure 2 As shown in the figure, the input layer nodes of the policy neural network are the initial water level of each cascade reservoir during the scheduling period, the inflow of the leading reservoir during the scheduling period, and the number of periods in the entire scheduling period. The nodes of the input layer correspond to the states in the scheduling environment. The middle layer of the network has no special structural restrictions. Based on the complexity of the model and the commonly used network structure of the PPO algorithm, the study selected two fully connected layers plus the ReLU activation function as the middle layer of the network. The output layer of the network is the final water level of each cascade reservoir during the scheduling period. The nodes of the output layer correspond to the actions of the agent in the scheduling environment.

[0050] After completing the reinforcement learning cascade reservoir operation model environment and algorithm construction, the model environment interacts with the agent to obtain data samples and update the agent's strategy until the target requirements are met or convergence is achieved to solve the cascade reservoir optimization operation model. The cascade reservoir optimization operation modeling and solution process based on proximal strategy optimization is as follows: Figure 3 shown.

Claims

1. A cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning, characterized by: include: Cascade reservoir graph data modeling module, used to construct a directed graph model with reservoirs as nodes and water flow relationships as edges; Node features include water level, flow, power generation, water abandonment and reservoir capacity, while edge features include flow, flow velocity and water level difference; The graph neural network analysis module uses a message passing neural network (MPNN) to aggregate neighborhood node information and update node features through multiple rounds of message passing. The global dependency modeling module captures the global spatiotemporal dependencies between nodes based on the self-attention mechanism of the Transformer encoder; The reinforcement learning optimization module uses a proximal strategy to optimize the PPO algorithm to train the scheduling strategy. The input is the reservoir status, including water level, inflow, and time period. The output is the final water level action of each reservoir. The objective function is to maximize the comprehensive benefits of power generation, minimize the amount of water abandoned, and optimize the remaining storage capacity. The real-time scheduling decision module generates scheduling instructions based on the trained strategy to control the reservoir water level and discharge flow.

2. The cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The node feature vector in the cascade reservoir graph data model is: ； Edge eigenvector for: ； Where, Reservoir node The eigenvector of Reservoir The water level characteristics, Reservoir The flow characteristics of Reservoir The power generation characteristics, Reservoir The characteristics of the water discarded Reservoir The storage capacity characteristics, Represents a connected reservoir node and The eigenvectors of the edges, Indicates that from the reservoir Flow to reservoir The flow characteristics of Reservoir To the reservoir The flow velocity characteristics of Reservoir and reservoirs The water level difference characteristics between them.

3. The cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The node update rule of the MPNN is: ； Where, and are the learned weight and bias parameters, represents the concatenation operation of feature vectors, is the activation function; Representation node In the Update characteristics after round message passing, Representation node In the Information aggregated from neighboring nodes during rounds of message passing.

4. The cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The node update rule of the MPNN is: ； Where, is the node feature output by MPNN, N is the total number of reservoirs, Representation node The feature representation of contains global dependencies.

5. The cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The objective function of the PPO algorithm is: ； in, For the comprehensive benefits of cascade reservoirs, for Stage reservoir of power generation, for Stage reservoir The amount of discarded water, for Stage reservoir The remaining storage capacity, weight 、 、 Dynamically adjust according to scheduling needs.

6. The cascade reservoir optimization scheduling system based on graph neural network and reinforcement learning according to claim 1 is characterized in that: The environment state of the reinforcement learning optimization module Including the initial water level of each reservoir, the inflow flow of the leading reservoir and the number of time periods, action The final water level of each reservoir .

7. A method for optimizing the scheduling system of cascade reservoirs based on graph neural networks and reinforcement learning according to any one of claims 1 to 6, characterized in that: The following steps are involved: S1, build the cascade reservoir graph data model and define node and edge features; S2, uses MPNN to extract local node relationships and combines it with the Transformer encoder to capture global dependencies; S3, trains a reinforcement learning model based on the PPO algorithm and generates a scheduling strategy through an interactive environment; S4, applies the training strategy in real time, outputs the reservoir final water level action and verifies the constraints.

8. The method for optimizing the scheduling system of cascade reservoirs based on graph neural networks and reinforcement learning according to claim 7, characterized in that: The feedback value of the PPO algorithm in step S3 Calculated as: ； When a constraint is violated Set as , and terminate the current scheduling process.

9. The method for optimizing the scheduling system of cascade reservoirs based on graph neural networks and reinforcement learning according to claim 7, characterized in that: The neural network of the scheduling strategy includes two fully connected layers and a ReLU activation function. The input is the environmental state and the output is the final water level action of the reservoir.

10. The method for optimizing the scheduling system of cascade reservoirs based on graph neural network and reinforcement learning according to claim 7, characterized in that: The method supports dynamic adjustment of weights 、 、 , to adapt to the multi-objective priorities in different scheduling scenarios.

Images