A multi-agent reinforcement learning driven stochastic intelligent dispatching method for cascade hydropower

Abstract

The present application belongs to the field of water and electricity dispatching operation, and relates to a cascade hydropower random intelligent dispatching method driven by multi-agent reinforcement learning. The steps comprise: step one: generating multiple groups of runoff random scenes by using Monte Carlo forward simulation; step two: constructing a two-stage random optimization model with the maximization of power generation benefit as the target based on the scene method; step three: modeling the single-agent reinforcement learning in the spatial dimension of the cascade hydropower; step four: solving the two time period sub-problems decomposed in the time dimension by the single-agent of the cascade hydropower; step five: proposing a multi-agent reinforcement learning step-by-step optimization algorithm MARL-POA, and deploying the reinforcement learning agents with different strategies to cooperatively optimize the dispatching; step six: obtaining the rolling time domain cascade hydropower dispatching scheme as the available information is updated. The present application performs the time-space dimension reduction of the cascade hydropower optimization, effectively reduces the calculation scale, and can quickly and dynamically formulate a reliable cascade hydropower operation plan under the runoff forecast uncertainty.

Description

Technical Field

[0001] This invention belongs to the field of hydropower scheduling, specifically involving a multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower. Background Technology

[0002] As a core clean energy source, hydropower plays a crucial role in enhancing the stability of the power system and mitigating fluctuations in renewable energy sources. However, in actual engineering scheduling, cascade hydropower station systems face complex nonlinear spatiotemporal hydraulic-electric coupling relationships and the randomness of natural runoff, posing severe challenges to scheduling plan formulation. Compared to short-term scheduling, long-term scheduling of cascade hydropower faces more significant runoff randomness and volatility. The accuracy of long-term runoff forecasts decreases rapidly with the extension of the forecast period. If runoff randomness is ignored and scheduling schemes are formulated solely based on deterministic optimization models, the reliability of the operational strategy will be severely reduced. Especially when forecast deviations are large or extreme hydrological events occur, the applicability of deterministic scheduling schemes is clearly insufficient. Furthermore, the high-dimensional stochastic modeling complexity of long-term scheduling of cascade systems further exacerbates the difficulty of solving the problem.

[0003] To mitigate the impact of runoff uncertainty on the long-term scheduling benefits of cascade hydropower, the main methods include scenario-based approaches, chance-constrained programming, and robust optimization. Scenario-based approaches solve for the optimal scheduling decision based on a set of representative scenarios; chance-constrained programming relaxes hard constraints with probability confidence levels; and robust optimization optimizes the worst-case scenario while ensuring the feasibility of stochastic scenarios. While considering uncertainty, the operational plan developed has higher reliability and robustness, but it also suffers from problems such as increased computational dimensionality, sensitivity to subjective parameters, and overly conservative solutions.

[0004] Cascade hydropower scheduling is a typical high-dimensional, non-convex, nonlinear, multi-stage optimization problem. Its solution process presents a trade-off between accuracy, efficiency, and stability. Existing methods can be divided into two categories. The first category includes traditional algorithms such as linear programming, nonlinear programming, dynamic programming, and their improved algorithms. Linear programming is computationally efficient but oversimplifies the nonlinear characteristics of hydropower, leading to problem distortion; nonlinear programming is prone to local optima and has a high risk of failure; dynamic programming, while guaranteeing global optimality, suffers from the "curse of dimensionality." The second category includes heuristic intelligent algorithms such as particle swarm optimization, ant colony optimization, and genetic algorithms. While these can circumvent non-convex constraints, they suffer from defects such as solution oscillation, poor interpretability, and parameter sensitivity, limiting their engineering applicability. Reinforcement learning, by exploring and updating the state-action value function, can avoid the "curse of dimensionality" problem caused by state combination traversal; however, it is limited by the convergence difficulties caused by the excessively large state and action spaces in multi-stage cascade hydropower and centralized decision-making methods. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention aims to provide a multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects. Specifically, it offers a method for rapidly and dynamically optimizing large-scale spatiotemporally coupled high-dimensional nonlinear problems in cascade hydropower projects, addressing the high uncertainty of long-term runoff forecasting. This provides a reliable scheduling plan for the engineering operation of cascade hydropower projects. The method is validated using a large-scale cascade hydropower project in the Lancang River basin as an engineering example. Results show that the method significantly reduces computation time while maintaining the accuracy of optimization results, and the proposed scheduling scheme exhibits better reliability and robustness. This invention aligns with practical hydropower engineering scheduling principles and can be rapidly deployed and promoted, providing important technical references for the operation and scheduling of other cascade hydropower systems.

[0006] The technical solution adopted in this invention is as follows:

[0007] A multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects, comprising the following steps:

[0008] Step 1: Establish a seasonal difference autoregressive moving average model SARIMAX, use Monte Carlo forward simulation to generate multiple sets of runoff random scenarios, and divide them into several representative scenarios through k-means clustering;

[0009] Step 2: Construct a two-stage stochastic optimization model based on the scenario method, with the goal of maximizing power generation efficiency;

[0010] Step 3: Perform single-agent reinforcement learning modeling for the spatial dimension of cascade hydropower;

[0011] Step 4: Based on the single-agent reinforcement learning modeling in Step 3, and combined with the Q-learning algorithm, the single agent of the cascade hydropower project is used to solve the two time-division sub-problems decomposed in the time dimension;

[0012] Step 5: Propose the MARL-POA (Multi-Agent Reinforcement Learning Progressive Optimization Algorithm) for cascade hydropower, and deploy reinforcement learning agents with different strategies to collaboratively optimize scheduling;

[0013] Step Six: As available information is updated, obtain the rolling time-domain cascade hydropower scheduling scheme.

[0014] Preferably, the seasonal differential autoregressive moving average model SARIMAX in step one is a statistical method applied to the prediction of non-stationary time series. This model can simultaneously process the seasonal fluctuations, trend characteristics and external influencing factors of the time series. The model establishment includes inputting historical runoff data in the cascade hydroelectric domain, differential processing, selecting model parameters, residual analysis and verification.

[0015] The Monte Carlo forward simulation generates multiple sets of random runoff scenarios by using the SARIMAX model to generate multiple future runoff sequences. The error terms in the simulation follow the normal distribution of the model residuals to reflect the prediction uncertainty. To avoid the accumulation of errors and the generation of negative runoff values, the sequences need to be logarithmically transformed before prediction and then restored to their physical meaning through exponential transformation after prediction.

[0016] The k-means clustering algorithm is an unsupervised data partitioning learning method. Using this algorithm, The Monte Carlo simulation of runoff stochastic scenarios is reduced to A number of representative scenarios are identified, and the probability of occurrence for each representative scenario is recorded.

[0017] Preferably, the scenario method in step two uses a set of probabilistic runoff prediction scenarios to determine the actual stochastic runoff;

[0018] The power generation efficiency target is the sum of the power generation in the first month of the dispatch period and the expected power generation in all representative scenarios for the remaining months;

[0019] In the two-stage stochastic optimization, the first stage is the first month after the start of the runoff forecast prediction period, and the second stage is all the remaining months of the prediction period except for the first month.

[0020] Preferably, the single-agent reinforcement learning modeling of the cascade hydropower spatial dimension in step three involves modeling the optimization problem as a Markov decision process in the cascade spatial dimension, and determining the state space, action space, reward, and policy function of the reinforcement learning model.

[0021] Preferably, the Q-learning algorithm in step four is a model-free reinforcement learning algorithm based on value iteration;

[0022] The two-time sub-problems are optimization sub-problems decomposed from the long-term scheduling of cascade hydropower in the time dimension.

[0023] Preferably, the rolling time-domain cascade hydropower scheduling scheme in step six involves the time-domain optimization window advancing over time. Each advancement integrates the latest flow observation data and the previous round of scheduling scheme, repeating the prediction and optimization process to dynamically adjust the scheduling scheme.

[0024] This invention offers the following advantages: The method transforms the long-term scheduling of cascade hydropower systems into a series of sub-problems using a stepwise optimization approach, achieving temporal dimensionality reduction. Multiple reinforcement learning agents explore the optimal scheduling actions for each sub-problem. Since the agents prioritize actions more likely to reach the optimal goal, the massive computational burden caused by the original discrete-state permutation and combination traversal is avoided. Spatial dimensionality reduction is also achieved. Therefore, dimensionality reduction at both the temporal and spatial levels effectively reduces the computational scale and the optimization computation time for cascade hydropower. Facing the high uncertainty of long-term runoff, the scenario-based approach and rolling time-domain dynamic adjustment ensure that the obtained cascade hydropower scheduling scheme has better reliability and robustness against random and complex changes in runoff. This method has significant guiding significance for the practical engineering scheduling of cascade hydropower. Attached Figure Description

[0025] Figure 1 A flowchart of a multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower;

[0026] Figure 2 To enhance the interaction process between the learning agent and the cascade hydropower system;

[0027] Figure 3 Flowchart of the computation of MARL-POA, a stepwise optimization algorithm for multi-agent reinforcement learning;

[0028] Figure 4 A comparison chart showing the computational efficiency and accuracy of the three algorithms MARL-POA, POA, and POA-DPSA under different discretization methods;

[0029] Figure 5 The diagram shows the rolling update process of the scheduling scheme for Xiaowan Hydropower Station after considering the uncertainty of runoff forecast. In the diagram, (a) represents the first rolling optimization, (b) represents the second rolling optimization, (c) represents the third rolling optimization, (d) represents the fourth rolling optimization, (e) represents the fifth rolling optimization, (f) represents the sixth rolling optimization, (g) represents the seventh rolling optimization, (h) represents the eighth rolling optimization, (i) represents the ninth rolling optimization, (j) represents the tenth rolling optimization, and (k) represents the eleventh rolling optimization.

[0030] Figure 6The diagram shows the rolling update process of the scheduling scheme for the Nuozhadu Hydropower Station after considering the uncertainty of runoff forecast. In the diagram, (l) represents the first rolling optimization, (m) represents the second rolling optimization, (n) represents the third rolling optimization, (o) represents the fourth rolling optimization, (p) represents the fifth rolling optimization, (q) represents the sixth rolling optimization, (i) represents the seventh rolling optimization, (s) represents the eighth rolling optimization, (t) represents the ninth rolling optimization, (u) represents the tenth rolling optimization, and (v) represents the eleventh rolling optimization.

[0031] Figure 7 A comparison chart of actual power generation under scheduling schemes that take into account runoff uncertainty and deterministic scheduling schemes. Detailed Implementation

[0032] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0033] This invention proposes a multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects, such as... Figure 1 As shown, it includes the following steps:

[0034] Step 1: Generate multiple sets of stochastic runoff scenarios using Monte Carlo forward simulation:

[0035] A seasonal difference autoregressive moving average model SARIMAX was established. Monte Carlo forward simulation was used to generate multiple sets of runoff random scenarios, which were then divided into several representative scenarios by k-means clustering.

[0036] The seasonal difference autoregressive moving average (SARIMAX) model is a statistical method for forecasting non-stationary time series. This model can simultaneously handle the seasonal fluctuations, trend characteristics, and external influencing factors of the time series. The model establishment includes inputting historical runoff data within the cascade hydroelectric domain, difference processing, selecting model parameters, residual analysis, and validation.

[0037] Monte Carlo forward simulation generates multiple sets of stochastic runoff scenarios by using the SARIMAX model to generate multiple future runoff sequences. The error terms in the simulation follow the normal distribution of the model residuals to reflect the prediction uncertainty. To avoid the accumulation of errors and the generation of negative runoff values, the sequences need to be logarithmically transformed before prediction and then restored to their physical meaning through exponential transformation after prediction.

[0038] k-means clustering is an unsupervised data partitioning learning method. This algorithm is used to... The Monte Carlo simulation of runoff scenarios was reduced to A number of representative scenarios are identified, and the probability of occurrence for each representative scenario is recorded.

[0039] Step 2: Construct a two-stage stochastic optimization model based on the scenario method, aiming to maximize power generation efficiency.

[0040] The scenario method uses a set of probabilistic runoff prediction scenarios to determine the actual stochastic runoff; the power generation benefit target is the sum of the power generation in the first month of the scheduling period and the expected power generation in all representative scenarios in the remaining months; in the two-stage stochastic optimization, the first stage is the first month after the start of the runoff forecast prediction period, and the second stage is all the remaining months in the prediction period except for the first month.

[0041] Step 3: Perform single-agent reinforcement learning modeling of cascade hydropower spatial dimension:

[0042] Single-agent reinforcement learning modeling in the spatial dimension of cascade hydropower involves modeling the optimization problem as a Markov decision process in the cascade spatial dimension, and determining the state space, action space, reward, and policy function of the reinforcement learning model.

[0043] Step 4: Based on the single-agent reinforcement learning modeling in Step 3, and combined with the Q-learning algorithm, the single-agent model of the cascade hydropower project is used to solve the two time-division sub-problems decomposed in the time dimension:

[0044] Q-learning algorithm is a model-free reinforcement learning algorithm based on value iteration; the two-time sub-problem is an optimization sub-problem decomposed into the time dimension of long-term scheduling of cascade hydropower.

[0045] Step 5: Propose the MARL-POA (Multi-Agent Reinforcement Learning-Based Stepwise Optimization) algorithm for cascade hydropower, and deploy reinforcement learning agents with different strategies for collaborative optimization scheduling.

[0046] Step Six: As available information is updated, obtain the rolling time-domain cascade hydropower scheduling scheme:

[0047] The rolling time-domain cascade hydropower dispatching scheme advances the time-domain optimization window over time. Each advancement integrates the latest flow observation data and the previous round of dispatching schemes, repeating the prediction and optimization process to dynamically adjust the dispatching scheme.

[0048] In step one, the specific steps for generating multiple sets of stochastic runoff scenarios using Monte Carlo forward simulation are as follows:

[0049] (1.1) The runoff series is differentially processed to satisfy the stationarity assumption;

[0050] (1.2) Select model parameters, including the autoregressive order, by minimizing the Bayesian Information Criterion (BIC) through grid search. Moving average order Seasonal autoregression order and seasonal moving average order ;

[0051] (1.3) Residual analysis and verification: ensure that the residuals follow a normal distribution and are free from autocorrelation, and confirm that the fitted model can fully capture the characteristics of runoff time series through Ljung-Box test;

[0052] (1.4) Monte Carlo simulation prediction: Multiple future runoff sequences are generated using the SARIMAX model. The error terms in the simulation follow the normal distribution of the model residuals to reflect the prediction uncertainty. To avoid negative runoff values ​​caused by error superposition, the sequences need to be logarithmically transformed before prediction and then restored to their physical meaning through exponential transformation after prediction.

[0053] (1.5) Using k-means clustering to... The Monte Carlo simulation runoff scenario was reduced to A number of representative scenarios were identified, and the probability of occurrence for each scenario was recorded.

[0054] In step two: the specific steps for constructing the optimization model based on the scenario method are as follows:

[0055] Objective function:

[0056]

[0057] In the formula: The total power generation of the cascade hydropower station system during the dispatching cycle is expressed in kWh. This refers to the serial number of the hydropower station. This refers to the number of hydropower stations in a cascade hydropower system. The time period number. This represents the total number of scheduling periods; For the first The power output coefficient of the hydroelectric power station; For the first Runoff Scenario Hydropower station Power generation flow rate per time period, per unit ; For the first Runoff Scenario Hydropower station Average net head over a period of time, in meters; For each time period, the unit is... ; This indicates the number of runoff prediction scenarios; Representing the The probability of occurrence of a runoff scenario. ;

[0058] The constraints include:

[0059] Initial and final water level constraints:

[0060]

[0061] In the formula: and These are the initial and final water levels, respectively. and The first The initial water level and expected final water level of the reservoir, in meters (m).

[0062] Reservoir water level constraints:

[0063]

[0064] In the formula: and The first reservoir The lower and upper limits of the water level at the end of the time period For the first The first reservoir Water level at the end of the time period, in meters;

[0065] Output constraints:

[0066]

[0067] In the formula: and The first reservoir The lower and upper limits of output for each time period, in kW; For the first Hydropower station Power generation flow rate per time period, per unit ; For the first Hydropower station Average net head over a period of time, in meters;

[0068] Water balance constraints:

[0069]

[0070] In the formula: For the first reservoir Storage capacity per time period, per unit ; The first reservoir Total inflow, outflow, interval flow, and wastewater discharge for each time period, per unit ;

[0071] Power generation flow constraints:

[0072]

[0073] In the formula: The first reservoir Lower and upper limits of power generation flow for a given period, in units ; For the first The first reservoir Power generation flow at the end of the period, per unit ;

[0074] Outbound flow constraints:

[0075]

[0076] In the formula: The first reservoir Lower and upper limits of outbound flow for a given period, in units ; For the first The first reservoir Outbound flow at the end of the period, unit ;

[0077] Reservoir water level-capacity relationship function constraints:

[0078]

[0079] In the formula: For the first reservoir Water level at the end of the time period, in meters; This is a function relating water level and reservoir capacity.

[0080] Downstream water level - outflow rate relationship function constraints:

[0081]

[0082] In the formula: For the first reservoir The average downstream water level during the specified period, in meters (m). This is a function relating downstream water level to outflow rate.

[0083] In Step 3: The specific steps for single-agent reinforcement learning modeling of cascade hydropower spatial dimension are as follows:

[0084] (3.1) Establishing the state space

[0085]

[0086] In the formula: For the first The first hydroelectric power station The water level at the beginning of the period, For the first The first hydroelectric power station Water level at the end of the time period, in meters; For the first Traffic flow during specific time periods For the first The upstream reservoir of the hydroelectric power station Outbound flow rate per time period, per unit .

[0087] (3.2) Establishing the action space

[0088]

[0089] In the formula: For the intelligent agent for the first reservoir The selected outbound flow rate for the specified time period, in units of ;

[0090] To reduce the action space, the outflow constraint is coupled with the water level constraint in advance. That is, the upper and lower limits of the outflow under the water level constraint are calculated and their intersection is taken with the outflow constraint. The lower and upper limits of the outflow after constraint coupling are shown below:

[0091]

[0092] In the formula: These are the first two constrained couplings. The first hydroelectric power station Lower and upper limits of outbound flow for a given period, in units ;

[0093] Based on the upper and lower limits of the outbound flow obtained by taking the intersection, an adaptive step-size non-uniform discretization is performed. The specific discretization process is as follows:

[0094] Set initial step size Step size growth rate and number of iterations The agent's best action in the previous round As the central point; among which, This represents the optimal outbound flow rate from the previous iteration. This is the starting point for discretizing towards the lower limit of warehouse exit. This is the starting point for discretization towards the upper limit of outbound shipments;

[0095] Left sequence generation, from Discretize the lower limit of outbound flow and perform iterative calculation. ,until If the outbound flow rate is less than the lower limit, the lower limit of the outbound flow rate is added to the sequence; where, The left sequence of discrete outflow volume generated by the iteration, in units ;

[0096] Right sequence generation, from Discretize the upper limit of outbound flow and perform iterative calculation. ,until If the outbound flow exceeds the upper limit, the upper limit of the outbound flow is added to the sequence; where, Discrete outflow rate of the right sequence generated by iteration, unit ;

[0097] The left and right sequences and the center point are merged to obtain a non-uniform discrete total sequence. This method adjusts... and By acquiring discrete sequences with varying precision and increasing the exploration density of potential optimal action neighborhoods, the agent can explore more efficiently, ultimately yielding the action space. Since the initial and final water levels of a hydropower station are usually different in each of the two time periods, the action space of different intelligent agents is also different.

[0098] (3.3) Setting rewards

[0099]

[0100] In the formula: For the first The power generation of the power station in two time periods, It is a sufficiently large constant used for exceeding the limit penalty.

[0101] (3.4) Acquisition Strategy

[0102] A heterogeneous strategy is adopted, with the target strategy being the action-value function and the behavioral strategy being the greedy coefficient method.

[0103] Action value function:

[0104]

[0105] In the formula: For intelligent agents targeting Local observation status of the hydropower station The chosen action; Value discount rate;

[0106] The update formula for the action value function:

[0107]

[0108] In the formula: Indicates the first Hydropower station The maximum cumulative reward that can be obtained under this state. The learning rate;

[0109] Greedy coefficient behavior strategy:

[0110]

[0111] Cosine annealing decay method for the greedy coefficient:

[0112]

[0113] In the formula: These are the agent's maximum and minimum exploration rates, respectively. These represent the current exploration round and the maximum exploration round for the agent, respectively.

[0114] In step four: the specific steps for solving the subproblem using a single agent combined with the Q-learning algorithm are as follows:

[0115] (4.1) Initialize the Q table, all The table is initialized to 0. The number of cascade hydropower stations is ;

[0116] (4.2) Action selection: using a greedy coefficient strategy based on the current action. Status of the hydroelectric power station Select outbound flow ;

[0117] (4.3) Execution of actions , transferred to the The new status of the hydroelectric power station And receive instant power generation rewards. The interaction between the agent and the environment yields a quadruple. Store in the experience replay cache;

[0118] (4.4) Single-step update is adopted, and the storage of water and electricity scheduling experience and the updating of Q table are carried out simultaneously;

[0119] (4.5) Let ,like Proceed to step (4.6); otherwise, let Return to step (4.2);

[0120] (4.6) Repeat steps (4.2) to (4.5) until convergence.

[0121] Step 5: The specific steps of the MARL-POA (Multi-Agent Reinforcement Learning Progressive Optimization Algorithm) for cascade hydropower are as follows:

[0122] (5.1) Set the total number of scheduling periods for cascade hydropower stations Discrete parameters of non-uniform motion space and Algorithm convergence accuracy ,forecast One runoff scenario;

[0123] (5.2) The initial cascade reservoir scheduling water levels and corresponding outflow sequences are obtained using the equal flow method; the total power generation during the initial decision-making process is... ;

[0124] (5.3) Settings ;

[0125] (5.4) Settings ;

[0126] (5.5) Perform the first stage calculation, the... The runoff scenario is fixed except for the first All water level sequences other than those at the end of the time period are obtained from the previous round of optimization. The first runoff scenario Outflow sequence of cascade reservoirs in different time periods , This refers to the serial number of the hydropower station. , This represents the number of hydropower stations in the cascade hydropower system; if this is the first round of optimization, then obtain the initial outflow sequence, and so on. One reinforcement learning agent (one agent for every two time periods) There are a total of time periods (the first intelligent agent) and the current scene and Interaction of time-based cascade hydropower systems, the first The hydroelectric power station is in the flow sequence The agent's action space is obtained by non-uniform discretization with the center point as the center, and the reward is the first... Time period selection action Hydropower station's power generation in two time periods Power generation in two time periods Calculate according to the following formula:

[0127]

[0128] In the formula: For the first reservoir Initial storage capacity and The difference in storage capacity at the end of the time period; For the first Runoff Scenario Hydropower station Power generation flow during a given time period; For the first Runoff Scenario Hydropower station Average net head over the period; This represents the time period for each segment.

[0129] After the agent completes its exploration, update the [number of updates]. The first runoff scenario Water level sequence of cascade hydropower system at the end of the period and the individual runoff scenarios and Outbound flow sequence for different time periods Proceed to step (5.6).

[0130] (5.6) Order Proceed to step (5.7); otherwise, return to step (5.5).

[0131] (5.7) Let Proceed to step (5.8); otherwise, return to step (5.4).

[0132] (5.8) Perform the second stage calculation, fix all water level sequences except for the end of the first time period, and obtain the water level at the end of the second time period for different runoff scenarios and the probability of the corresponding scenario occurring. , Representing the The probability of occurrence of each runoff scenario is used to obtain the outflow sequence of the first time period in the previous optimization. If it is the first round of optimization, then obtain the initial outflow sequence. The first reinforcement learning agent interacts with the environment of the cascade hydropower system in the first and second time periods. The hydroelectric power station is in the flow sequence The agent's action space is obtained by non-uniform discretization using the central point as the center. The reward is the selected action. At that time, the power generation is calculated by weighting the probabilities of different runoff scenarios. Electricity generation Calculate according to the following formula:

[0133]

[0134] In the formula: For the first The initial reservoir capacity of the first period and the first period The difference in reservoir capacity at the end of the second time period for each runoff scenario; The first In the runoff scenario, the first The outflow, average net head, and interval flow of the hydropower station in the second time period; The first The outflow, average net head, and interval flow of the hydropower station in the first time period;

[0135] After the agent completes its exploration, it updates the water level sequence of the cascade hydropower system at the end of the first time period. Update the outflow sequence for the only runoff scenario in the first time period. and the second period Outflow Matrix for Each Runoff Scenario Proceed to step (5.9); where, For the second period cascade outflow sequences for individual runoff scenarios;

[0136] (5.9) Calculation results Total power generation during the period ,like If the result is positive, the algorithm is considered converged, and the output is the water level process curve for each hydropower station and the corresponding time-period power generation flow; otherwise, let... Then return to step (5.3) for the next round of optimization.

[0137] The specific applications of this invention are as follows:

[0138] (1) Project Background

[0139] Taking the Xiaowan and Nuozhadu hydropower stations on the middle and lower reaches of the Lancang River in Yunnan Province as examples, this paper analyzes and verifies the proposed algorithm model. As the only two reservoirs in the Lancang River basin with multi-year regulation capabilities, the scheduling of these two power stations is crucial for the hydropower utilization, flood control, and other aspects of the cascade hydropower system when formulating long-term scheduling plans. The total installed capacity of the Xiaowan and Nuozhadu power stations reaches 10050MW, and the total regulating reservoir capacity is approximately 28 billion cubic meters. According to runoff characteristics, the Lancang River basin experiences the pre-flood season from January to May, the flood season from June to October, and the post-flood season from November to December.

[0140] (2) Comparison of algorithm efficiency and accuracy

[0141] Based on 62 years of historical runoff data from the Xiaowan Nuozhadu cascade hydropower project, 10 typical inflow scenarios were identified. Within the computational framework of this invention, three algorithms—MARL-POA, POA, and POA-DPSA—were used for optimized scheduling. Comparisons of computation time and power generation under different discrete methods for each algorithm were obtained. Figure 4As shown, when the computational accuracy is the same, the MARL-POA algorithm is 50 to 200 times faster than the POA algorithm and 35 times faster than the POA-DPSA algorithm; when the computation time is basically the same, the MARL-POA algorithm has higher computational accuracy than both the POA and POA-DPSA algorithms. This demonstrates the superiority of the proposed MARL-POA algorithm in terms of both computational efficiency and computational accuracy.

[0142] (3) Rolling calculation results considering runoff forecast uncertainty

[0143] The rolling optimization processes for water level regulation at Xiaowan and Nuozhadu are as follows: Figure 5 and Figure 6 As shown, in the initial months, due to the high uncertainty in runoff forecasting, water level scheduling varies significantly under different runoff scenarios. However, the method of this invention considers different future runoff scenarios and their probabilities of occurrence, and calculates the robust scheduling water level for the first month in the time domain on a rolling basis. In the last few months, as the time range of runoff forecasting shortens, the differences between different runoff scenarios decrease, leading to a gradual convergence of scheduling water levels. The scheduling cycle ends after determining the scheduling water levels for the last two periods. After obtaining the water level scheduling process, the actual power generation is calculated by inputting the actual runoff sequence. The actual power generation of the scheduling scheme considering runoff uncertainty and the deterministic scheduling scheme are shown below. Figure 7 As shown, the dispatch scheme obtained by the method of the present invention increases the power generation by 1.428 billion kWh. This is due to the fact that the water inflow of the basin may deviate from the forecast average during the multi-scenario simulation. The water level of Xiaowan Hydropower Station is gradually reduced from April to June before the flood season to reduce the risk of water abandonment during the flood season, which is more in line with the dispatch concept in actual engineering.

Claims

1. A multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects, characterized in that, The specific steps are as follows: Step 1: Establish a seasonal difference autoregressive moving average model SARIMAX, use Monte Carlo forward simulation to generate multiple sets of runoff random scenarios, and divide them into several representative scenarios through k-means clustering; Step 2: Construct a two-stage stochastic optimization model based on the scenario method, with the goal of maximizing power generation efficiency; The scenario method in step two uses a set of probabilistic runoff prediction scenarios to determine the actual stochastic runoff; the power generation benefit target is the sum of the power generation in the first month of the scheduling period and the expected power generation of all representative scenarios in the remaining months; in the two-stage stochastic optimization, the first stage is the first month after the start of the runoff forecast prediction period, and the second stage is all the remaining months in the prediction period except for the first month. Step 3: Perform single-agent reinforcement learning modeling for the spatial dimension of cascade hydropower; The single-agent reinforcement learning modeling of the cascade hydropower spatial dimension in step three involves modeling the optimization problem as a Markov decision process in the cascade spatial dimension, and determining the state space, action space, reward, and policy function of the reinforcement learning model. Step 4: Based on the single-agent reinforcement learning modeling in Step 3, and combined with the Q-learning algorithm, the single agent of the cascade hydropower project is used to solve the two time-division sub-problems decomposed in the time dimension; Step 5: Propose the MARL-POA (Multi-Agent Reinforcement Learning Progressive Optimization Algorithm) for cascade hydropower, and deploy reinforcement learning agents with different strategies to collaboratively optimize scheduling; The specific steps of the MARL-POA (Multi-Agent Reinforcement Learning Progressive Optimization Algorithm) for cascade hydropower in step five are as follows: (5.1) Set the total number of scheduling periods for cascade hydropower stations Discrete parameters of non-uniform motion space and Algorithm convergence accuracy ,forecast One runoff scenario; (5.2) The initial cascade reservoir scheduling water levels and corresponding outflow sequences are obtained by the equal flow method; The total power generation during the initial decision-making process is ; (5.3) Settings ; (5.4) Settings ; (5.5) Perform the first stage calculation, the... The runoff scenario is fixed except for the first All water level sequences other than those at the end of the time period are obtained from the previous round of optimization. The first runoff scenario Outflow sequence of cascade reservoirs in different time periods , This refers to the serial number of the hydropower station. This represents the number of hydropower stations in the cascade hydropower system; if this is the first round of optimization, then obtain the initial outflow sequence, and so on. A reinforcement learning agent and the current scenario. and Interaction of time-based cascade hydropower systems, the first The hydroelectric power station is in the flow sequence The agent's action space is obtained by non-uniform discretization with the center point as the center, and the reward is the first... Time period selection action Hydropower station's power generation in two time periods Power generation in two time periods Calculate according to the following formula: ； In the formula: For the first reservoir Initial storage capacity and The difference in storage capacity at the end of the time period; For the first Runoff Scenario Hydropower station Power generation flow during a given time period; For the first Runoff Scenario Hydropower station Average net head over the period; The duration of each time period; After the agent completes its exploration, update the [number of updates]. The first runoff scenario Water level sequence of cascade hydropower system at the end of the period and the individual runoff scenarios and Outbound flow sequence for different time periods Proceed to step (5.6); (5.6) Order ,like Proceed to step (5.7); otherwise, return to step (5.5). (5.7) Let ,like Proceed to step (5.8); otherwise, return to step (5.4). (5.8) Perform the second stage calculation, fix all water level sequences except for the end of the first time period, and obtain the water level at the end of the second time period for different runoff scenarios and the probability of the corresponding scenario occurring. , Representing the The probability of occurrence of each runoff scenario is used to obtain the outflow sequence of the first time period in the previous optimization. If it is the first round of optimization, then obtain the initial outflow sequence. The first reinforcement learning agent interacts with the environment of the cascade hydropower system in the first and second time periods. The hydroelectric power station is in the flow sequence The agent's action space is obtained by non-uniform discretization using the central point as the fulcrum; the reward is the selected action. At that time, the power generation is calculated by weighting the probabilities of different runoff scenarios. Electricity generation Calculate according to the following formula: ; In the formula: For the first The initial reservoir capacity of the first period and the first period The difference in reservoir capacity at the end of the second time period for each runoff scenario; The first In the runoff scenario, the first The outflow, average net head, and interval flow of the hydropower station in the second time period; The first The outflow, average net head, and interval flow of the hydropower station in the first time period; After the agent completes its exploration, it updates the water level sequence of the cascade hydropower system at the end of the first time period. Update the outflow sequence for the only runoff scenario in the first time period. and the second period Outflow Matrix for Each Runoff Scenario Proceed to step (5.9); where, For the second period cascade outflow sequences for individual runoff scenarios; (5.9) Calculation results Total power generation during the period ,like If the result is positive, the algorithm is considered converged, and the output is the water level process curve for each hydropower station and the corresponding time-period power generation flow; otherwise, let... Then return to step (5.3) for the next round of optimization; Step Six: As available information is updated, obtain the rolling time-domain cascade hydropower scheduling scheme.

2. The multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects according to claim 1, characterized in that, In step one, model building includes inputting historical runoff data within the cascade hydroelectric current domain, differential processing, selecting model parameters, residual analysis, and validation. The Monte Carlo forward simulation generates multiple sets of random runoff scenarios by using the SARIMAX model to generate multiple future runoff sequences. Before prediction, a logarithmic transformation is performed on the sequences, and after prediction, an exponential transformation is used to restore their physical meaning. The k-means clustering algorithm is then used to... The Monte Carlo simulation of runoff stochastic scenarios is reduced to A number of representative scenarios are identified, and the probability of occurrence for each representative scenario is recorded.

3. The multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects according to claim 1, characterized in that, The rolling time-domain cascade hydropower scheduling scheme in step six involves advancing the time-domain optimization window over time. Each advancement integrates the latest flow observation data and the previous scheduling scheme, repeating the prediction and optimization process to dynamically adjust the scheduling scheme.

4. The multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects according to claim 1, characterized in that, In step two: the specific steps for constructing the optimization model based on the scenario method are as follows: Objective function: ; In the formula: The total power generation of the cascade hydropower station system during the dispatching cycle is expressed in kWh. This refers to the serial number of the hydropower station. This refers to the number of hydropower stations in a cascade hydropower system. The time period number. , This represents the total number of scheduling periods; For the first The power output coefficient of the hydroelectric power station; For the first Runoff Scenario Hydropower station Power generation flow rate per time period, per unit ; For the first Runoff Scenario Hydropower station Average net head over a period of time, in meters; For each time period, the unit is... ; This indicates the number of runoff prediction scenarios; Representing the The probability of occurrence of a runoff scenario. ; The constraints include: Initial and final water level constraints: ; In the formula: and These are the initial and final water levels, respectively. and The first The initial water level and expected final water level of the reservoir, in meters (m). Reservoir water level constraints: ; In the formula: and The first reservoir The lower and upper limits of the water level at the end of the time period For the first The first reservoir Water level at the end of the time period, in meters; Output constraints: ; In the formula: and The first reservoir The lower and upper limits of output for each time period, in kW; For the first Hydropower station Power generation flow rate per time period, per unit ; For the first Hydropower station Average net head over a period of time, in meters; Water balance constraints: ; In the formula: For the first reservoir Storage capacity per time period, per unit ; The first reservoir Total inflow, outflow, interval flow, and wastewater discharge for each time period, per unit ; Power generation flow constraints: ; In the formula: and The first reservoir Lower and upper limits of power generation flow for a given period, in units ; For the first The first reservoir Power generation flow at the end of the period, per unit ; Outbound flow constraints: ; In the formula: and The first reservoir Lower and upper limits of outbound flow for a given period, in units ; For the first The first reservoir Outbound flow at the end of the period, unit ; Reservoir water level-capacity relationship function constraints: ; In the formula: For the first reservoir Water level at the end of the time period, in meters; This is a function relating water level and reservoir capacity. Downstream water level - outflow rate relationship function constraints: ; In the formula: For the first reservoir The average downstream water level during the specified period, in meters (m). This is the function relating downstream water level to outflow rate.

5. The multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects according to claim 1, characterized in that, In Step 3: The specific steps for single-agent reinforcement learning modeling of cascade hydropower spatial dimension are as follows: (3.1) Establishing the state space ; ; In the formula: For the first The first hydroelectric power station The water level at the beginning of the period, For the first The first hydroelectric power station Water level at the end of the time period, in meters; For the first Traffic flow during specific time periods For the first The upstream reservoir of the hydroelectric power station Outbound flow rate per time period, per unit ; (3.2) Establishing the action space ; ; In the formula: For the intelligent agent for the first reservoir The selected outbound flow rate for the specified time period, in units of ; To reduce the action space, the outflow constraint is coupled with the water level constraint in advance. That is, the upper and lower limits of the outflow under the water level constraint are calculated and their intersection is taken with the outflow constraint. The lower and upper limits of the outflow after constraint coupling are shown below: ; In the formula: and These are the first two constrained couplings. The first hydroelectric power station Lower and upper limits of outbound flow for a given period, in units ; Based on the upper and lower limits of the outbound flow obtained by taking the intersection, non-uniform discretization with an adaptive step size is performed. The specific discretization process is as follows: Set initial step size Step size growth rate and number of iterations The agent's best action in the previous round As the central point; among which, This represents the optimal outbound flow rate from the previous iteration. This is the starting point for discretizing towards the lower limit of warehouse exit. This is the starting point for discretization towards the upper limit of outbound shipments; Left sequence generation, from Discretize the lower limit of outbound flow and perform iterative calculation. ,until If the outbound flow rate is less than the lower limit, the lower limit of the outbound flow rate is added to the sequence; where, The left sequence of discrete outflow volume generated by the iteration, in units ; Right sequence generation, from Discretize the upper limit of outbound flow and perform iterative calculation. ,until If the outbound flow exceeds the upper limit, the upper limit of the outbound flow is added to the sequence; where, Discrete outflow rate of the right sequence generated by iteration, unit ; The left and right sequences and the center point are merged to obtain a non-uniform discrete total sequence, which is then adjusted. and By acquiring discrete sequences with varying precision and increasing the exploration density of potential optimal action neighborhoods, the agent can explore more efficiently, ultimately yielding the action space. ; (3.3) Setting rewards ; ; In the formula: For the first The power generation of the power station in two time periods, This is a constant used for exceeding the limit penalty; (3.4) Acquisition Strategy ; A heterogeneous strategy is adopted, with the target strategy being the action-value function and the behavioral strategy being the greedy coefficient method. Action value function: ; In the formula: For the agent targeting the first Local observation status of the hydropower station The chosen action; Value discount rate; The update formula for the action value function: ; In the formula: Indicates the first Hydropower station The maximum cumulative reward that can be obtained under this state. The learning rate; Greedy coefficient behavior strategy: ; Cosine annealing decay method for the greedy coefficient: ; In the formula: and These are the agent's maximum and minimum exploration rates, respectively. and These represent the current exploration round and the maximum exploration round for the agent, respectively.

6. The multi-agent reinforcement learning-driven stochastic intelligent scheduling method for cascade hydropower projects according to claim 3, characterized in that, In step four: the specific steps for solving the subproblem using a single agent combined with the Q-learning algorithm are as follows: (4.1) Initialize the Q table, all The table is initialized to 0. The number of cascade hydropower stations is ; (4.2) Action selection: using a greedy coefficient strategy based on the current action. Status of the hydroelectric power station Select outbound flow ; (4.3) Perform the action , transferred to the The new status of the hydroelectric power station And receive instant power generation rewards. The interaction between the agent and the environment yields a quadruple. Store in the experience replay cache; (4.4) Single-step update is adopted, and the storage of water and electricity scheduling experience and the updating of Q table are carried out simultaneously; (4.5) Let ,like Proceed to step (4.6); otherwise, let Return to step (4.2); (4.6) Repeat steps (4.2) to (4.5) until convergence.

Images