Deep reinforcement learning environment construction and reservoir intelligent scheduling decision method and system

Abstract

The invention discloses a deep reinforcement learning environment construction and reservoir intelligent scheduling decision method and system, and relates to the technical field of water resource management and intelligent control. Compared with a traditional scheduling method, the power generation income is improved by 12.8%, and the comprehensive benefit is improved by 25%; the constraint violation rate is greatly reduced: through an intelligent constraint processing mechanism, the constraint violation rate is reduced to 2.1% from 7.2% of a traditional method; the performance reduction range of the system under the extreme hydrological condition is controlled within 8.2% through a multi-algorithm fusion framework; due to an end-to-end learning mechanism, the system has self-learning and self-adaptive capabilities, and a scheduling strategy can be continuously optimized; an interpretability analysis tool is integrated, so that the trust degree of engineering technicians to the intelligent scheduling system is improved; the system design fully considers engineering application requirements, and supports progressive deployment and man-machine cooperative scheduling.

Description

Technical Field

[0001] This invention relates to the fields of water resource management and intelligent control technology, specifically to the construction of a deep reinforcement learning environment and a method and system for intelligent reservoir scheduling decision-making. Background Technology

[0002] Reservoir scheduling, as the "central nervous system" of watershed management and engineering operation, requires real-time balancing of objectives such as power generation, flood control, water supply, ecology, and navigation. However, the strong uncertainty of hydrological processes, the strong coupling of reservoir groups, and the nonlinearity of the reservoir capacity-water level-output relationship render traditional methods inadequate. Empirical rules rely on the long-term accumulation of schedulers and often fail to generalize to extreme inflows and cascading effects. While classical mathematical programming can provide optimality conclusions, it suffers from high computational costs and poor solution stability under high-dimensional, non-convex, and time-varying constraints. Heuristic algorithms such as swarm intelligence can extend feasible domain search, but lack online learning and adaptive capabilities, making it difficult to form a sustainable iterative "self-improvement" strategy system. These pain points drive the necessity of next-generation scheduling technologies that emphasize intelligence, data-driven approaches, and robust safety.

[0003] Looking further, the weaknesses of existing technologies are mainly concentrated in four aspects: First, the spatiotemporal coupling and long-sequence decision processing capabilities are insufficient, making it difficult to simultaneously characterize short-duration flood processes and multi-year-scale water inflow cycles; second, the default identification and handling mechanisms under strongly constrained scenarios (flood control level, ecological discharge, unit ramp-up rate, etc.) are imperfect, often triggering unacceptable risk exposures in extreme situations; third, the lack of dynamic trade-offs among multiple objectives and the static and rigid weight settings make it difficult to adapt and migrate with the water inflow situation and scheduling phase; fourth, the strategy generation process lacks interpretability and transparency, affecting trust and implementation on the engineering side.

[0004] Therefore, there is an urgent need for an intelligent scheduling paradigm that can continuously optimize within the closed loop of "learning-decision-correction-interpretation". Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for constructing a deep reinforcement learning environment and making intelligent reservoir scheduling decisions, so as to solve the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for constructing a deep reinforcement learning environment, comprising at least the following steps:

[0007] First, the Markov decision process is formalized.

[0008] The reservoir scheduling problem is formalized as a Markov decision process, using a quintuple. ,

[0009] Next, we designed the episode length and performed time modeling, training the episode length... It is a key hyperparameter affecting policy learning. In a limited-view MDP, the value function is expressed as:

[0010]

[0011] When the length of the series Smaller than the effective field of view At this time, the value estimate will be truncated, causing the agent to ignore it. On the one hand, delayed rewards after each step; on the other hand, excessively long episodes increase training time and sample requirements, reducing training efficiency.

[0012] Then, the reward function is designed, and the detailed structure of the reward function takes into account multiple physical objectives.

[0013] Power generation revenue is determined by both the water head and the effective outflow, and by the time period. The power generation capacity is calculated as follows:

[0014]

[0015] in, The average head, This is the tailwater level. For power generation flow, It is the acceleration due to gravity. To improve overall power generation efficiency;

[0016] Based on this, the power generation reward and the guaranteed output penalty are defined as follows:

[0017]

[0018]

[0019] in, and These are the weighting coefficients;

[0020] When the reservoir capacity exceeds the upper limit constraint at the end of the flood season Triggering flood control penalties:

[0021]

[0022] in, For a moment The final storage capacity, Due to the upper limit of storage capacity constraints, For flood control penalty weighting coefficient;

[0023] Ecological flow guarantee penalty when outflow is lower than ecological base flow Triggered at time:

[0024]

[0025] in, Weights for ensuring ecological flow;

[0026] Shipping guarantees use a square root penalty approach, where actual outflow is lower than navigation demand. hour:

[0027]

[0028] in, Weighting for ensuring shipping traffic flow;

[0029] In addition, to limit the range of motion adjustment, the original motion Actions after constraint correction The difference between them is mapped to a smoothing penalty:

[0030]

[0031] in, The weight indicating the limit of the action adjustment range;

[0032] Taking all factors into account, the single-step reward is expressed as follows:

[0033]

[0034] An Actor-Critic framework is constructed, leveraging the advantages of both value-based and policy-based methods. Through the collaborative optimization of two neural networks—the Actor network and the Critic network—in the reservoir scheduling problem, the Critic recursively estimates the value function, backpropagating multi-year delayed rewards to the current decision, providing the Actor with accurate long-term value signals. The baseline subtraction mechanism of the advantage function automatically adapts to differences in reward scales across different seasons.

[0035] Finally, the PPO algorithm and DDPG algorithm are introduced for application.

[0036] The intelligent reservoir scheduling and decision-making system adopts an end-to-end architecture of data-agent-constraint-fusion-training-decision support, including a data acquisition and preprocessing layer, an agent management layer with a multi-algorithm fusion framework, a constraint processing and security layer, a multi-objective decision fusion layer, a training and optimization layer, and a decision support and visualization layer.

[0037] Using real-time hydrological and meteorological data as input, and through a reinforcement learning decision kernel that integrates multiple algorithms, the system outputs executable release / output commands under strict constraints on perception and hierarchical handling. Driven by both online feedback and offline historical samples, the system continuously iterates and optimizes to achieve a learnable, controllable, and engineering-interpretable intelligent scheduling closed loop.

[0038] The data acquisition and preprocessing layer is used to integrate multi-source observations and establish a cleaning, normalization, and feature construction process for time-varying decision-making. The multi-source observations include at least water level, inflow, and meteorological data. At the same time, it combines historical sequences to generate predictive elements such as future inflow.

[0039] The intelligent agent management layer adopts a multi-algorithm fusion framework, which incorporates three types of intelligent agents: DQN (discrete start-up / operating condition switching), DDPG (continuous flow / output fine control), and PPO (robust strategy for constraint-intensive and perturbation scenarios), and sets an adaptive algorithm selector.

[0040] The constraint processing and safety assurance layer is used for the systematic engineering implementation, normalizes and advances constraint detection, monitors water level, flow rate, unit ramp rate, ecological discharge, etc. in real time, and provides three switchable mechanisms: C1 "projection correction", C2 "termination of heavy penalty" and C3 "amplitude penalty + correction".

[0041] The multi-objective decision fusion layer is based on a six-dimensional weight system, which maps objectives such as power generation revenue, water level constraints, flood control safety, ecological flow, navigation security and action smoothing into a unified reward space, and supports dynamic weight adjustment according to hydrological period, operation stage and external instructions.

[0042] The training and optimization layer adopts a hybrid paradigm of "offline pre-training + online fine-tuning". In the offline stage, long-sequence historical samples and simulation environment are used for sufficient training and cross-validation. In the online stage, the strategy is updated in small steps and fast iterations in combination with real-time performance feedback. The model version management, performance monitoring and rollback mechanism ensure the safe operation and controllable evolution.

[0043] The decision support and visualization layer presents the entire process of state-algorithm selection-constraint handling-parameter fusion in a visual manner, accompanied by multi-dimensional performance analysis and operation reports, and supports human-machine collaborative review and strategy root cause analysis.

[0044] The intelligent reservoir scheduling decision-making method includes at least the following steps:

[0045] A multidimensional state space is constructed, which is a 5-dimensional observation space including water level, inflow, predicted inflow, constraint state, and time coding;

[0046] A multi-algorithm fusion agent is established, which includes a DDPG continuous control agent, a PPO robust policy agent, and a DQN discrete action agent. An adaptive algorithm selection mechanism is designed in the multi-algorithm fusion agent.

[0047] A multi-constraint sensing and processing mechanism is implemented, which includes three constraint processing strategies: C1 corrective processing, C2 termination processing, and C3 magnitude penalty processing.

[0048] A multi-objective intelligent decision fusion is established, and the dynamic balance of multiple objectives is achieved through the configuration of six-dimensional reward weight coefficients. The multiple objectives include at least power generation, flood control, ecology and shipping.

[0049] To achieve end-to-end intelligent decision-making, an end-to-end learning framework is established, from raw observation data to scheduling decisions.

[0050] Furthermore, the construction of the multidimensional state space includes: adaptively selecting the observation type according to the scheduling scenario, wherein observation type 1 is basic inflow observation, observation type 2 is water level and inflow observation, observation type 3 adds predicted inflow information, observation type 4 adds constraint state awareness, and observation type 5 is complete state information observation containing time encoding.

[0051] Type 1: [Inflow] - Basic Observation

[0052] Observation Type 2: [Water Level, Inflow] - Basic Condition Observation

[0053] Observation Type 3: [Water Level, Inflow, Forecast Inflow] - Includes forecast information

[0054] Observation Type 4: [Water Level, Inflow, Predicted Inflow, Constraint Status] - Includes constraint sensing

[0055] Observation type 5: [Water level, Inflow, Predicted inflow, Constraint status, Time coding] - Complete status information.

[0056] Furthermore, the adaptive algorithm selection mechanism dynamically selects the optimal algorithm based on hydrological conditions and scheduling objectives. Under extreme flood conditions, the PPO algorithm is selected to ensure robustness; under normal scheduling conditions, the DDPG algorithm is selected to achieve fine control; and under complex conditions, an integrated decision-making method is selected.

[0057] If hydrological conditions equal "extreme flood":

[0058] Choose the PPO algorithm # Robustness first

[0059] elif Hydrological conditions == "Normal scheduling":

[0060] Select DDPG algorithm # Fine control priority

[0061] else:

[0062] Choose integrated decision-making # Multi-algorithm fusion.

[0063] Furthermore, the multi-constraint awareness processing mechanism adopts one of the C1, C2, and C3 mechanisms according to different situations, that is, the constraint processing strategy is dynamically selected according to the scheduling stage and security requirements;

[0064] The C1 mechanism is a corrective processing mechanism where the environment automatically projects the default action to the feasible domain boundary.

[0065] The C2 mechanism is a termination-type process; a breach of contract will immediately terminate the round and impose a severe penalty.

[0066] The C3 mechanism is a range-based penalty system that combines action correction with progressive penalties based on the severity of the breach.

[0067] Furthermore, the six-dimensional reward weight coefficient configuration system is represented as follows:

[0068] Reward function = α1 × Power generation reward + α2 × Water level constraint + α3 × Flood control safety + α4 × Ecological flow + α5 × Navigation guarantee + α6 × Action smoothing

[0069] Where α represents the weight;

[0070] A multi-objective adaptive weight adjustment mechanism is implemented to dynamically adjust the weights of each objective according to different hydrological periods.

[0071] Compared with the prior art, the beneficial effects of the present invention are:

[0072] Compared to traditional scheduling methods, this invention increases power generation revenue by 12.8% and overall benefits by 25%; significantly reduces constraint violation rate: through an intelligent constraint processing mechanism, the constraint violation rate is reduced from 7.2% in traditional methods to 2.1%; enhances system robustness: the multi-algorithm fusion architecture keeps the system's performance degradation under extreme hydrological conditions within 8.2%; achieves true intelligence: the end-to-end learning mechanism enables the system to have self-learning and adaptive capabilities, continuously optimizing scheduling strategies; provides decision-making transparency: integrated interpretable analysis tools increase the trust of engineers in the intelligent scheduling system; and has good engineering practicality: the system design fully considers engineering application needs, supporting gradual deployment and human-machine collaborative scheduling. Attached Figure Description

[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a framework diagram of the intelligent reservoir scheduling decision-making of the present invention;

[0075] Figure 2 This is a schematic diagram of the multi-algorithm fusion intelligent agent structure of the present invention;

[0076] Figure 3 This is a flowchart of the multi-constraint processing mechanism of the present invention;

[0077] Figure 4 This is a diagram illustrating the multi-objective convergence process during the training of this invention.

[0078] Figure 5 This is a diagram illustrating the decision-making process of the agent before, during, and after training in this invention.

[0079] Figure 6 This is a diagram illustrating the long-sequence scheduling process of the Three Gorges Dam as described in this invention.

[0080] Figure 7 This invention relates to the intelligent agent's decision-making law for water level and outflow. Detailed Implementation

[0081] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0082] Please see Figures 1-7 Based on the core concept of this invention, the following specific embodiments are proposed:

[0083] Example 1:

[0084] The method for constructing a deep reinforcement learning environment includes at least the following steps:

[0085] First, the Markov decision process is formalized.

[0086] The reservoir scheduling problem can be formalized as a Markov decision process, using a quintuple. The elements are defined as follows:

[0087] state space :state Encoding time All key information that influences future decisions;

[0088] By adopting a modular design, the complete state is a 5-dimensional vector:

[0089]

[0090] in: The current reservoir water level, in meters (m), reflects the reservoir's energy storage status. The inflow rate for the day is expressed in m³ / s and is determined by the upstream water flow. These are the upper limits of the water level constraints for the day, in meters, and are dynamically determined by the dispatching procedures. The average forecast inflow for the next 7 days is given in m³ / s to capture short-term hydrological trends. For periodic encoding of dates, This coding method indicates the day of the year and maintains seasonal continuity.

[0091] Action space :action For time period The outbound flow decision. To stabilize the training process, this invention employs a discrete action space:

[0092]

[0093] in To achieve discretization granularity, the discreteness is set to 50 in this study. The action boundary is set as follows: The lower realm m³ / s, upper bound This represents the maximum discharge capacity of the reservoir. An equal-spaced division strategy is adopted.

[0094]

[0095] State transition function : transfer function The system dynamics are described by the water balance equation:

[0096]

[0097]

[0098] in: For a moment The reservoir capacity (unit: 10,000 m³) is calculated using the inverse function of reservoir capacity and water level. Calculated from water level; s (1 day) is the time step; The reservoir capacity-water level function is monotonically increasing, and is obtained through interpolation by looking up a table of reservoir characteristic curves. The transfer process itself is deterministic, but the inflow... As a component of the state, it is driven by external hydrological processes, introducing randomness and uncertainty into the environment.

[0099] reward function Instant rewards The comprehensive power generation revenue and various operational constraints are detailed in Section 3.4. Its basic form is as follows:

[0100]

[0101] in: They are time steps The rewards include power generation revenue bonuses, guaranteed output bonuses, flood control and safety bonuses, ecological flow bonuses, shipping support bonuses, and smooth operation bonuses.

[0102] Discount factor Discount factor Balancing immediate rewards with long-term benefits, this invention sets... The effective horizon corresponding to this value is:

[0103]

[0104] This time span is approximately seven months, covering the transition period between the flood and dry seasons, allowing the agent to learn cross-seasonal storage and release coordination strategies. Theoretically, reservoir scheduling needs to consider long-term impacts throughout the year and even across years, but excessively large... This can lead to increased variance in value estimation and training instability. Sensitivity analysis shows that... Within the range, Achieving the optimal balance between convergence speed and long-term planning capabilities.

[0105] Next, we designed the episode length and performed time modeling, training the episode length... It is a key hyperparameter affecting policy learning. In a limited-view MDP, the value function is expressed as:

[0106]

[0107] When the length of the series Smaller than the effective field of view At this time, the value estimate will be truncated, causing the agent to ignore it. On the one hand, delayed rewards after each step; on the other hand, excessively long episodes increase training time and sample requirements, reducing training efficiency.

[0108] Implicit termination in traditional TV series design violates Markov property, even in the same physical state. Different optimal actions exist at different times, but implicit termination cannot convey time information to the agent. This invention employs explicit time encoding and a fixed-duration termination strategy: [The text abruptly shifts to a different topic] Mapped to sine-cosine pairs This encoding maintains cyclical continuity; each episode is strictly executed. The process is terminated after a certain step, and even if the constraints are violated, it is not terminated prematurely; learning is guided only through rewards and punishments.

[0109] Then, the reward function is designed, and the detailed structure of the reward function takes into account multiple physical objectives.

[0110] Power generation revenue is determined by both the water head and the effective outflow, and by the time period. The power generation capacity is calculated as follows:

[0111]

[0112] in, The average head (unit: m) is the average head. Tailwater level (unit: m). Power generation flow rate (unit: m³ / s). The acceleration due to gravity (m / s²) To improve overall power generation efficiency;

[0113] Based on this, the power generation reward and the guaranteed output penalty are defined as follows:

[0114]

[0115]

[0116] in, and These are the weighting coefficients;

[0117] When the reservoir capacity exceeds the upper limit constraint at the end of the flood season Triggering flood control penalties:

[0118]

[0119] in, For a moment The final storage capacity ( ), For the upper limit of storage capacity constraint ( ), For flood control penalty weighting coefficient;

[0120] Ecological flow guarantee penalty when outflow is lower than ecological base flow Triggered at time:

[0121]

[0122] in, Weights for ensuring ecological flow;

[0123] Shipping guarantees use a square root penalty approach, where actual outflow is lower than navigation demand. hour:

[0124]

[0125] in, Weighting for ensuring shipping traffic flow;

[0126] In addition, to limit the range of motion adjustment, the original motion Actions after constraint correction The difference between them is mapped to a smoothing penalty:

[0127]

[0128] in, The weight indicating the limit of the action adjustment range;

[0129] Taking all factors into account, the single-step reward is expressed as follows:

[0130]

[0131] The Actor-Critic framework is built, which leverages the advantages of value-based and policy-based methods. Through the collaborative optimization of two neural networks, the Actor network and the Critic network, in the reservoir scheduling problem, the Critic recursively estimates the value function and can backpropagate the delayed rewards over many years to the current decision, providing the Actor with accurate long-term value signals. The baseline subtraction mechanism of the advantage function automatically adapts to the differences in reward scales in different seasons, avoiding the gradient vanishing or exploding problem caused by inconsistent reward scales.

[0132] The Actor network (policy network) outputs action decisions based on the state, while the Critic network (value network) evaluates the long-term value of the state and provides learning signals to the Actor.

[0133] Actor networks will implement strategies Parameterized as a neural network, where This represents the network parameters. For a discrete action space, the Actor outputs the selection probability of each action. The training objective of the Actor is to maximize the expected cumulative reward, which can be formalized as:

[0134]

[0135] in Indicates by strategy The generated state-action trajectory For the length of the series, This is the discount factor (see Section 3.2 for details). For a moment The immediate reward. Using the policy gradient theorem, the objective function can be adjusted with respect to the parameters. The gradient is expressed as:

[0136]

[0137] in The dominance function measures the action. Compared to the state The relative advantages and disadvantages of the average level. The advantage function significantly reduces gradient variance and improves training stability by subtracting the value baseline.

[0138] The Critic network is responsible for learning the state-value function. ,in These are the network parameters. Critic optimizes these parameters by minimizing the Temporal Difference (TD) error:

[0139]

[0140] The advantage function can be calculated from the Critic output, and its simplest form is the TD error itself:

[0141]

[0142] To achieve a better balance between bias and variance, the Generalized Advantage Estimation (GAE) method is often used in practice, which introduces a smoothing parameter. To synthesize the TD error of different synchronization numbers:

[0143]

[0144] in Control bias-variance tradeoff: Corresponding to single-step TD error (low variance but high bias). This corresponds to the Monte Carlo return (unbiased but with high variance). The experiment in this invention is set up as follows: A good balance is achieved between the stability and accuracy of long-term value estimation.

[0145] Finally, the PPO algorithm and DDPG algorithm are introduced for application.

[0146] Proximal Policy Optimization (PPO), proposed by Schulman et al. in 2017, aims to provide training stability and performance guarantees comparable to Trust Region Policy Optimization (TRPO), while significantly reducing computational complexity by using only first-order optimization methods. PPO is an on-policy algorithm based on the Actor-Critic framework. It limits the policy update magnitude through a concise clipping mechanism, maintaining high sample efficiency while avoiding policy performance degradation.

[0147] Traditional policy gradient methods employ importance sampling:

[0148]

[0149] in The ratio of the probabilities of the new and old strategies. This represents the old policy before the update. This ratio characterizes the change in policy preference: This indicates that the new strategy favors this action. Conversely.

[0150] The PPO pruning objective function achieves soft constraints by limiting the probability ratio:

[0151]

[0152] in Will Limited to Within the interval, The pruning threshold is set to 0.2 in this paper. This objective function ensures that the probability ratio is within a certain range. Variation within range: When the ratio exceeds this range, the gradient is truncated to zero to prevent over-updating.

[0153] Training process: In each iteration, trajectory data is collected using the current strategy. Critic updates the value function:

[0154]

[0155] The Actor performs multiple rounds of updates based on the pruning target and incorporates an entropy regularization term to encourage exploration:

[0156]

[0157] in For policy entropy, , .

[0158] The goal of DDPG is to maximize the expected return under a deterministic policy.

[0159]

[0160] in It is a strategy Induced state distribution.

[0161] The DPG theorem states that:

[0162]

[0163] That is, through the chain rule, the gradient of the Actor comes from the derivative of the Critic with respect to the action.

[0164] The Critic network learns by minimizing the TD error, with the objective function being:

[0165]

[0166] Among them, the TD objective is:

[0167]

[0168] This format is similar to DQN, except that the action is not performed through... Instead, it is a deterministic action output by the target Actor network.

[0169] Using the DPG theorem above, the gradient of the Actor is:

[0170]

[0171] That is, first calculate the partial derivative of Critic with respect to the input action. Then calculate the partial derivative of the Actor's output with respect to the parameters. The two are multiplied and then backpropagated to update. .

[0172] To ensure stable training, DDPG introduces a "soft update" mechanism for the target network:

[0173]

[0174]

[0175] in (Usually taken as 0.001).

[0176] Example 2:

[0177] This embodiment proposes a reservoir intelligent scheduling decision-making method based on the above embodiment one, which includes at least the following steps:

[0178] Construct a multidimensional state space, which is a 5-dimensional observation space that includes water level, inflow, predicted inflow, constraint state, and time coding;

[0179] A multi-algorithm fusion agent is established, which includes a DDPG continuous control agent, a PPO robust policy agent, and a DQN discrete action agent. An adaptive algorithm selection mechanism is designed in the multi-algorithm fusion agent.

[0180] A multi-constraint sensing and processing mechanism is implemented, which includes three constraint processing strategies: C1 corrective processing, C2 termination processing, and C3 magnitude penalty processing.

[0181] Establish a multi-objective intelligent decision fusion, and achieve dynamic balance of multiple objectives through the configuration of six-dimensional reward weight coefficients. The multiple objectives include at least power generation, flood control, ecology and shipping.

[0182] To achieve end-to-end intelligent decision-making, an end-to-end learning framework is established, from raw observation data to scheduling decisions.

[0183] The construction of the multidimensional state space includes: adaptively selecting the observation type according to the scheduling scenario, where observation type 1 is the basic inflow observation, observation type 2 is the water level and inflow observation, observation type 3 adds predicted inflow information, observation type 4 adds constraint state awareness, and observation type 5 is the observation of complete state information including time coding.

[0184] Type 1: [Inflow] - Basic Observation

[0185] Observation Type 2: [Water Level, Inflow] - Basic Condition Observation

[0186] Observation Type 3: [Water Level, Inflow, Forecast Inflow] - Includes forecast information

[0187] Observation Type 4: [Water Level, Inflow, Predicted Inflow, Constraint Status] - Includes constraint sensing

[0188] Observation type 5: [Water level, Inflow, Predicted inflow, Constraint status, Time coding] - Complete status information.

[0189] The adaptive algorithm selection mechanism dynamically selects the optimal algorithm based on hydrological conditions and scheduling objectives. Under extreme flood conditions, the PPO algorithm is selected to ensure robustness; under normal scheduling conditions, the DDPG algorithm is selected to achieve fine control; and under complex conditions, an integrated decision-making method is selected.

[0190] If hydrological conditions equal "extreme flood":

[0191] Choose the PPO algorithm # Robustness first

[0192] elif Hydrological conditions == "Normal scheduling":

[0193] Select DDPG algorithm # Fine control priority

[0194] else:

[0195] Choose integrated decision-making # Multi-algorithm fusion.

[0196] The multi-constraint awareness processing mechanism adopts one of the C1, C2 and C3 mechanisms according to different situations, that is, dynamically selects the constraint processing strategy according to the scheduling stage and security requirements.

[0197] The C1 mechanism is a corrective processing mechanism where the environment automatically projects the default action to the feasible domain boundary.

[0198] The C2 mechanism is a termination-type process; a breach of contract will immediately terminate the round and impose a severe penalty.

[0199] The C3 mechanism is a range-based penalty system that combines action correction with progressive penalties based on the severity of the breach.

[0200] The six-dimensional reward weight coefficient configuration system is represented as follows:

[0201] Reward function = α1 × Power generation reward + α2 × Water level constraint + α3 × Flood control safety + α4 × Ecological flow + α5 × Navigation guarantee + α6 × Action smoothing

[0202] Where α represents the weight;

[0203] A multi-objective adaptive weight adjustment mechanism is implemented to dynamically adjust the weights of each objective according to different hydrological periods.

[0204] Example 3:

[0205] This embodiment proposes a reservoir intelligent scheduling decision system based on the above embodiment 2. It adopts an end-to-end architecture of data-agent-constraint-fusion-training-decision support, including a data acquisition and preprocessing layer, an agent management layer with a multi-algorithm fusion framework, a constraint processing and security layer, a multi-objective decision fusion layer, a training and optimization layer, and a decision support and visualization layer.

[0206] Using real-time hydrological and meteorological data as input, and through a reinforcement learning decision kernel that integrates multiple algorithms, the system outputs executable release / output commands under strict constraints on perception and hierarchical handling. Driven by both online feedback and offline historical samples, the system continuously iterates and optimizes to achieve a learnable, controllable, and engineering-interpretable intelligent scheduling closed loop.

[0207] The overall system block diagram is shown in the architecture diagram. Figure 1The corresponding functional modules will be explained one by one below;

[0208] The data acquisition and preprocessing layer integrates multi-source observations to establish a cleaning, normalization, and feature construction process for time-varying decision-making. Multi-source observations include at least water level, inflow, and meteorological data. Simultaneously, it combines historical sequences to generate predictive elements such as future inflows, providing a forward-looking state description for policy learning. The entire data layer serves both online inference and provides a unified data foundation for offline training, ensuring consistency between training and deployment and controllable distribution bias.

[0209] The intelligent agent management layer adopts a multi-algorithm fusion framework, which incorporates three types of intelligent agents: DQN (Discrete Start-up / Switching), DDPG (Continuous Flow / Output Fine Control), and PPO (Robust Policy in Constraint-Dense and Disturbance Scenarios), and sets up an adaptive algorithm selector. This selector dynamically optimizes candidate algorithms based on hydrological conditions, current operational objectives, and historical performance, so that the strategy favors the stability of PPO during high-risk flood periods, releases the fine control advantages of DDPG during normal operation periods, and can also adopt integrated decision-making when necessary to improve adaptability in uncertain scenarios.

[0210] The constraint processing and safety assurance layer is used for the systematic engineering implementation, normalizing and preemptively monitoring constraint detection. It provides real-time monitoring of water level, flow rate, unit ramp rate, and ecological discharge, and offers three switchable mechanisms: C1 "projection correction", C2 "termination and heavy penalty", and C3 "amplitude penalty + correction". When a violation risk is detected, the processing path is adaptively selected according to the scheduling stage and severity, ensuring both the feasibility and safety boundary of instruction implementation, and shaping the search direction of the intelligent agent with reasonable penalty signals, reducing the overfitting of the strategy to extreme samples and the neglect of soft constraints.

[0211] The multi-objective decision fusion layer is based on a six-dimensional weight system, which maps objectives such as power generation revenue, water level constraints, flood control safety, ecological flow, navigation security, and action smoothing into a unified reward space. It also supports dynamic weight adjustment based on hydrological period, operation stage, and external instructions. After obtaining the components of each objective, the fusion device performs weighted synthesis and action optimization, so that the final instruction achieves a comprehensive balance of revenue, risk, and stability while meeting hard constraints. Moreover, the weights and components are traceable, which facilitates review and interpretation by the engineering side.

[0212] The training and optimization layers adopt a hybrid paradigm of "offline pre-training + online fine-tuning". In the offline stage, long-sequence historical samples and simulation environment are used for sufficient training and cross-validation. In the online stage, the strategy is updated in small steps and quickly by combining real-time performance feedback. The safe operation and controllable evolution are ensured by model version management, performance monitoring and rollback mechanism. Empirical results show that in long-sequence simulation of typical watersheds, the system achieves a significant improvement in annual power generation and comprehensive benefits compared with traditional rule scheduling, while the constraint violation rate is significantly reduced, which verifies the effectiveness of multi-algorithm fusion and constraint hierarchical handling.

[0213] At the decision support and visualization level, the entire process of state—algorithm selection—constraint handling—parameter fusion is presented in a visual manner, accompanied by multi-dimensional performance analysis and operation reports, supporting human-machine collaborative review and strategy root cause analysis. In terms of hardware and deployment, the system is equipped with GPU-accelerated computing nodes, distributed storage, and high-speed networks for engineering sites to ensure the timeliness of data processing and strategy reasoning, and supports gradual replacement and bypass switching to reduce deployment risks and shorten the implementation cycle.

[0214] Furthermore, the system operation process can be summarized as follows: after data enters the standardized pipeline, it forms the current state; the algorithm selector performs immediate optimization of the policy agent; the agent gives the initial action, which is corrected by the constraint processor with penalty feedback; and then the multi-objective fusion unit generates the final executable instruction. Monitoring data after instruction execution is fed back to the training and monitoring modules, driving continuous policy improvement through a closed loop. This process ensures a unified link from observation to decision-making, from decision-making to evaluation, and then to retraining, enhancing the system's resilience and portability under unconventional water inflows and sudden operating conditions.

[0215] The intelligent reservoir scheduling decision-making system of the present invention can be deployed in the following hardware environments:

[0216] Computer: A high-performance computing server equipped with GPU acceleration;

[0217] Storage: Distributed storage system supporting large-scale hydrological data storage;

[0218] Network: High-speed network connection, supporting real-time data transmission.

[0219] Based on the above embodiments, the following performance verification example is proposed:

[0220] Based on long-term simulations and comparative experiments of the Three Gorges Reservoir, the performance of the proposed scheduling system, which combines "multi-algorithm fusion + hierarchical constraint handling + multi-objective dynamic trade-offs," is verified. The verification process revolves around four core diagrams: Figure 4 The convergence trajectory of multi-objective rewards during training is given; Figure 5 The comparison of the agent's decision-making process at different stages before, during, and after training is presented. Figure 6To simulate the long-sequence scheduling process of the Three Gorges Dam; Figure 7 This invention presents the intelligent agent's decision-making law regarding water level and outflow. Overall results show that, compared to traditional rule-based scheduling, annual power generation increases by 12.8%, comprehensive scheduling efficiency improves by approximately 25%, flood control safety assurance rate reaches 99.8%, and ecological flow satisfaction rate increases to 96.5%, achieving simultaneous improvement in systemic benefits and safety.

[0221] First, combined Figure 4 The convergence curve reveals that the comprehensive reward (power generation, water level constraints, flood control, ecology, navigation, and action smoothing) composed of "six-dimensional weights" exhibits a step-like increase across three stages: early exploration, mid-term shaping, and late steady state. In the early stage, due to the coexistence of strategy exploration and constraint trial and error, the comprehensive reward fluctuates significantly. As the graded handling of the three mechanisms (C1 / C2 / C3) is repeatedly "explicitized" during training, constraint violations shift from frequent to sporadic, the proportion of penalty items decreases, and the comprehensive reward enters a "smooth growth period." In the later stage, the reward curve fluctuates narrowly around a high level, indicating that the strategy has formed a stable output / release behavior habit within the constrained feasible domain. This trend aligns with our fusion design of "dynamic weight configuration + adaptive weight adjustment": the weights of constraint and safety-related components are temporarily increased in scenarios such as flood-dry transitions and sudden changes in water inflow, thereby sacrificing a small amount of immediate power generation revenue in exchange for robust convergence within the risk boundary.

[0222] Secondly, comparison Figure 5 The decision trajectory shown before, during, and after training intuitively illustrates the evolution of the action sequence and water level process from "coarse-oscillating" to "refined-smooth": Before training, the action changes are large and lacks sensitivity to soft and hard constraints, easily leading to "tug-of-war" near the water level boundary; from the mid-term onwards, C1 projection correction and C3 amplitude penalty gradually suppress out-of-bounds driving, and PPO / DDPG are adaptively selected within their respective advantageous scenarios, with the water level process significantly converging to the target corridor; in the later stage of convergence, the action smoothing term suppresses excessively rapid ramp / downhill, the unit ramp rate is controlled, the ecological discharge satisfaction rate is improved, and fluctuations converge, giving the strategy a stronger "passivation" and regression ability against abnormal inflow impacts. This evolutionary path reflects the closed-loop coordination of "algorithm selection—constraint handling—weight fusion": when the flood risk increases, the selector favors PPO to maintain strategy stability; during normal periods, it favors DDPG to improve fine control and energy conversion efficiency.

[0223] Again, from Figure 6The long-sequence scheduling process allows for a quantifiable comparison of the differences between "our method" and "traditional rule-based scheduling" under seasonal scales and extreme scenarios: During the main flood season, in the process of flood control, peak reduction, and reservoir emptying, the agent maintains a flood control guarantee rate of 99.8% through the coordination of upstream pre-storage and downstream flood discharge windows; during the dry season, ecological discharge changes from "phased under-guarantee" to "continuous compliance," increasing the ecological satisfaction rate to 96.5%. From an annual perspective, the power generation strategy utilizes available headwaters "forward" without exceeding safety boundaries and significantly reduces water wastage, resulting in an annual power generation increase of 12.8% and an overall benefit increase of approximately 25%. These improvements stem from three combined factors: ① Graded constraint handling reduces "hard losses" caused by boundary violations; ② Smooth action constraints reduce efficiency losses caused by frequent start-stop cycles; ③ Adaptive algorithm selection allows the strategy to "maximize its strengths" in different water inflow types / operational stages.

[0224] Figure 7 This diagram illustrates the scatter distribution of agent policies trained with 1-year, 4-year, and 8-year episode length configurations in the water level-outflow decision space. Monthly coloring reveals the coupling relationship between decision regularity and seasonal patterns. The diagram uses an HSV color wheel to map the 12 months. The overall decision-making patterns clearly identify three decision levels and their seasonal patterns:

[0225] (1) High water level - low outflow zone (165-175m, 6000-12000 m³ / s): This zone is dominated by winter and spring seasons (purple, red, and yellow dots, November to April of the following year), corresponding to a high water level operation strategy during the dry season. Statistical data shows that the average outflow in this zone is 7431 m³ / s, with a standard deviation of only 1609 m³ / s, making it the zone with the most consistent decision-making among the three zones. The strategy during this stage is highly consistent in adopting a "water storage and peak regulation" approach, maintaining the water level above 165m to reserve storage capacity for power generation during the dry season. In contrast, the annual water level of the 1-year drama series ( Figure 7 a) The distribution of winter points (purple, red) in this region is more dispersed, with a standard deviation of 2879 m³ / s, reflecting that the strategy's learning of dry season water storage patterns is insufficient; 8-year drama series ( Figure 7 c) Although the region also exhibits good aggregation (standard deviation 1494 m³ / s), the overall outflow is slightly higher than that of the 4-year average (7590 m³ / s), indicating that its water storage strategy is somewhat conservative.

[0226] (2) Medium water level - flexible scheduling zone (155-165m, 8000-25000 m³ / s): This zone exhibits a distinct monthly mixed characteristic, with widespread distribution of points in late spring (yellow-green, April-May) and autumn (blue-purple, September-October), corresponding to flexible scheduling during the transition period between flood and dry seasons. The average outflow in this zone over four years was 11126 m³ / s, with a standard deviation of 5700 m³ / s, demonstrating the greatest decision-making flexibility—the strategy can dynamically adjust the outflow according to the inflow conditions and constraints of different months. In late spring (April-May), points are concentrated in the medium outflow range (12000-18000 m³ / s), completing the pre-flood drawdown; in autumn (September-October), points are simultaneously distributed in both the low outflow (conservative water storage) and high outflow (seizing the tail end of the flood to replenish water) ranges, reflecting the engineering wisdom of "adapting to circumstances". The decision-making process for 1-year dramas in this region is too dispersed (standard deviation 7913 m³ / s), lacking a clear monthly-action correspondence pattern; while the 8-year dramas are too conservative (average 10012 m³ / s), failing to make full use of storage capacity for peak shaving.

[0227] (3) Low water level-high outflow area (145-155m, 15000-37000 m³ / s): This area is mainly characterized by the summer and autumn flood season (greenish-blue, June-September), corresponding to low water level flood control operation during the flood season. The average outflow in this area over 4 years reached 18955 m³ / s, with a standard deviation of 6973 m³ / s. The strategy is to maintain a relatively high outflow to meet ecological and navigation needs while ensuring flood control safety. Figure 7 b shows that the points in July and August (dark cyan) are mainly concentrated in the low water level and high outflow area, which is highly consistent with the actual dispatching procedure of the Three Gorges Reservoir "low water level operation during the flood season"

[34] . It is worth noting that a small number of winter points (purple, red) also appear in this area, which corresponds to the low reservoir capacity operation in extremely dry years - after a continuous dry year, even if winter comes, the water level is difficult to recover to the ideal height. At this time, the strategy is to maintain a relatively high outflow to balance power generation and ecological needs. This "atypical" decision-making mode precisely reflects the interannual memory ability learned by the 4-year series: the strategy not only responds to the current season, but also considers the cumulative effect of the previous hydrological history. In contrast, the low water level area of ​​the 1-year series is more chaotic (standard deviation 8514 m³ / s), and the winter point (purple) appears abnormally in this area, reflecting the deviation of the strategy in understanding the seasonal constraints; although the monthly purity of this area is high in the 8-year series, the outflow distribution is slightly conservative and does not fully respond to the high flood flow.

[0228] The agent performs well in three dimensions: constraint satisfaction rate, decision regularity, and seasonal consistency. The learned strategies not only have excellent performance but also conform to the engineering logic and practical experience of reservoir scheduling.

[0229] Finally, the ablation and robustness results further explain the mechanism behind the aforementioned improvements. Regarding constraint handling, C2 (termination penalty) achieves the lowest violation rate of 0.8%, but incurs higher learning costs and more early round interruptions; C1 (pure correction) has a violation rate of 7.2%, achieving rapid stabilization but lacking sufficient "respect" for soft constraints; C3 (amplitude penalty + correction) achieves a better balance between a 2.1% violation rate and learning efficiency, making it suitable as the default strategy for engineering applications. In terms of algorithm adaptability, PPO is optimal in scenarios with dense constraints and strong perturbations, while DDPG is better in scenarios with continuous precision control; the fusion of multiple algorithms improves overall performance by approximately 15% compared to a single algorithm, validating the necessity of "optimal selection and complementarity." In summary, the three figures collectively support the comprehensive advantages of this system in terms of benefits, risks, and operability, and provide a quantitative basis for the selection of weights and mechanisms for engineering deployment.

[0230] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for constructing a deep reinforcement learning environment, characterized in that, At least the following steps are included: First, the Markov decision process is formalized. The reservoir scheduling problem is formalized as a Markov decision process, using a quintuple. , Next, we designed the episode length and performed time modeling, training the episode length... It is a key hyperparameter affecting policy learning. In a limited-view MDP, the value function is expressed as: When the length of the series Smaller than the effective field of view At this time, the value estimate will be truncated, causing the agent to ignore it. On the one hand, delayed rewards after each step; on the other hand, excessively long episodes increase training time and sample requirements, reducing training efficiency. Then, the reward function is designed, and the detailed structure of the reward function takes into account multiple physical objectives. Power generation revenue is determined by both the water head and the effective outflow, and by the time period. The power generation capacity is calculated as follows: in, The average head, This is the tailwater level. For power generation flow, It is the acceleration due to gravity. To improve overall power generation efficiency; Based on this, the power generation reward and the guaranteed output penalty are defined as follows: in, and These are the weighting coefficients; When the reservoir capacity exceeds the upper limit constraint at the end of the flood season Triggering flood control penalties: in, For a moment The final storage capacity, Due to the upper limit of storage capacity constraints, For flood control penalty weighting coefficient; Ecological flow guarantee penalty when outflow is lower than ecological base flow Triggered at time: in, Weights for ensuring ecological flow; Shipping guarantees use a square root penalty approach, where actual outflow is lower than navigation demand. hour: in, Weighting for ensuring shipping traffic flow; In addition, to limit the range of motion adjustment, the original motion Actions after constraint correction The difference between them is mapped to a smoothing penalty: in, The weight indicating the limitation of the action adjustment range; Taking all factors into account, the single-step reward is expressed as follows: An Actor-Critic framework is constructed, leveraging the advantages of both value-based and policy-based methods. Through the collaborative optimization of two neural networks—the Actor network and the Critic network—in the reservoir scheduling problem, the Critic network recursively estimates the value function, backpropagating multi-year delayed rewards to the current decision, providing the Actor with accurate long-term value signals. The baseline subtraction mechanism of the advantage function automatically adapts to differences in reward scales across different seasons. Finally, the PPO algorithm and DDPG algorithm are introduced for application.

2. A reservoir intelligent scheduling decision-making method, built based on the deep reinforcement learning environment construction method described in claim 1, characterized in that: At least the following steps are included: A multidimensional state space is constructed, which is a 5-dimensional observation space including water level, inflow, predicted inflow, constraint state, and time coding; A multi-algorithm fusion agent is established, which includes a DDPG continuous control agent, a PPO robust policy agent, and a DQN discrete action agent. An adaptive algorithm selection mechanism is designed in the multi-algorithm fusion agent. A multi-constraint sensing and processing mechanism is implemented, which includes three constraint processing strategies: C1 corrective processing, C2 termination processing, and C3 magnitude penalty processing. A multi-objective intelligent decision fusion is established, and the dynamic balance of multiple objectives is achieved through the configuration of six-dimensional reward weight coefficients. The multiple objectives include at least power generation, flood control, ecology and shipping. To achieve end-to-end intelligent decision-making, an end-to-end learning framework is established, from raw observation data to scheduling decisions.

3. The intelligent reservoir scheduling decision-making method according to claim 2, characterized in that: The construction of the multidimensional state space includes: adaptively selecting the observation type according to the scheduling scenario, where observation type 1 is basic inflow observation, observation type 2 is water level and inflow observation, observation type 3 adds predicted inflow information, observation type 4 adds constraint state perception, and observation type 5 is complete state information observation containing time coding.

4. The intelligent reservoir scheduling decision-making method according to claim 2, characterized in that: The adaptive algorithm selection mechanism dynamically selects the optimal algorithm based on hydrological conditions and scheduling objectives. Under extreme flood conditions, the PPO algorithm is selected to ensure robustness; under normal scheduling conditions, the DDPG algorithm is selected to achieve fine control; and under complex conditions, an integrated decision-making method is selected.

5. The intelligent reservoir scheduling decision-making method according to claim 2, characterized in that: The multi-constraint perception and processing mechanism adopts one of the C1, C2 and C3 mechanisms according to different situations, that is, dynamically selects the constraint processing strategy according to the scheduling stage and security requirements; The C1 mechanism is a corrective processing mechanism where the environment automatically projects the default action to the feasible domain boundary. The C2 mechanism is a termination-type process; a breach of contract will immediately terminate the round and impose a severe penalty. The C3 mechanism is a range-based penalty system that combines action correction with progressive penalties based on the severity of the breach.

6. The intelligent reservoir scheduling decision-making method according to claim 2, characterized in that: The six-dimensional reward weight coefficient configuration system is represented as follows: Reward function = α1 × Power generation reward + α2 × Water level constraint + α3 × Flood control safety + α4 × Ecological flow + α5 × Navigation guarantee + α6 × Action smoothing Where α represents the weight.

7. A reservoir intelligent scheduling and decision-making system, used in the reservoir intelligent scheduling and decision-making method according to any one of claims 2-6, adopting an end-to-end architecture of data-agent-constraint-fusion-training-decision support, characterized in that: This includes a data acquisition and preprocessing layer, an agent management layer employing a multi-algorithm fusion framework, a constraint processing and security layer, a multi-objective decision fusion layer, a training and optimization layer, and a decision support and visualization layer. Using real-time hydrological and meteorological data as input, and through a reinforcement learning decision kernel that integrates multiple algorithms, the system outputs executable release / output commands under strict constraints on perception and hierarchical handling. Driven by both online feedback and offline historical samples, the system continuously iterates and optimizes to achieve a learnable, controllable, and engineering-interpretable intelligent scheduling closed loop. The data acquisition and preprocessing layer is used to integrate multi-source observations and establish a cleaning, normalization, and feature construction process for time-varying decision-making. The multi-source observations include at least water level, inflow, and meteorological data. At the same time, it combines historical sequences to generate predictive elements such as future inflow. The agent management layer adopts a multi-algorithm fusion framework, which incorporates three types of agents: DQN, DDPG, and PPO, and sets an adaptive algorithm selector. The constraint processing and safety assurance layer is used for the systematic engineering implementation, normalizes and advances constraint detection, monitors water level, flow rate, unit ramp rate and ecological discharge in real time, and provides three switchable mechanisms: C1 "projection correction", C2 "termination of heavy penalty" and C3 "amplitude penalty + correction". The multi-objective decision fusion layer is based on a six-dimensional weight system, which maps objectives such as power generation revenue, water level constraints, flood control safety, ecological flow, navigation security and action smoothing into a unified reward space, and supports dynamic weight adjustment according to hydrological period, operation stage and external instructions. The training and optimization layers adopt a hybrid paradigm of offline pre-training and online fine-tuning. In the offline stage, long-sequence historical samples and simulation environment are used for sufficient training and cross-validation. In the online stage, real-time performance feedback is combined to perform small-step rapid iterative strategy updates. The model version management, performance monitoring and rollback mechanism ensure safe operation and controllable evolution. The decision support and visualization layer presents the entire process of state-algorithm selection-constraint handling-parameter fusion in a visual manner, accompanied by multi-dimensional performance analysis and operation reports, and supports human-machine collaborative review and strategy root cause analysis.

Images