Improved reservoir dispatching safety reinforcement learning method based on Lagrange method

Abstract

The invention discloses an improved reservoir dispatching safety reinforcement learning method based on a Lagrange method, and relates to the technical field of reservoir dispatching. The Lagrange duality method has obvious advantages in the aspect of security constraint processing, and compared with a corrective method and a penalty method, the Lagrange method has the advantages that the constraint weight is adaptively adjusted, so that the target optimization performance is remarkably improved while the constraint is satisfied; experimental results show that the method is superior to other methods in the aspects of constraint satisfaction and generating capacity, and the constraint violation degree is reduced by more than 80%; according to the method, an action mask mechanism is adopted, through the synergistic effect of preposed defense and posterior optimization, the training stability is effectively improved, a complete rule mask can precisely define a safety boundary, the violation rate is reduced to zero from 21%-79%, and meanwhile strategy convergence is accelerated.

Description

Technical Field

[0001] This invention relates to the field of reservoir scheduling technology, specifically to a reservoir scheduling safety reinforcement learning method based on the Lagrange method. Background Technology

[0002] Reservoir operation is essentially a safety-critical decision-making problem. Unlike general optimization tasks, reservoir operation is subject to strict physical and safety constraints: during the flood season (June-September), the flood control limit water level must be maintained to reserve flood control capacity; failure to do so may lead to downstream flooding; during the drawdown season (October-December), a strict water level drop rate (usually <0.6m / d) must be followed to protect the stability of the reservoir banks; excessively rapid drawdown may trigger geological disasters such as landslides.

[0003] During the dry season (January to May), a minimum operating water level must be maintained to ensure power generation and water supply. Excessively low water levels will lead to unit shutdowns and ecological degradation. Outflow from the reservoir must be controlled within the equipment's carrying capacity and the safe discharge range of the downstream river channel; exceeding these limits will endanger the dam's structural safety or the lives and property downstream. These stringent constraints form a strict "safety envelope," and any violation could lead to catastrophic consequences. Globally, over 58,000 large reservoirs supply 15% of the world's power generation and irrigation water. Their scheduling decisions directly impact the safety and economic development of hundreds of millions of people, and the safety requirements are as stringent as those in other safety-critical fields such as autonomous driving and medical diagnostics.

[0004] Traditional scheduling methods primarily rely on empirical procedures and mathematical optimization. Empirical procedures ensure safety through pre-defined "water level-time period-outflow" lookup tables, but they struggle to address non-stationary water flow patterns under climate change and cannot dynamically balance multiple objectives (flood control, power generation, ecology, and navigation). Mathematical optimization methods (such as dynamic programming and linear programming) can find optimal solutions, but they are limited by the "curse of dimensionality" and model assumptions, resulting in high computational costs and difficulty in real-time response under multi-temporal scale coupling and complex constraint combinations. In recent years, deep reinforcement learning (DRL) has shown potential in single-database and multi-database scheduling due to its end-to-end learning and adaptive decision-making capabilities. However, standard reinforcement learning, with its sole objective of maximizing cumulative reward, may frequently experiment or even severely violate safety constraints during the exploration process, leading to training failures or the development of dangerous, undeployable strategies. How to strictly ensure physical constraints such as water level and flow during the learning process and achieve safe reinforcement learning has become a core bottleneck restricting the implementation of intelligent reservoir scheduling. Its urgency and technical difficulty are comparable to "collision avoidance guarantee" in autonomous driving and "stability guarantee" in robot control.

[0005] Safe reinforcement learning (SRL) aims to train policies that maximize cumulative rewards while satisfying safety constraints, and has demonstrated significant application value in safety-critical fields such as autonomous driving, robot control, and power systems. Its core challenge lies in avoiding dangerous states during policy exploration while ensuring learning efficiency and final performance. Constrained Markov Decision Processes (CMDPs) provide a theoretical framework for this, formalizing safety constraints as upper bounds on expected costs and solving them using methods such as Lagrange duality and trust region optimization. However, general SafeRL methods often assume that the constraint functions are known and differentiable, making them difficult to apply directly in practical engineering systems. Especially in reservoir scheduling scenarios, safety constraints are characterized by multi-source heterogeneity (water level, flow rate, equipment capacity, downstream load), state dependence (constraints differ between flood and dry seasons), and hard thresholds (violations can lead to disasters). Existing methods face three prominent problems.

[0006] First, the constraint handling mechanism lacks systematic comparison and theoretical guarantee. Current reservoir scheduling literature mostly adopts a fixed-weight penalty term, adding the constraint violation to the reward function with a fixed coefficient. However, this approach has three major drawbacks: (1) The weight coefficient requires a lot of trial and error adjustment - too small a weight leads to frequent violations, and too large a weight makes the strategy overly conservative or even training failure. In practical applications, dozens of weights often need to be tested to find a relatively reasonable configuration; (2) It cannot guarantee the expected satisfaction of the constraint - there is a lack of convergence theory support, and there may still be unacceptable violation risks after training; (3) Static weights are difficult to adapt to dynamic constraints - the flood season has extremely high requirements for flood control safety, and the dry season has strict requirements for ecological flow protection, but fixed weights cannot dynamically adjust the constraint intensity according to the time period. Some studies adopt a termination strategy (violation means termination of the round), but this will lead to low exploration efficiency in long-term scheduling. More importantly, existing studies lack a systematic comparison of different constraint handling mechanisms - the corrective method (only correction without punishment), the penalty method (fixed weight), and the theoretical method (adaptive Lagrange) each have their advantages and disadvantages, but their relative effects in the reservoir scheduling scenario have not been quantitatively evaluated, making it difficult for researchers and engineers to choose the optimal solution.

[0007] Second, the proactive defense mechanism has not been fully utilized. In engineering practice, dispatchers typically pre-screen candidate actions based on current water levels, equipment capacity, and downstream carrying capacity to avoid entering dangerous states. This engineering wisdom of "pre-screening" has not yet been fully absorbed in SafeRL research. Invalid action masking technology has been proven to significantly improve training efficiency and reduce violation risks in fields such as robot control and UAV path planning. Its core idea is to mask actions that do not meet constraints before action sampling. However, in reservoir scheduling, existing research either only uses simple upper and lower flow limit masks without considering state-dependent dynamic constraints, or uses rule-based action filtering but has low integration with the reinforcement learning training process. There is a lack of empirical verification of the synergistic effect between the mask mechanism and the posterior constraint reinforcement method—the mask provides pre-screening, and Lagrangian provides posterior optimization; how the two form a "two-layer safety framework" remains unclear.

[0008] Third, the safety impact of nonlinear action mapping has not been systematically evaluated. Discrete action spaces are widely used due to their training stability and ease of deployment, but the action mapping method directly affects the strategy's safety exploration capability. Traditional linear mapping distributes actions uniformly across the entire flow range, ignoring the differentiated resolution requirements of different flow intervals: ecological baseflow (typically 5,000-8,000 m³ / s) requires high-precision control to meet downstream ecological needs, but linear mapping only allocates about 12% of actions in this interval; emergency flood discharge (>25,000 m³ / s) has lower resolution requirements, but the same 12% allocation of actions results in resource waste. Nonlinear mapping (logarithmic, square, cubic, etc.) can provide higher resolution in critical flow segments by adjusting the action density distribution, but the impact of different mapping functions on safety has not been systematically studied. In summary:

[0009] Reservoir scheduling, as a typical safety-critical decision-making problem, requires intelligent decision-making systems to strictly meet physical constraints such as water level and flow rate while optimizing overall benefits. However, standard reinforcement learning frequently violates safety constraints during the exploration process. Existing constraint handling methods (fixed-weight penalties) lack theoretical guarantees and are difficult to tune hyperparameters, severely restricting the practical deployment of reinforcement learning in reservoir scheduling. Therefore, a new solution is needed to address these issues. Summary of the Invention

[0010] The purpose of this invention is to provide a reservoir scheduling security reinforcement learning method based on the Lagrange method, so as to solve the problem of which constraint processing mechanism can balance security, performance and deployability in reservoir scheduling scenarios.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a reservoir scheduling safety reinforcement learning method based on the Lagrange method, comprising at least the following steps:

[0012] S1: Treat the reservoir scheduling problem as a sequential decision problem with security constraints, and use the Constrained Markov Decision Process (CMDP) to formally model the problem, representing the reservoir system as a seven-tuple;

[0013] S2: In the strategy sampling phase, a state-driven action mask mechanism is introduced. Before each decision, actions that obviously do not meet the physical constraints are dynamically filtered out based on the current reservoir state. This creates a "front-line defense" outside of strategy learning. The state-driven action mask mechanism calculates feasible outflow intervals based on the water balance equation and physical constraints. It is then transformed into action index intervals through action-flow mapping to filter out illegal actions during the strategy sampling phase.

[0014] S3: Perform nonlinear motion mapping and resolution design, and adopt a nonlinear motion mapping function within a unified framework. By adjusting The shape allows for different motion densities to be allocated across different flow ranges;

[0015] S4: At the policy optimization level, a constraint reinforcement method based on Lagrange duality is adopted to ensure that the policy satisfies the CMDP constraint in the long-term scale, thus transforming the constraint optimization problem of CMDP into a saddle point search problem in the policy space and dual space.

[0016] Furthermore, S1 includes at least the following steps:

[0017] Represent the reservoir system as a 7-tuple:

[0018]

[0019] in: For state space, Indicates time The system status; For the action space, Indicates the agent at time... The selected scheduling decision; As the state transition kernel, it describes the state transition process during the execution of actions. The system after Evolved to The probability of; This is an instant reward function used to characterize the comprehensive benefits of power generation, flood control, ecology, and shipping. The constraint cost function is used to quantify the degree of violation of safety constraints such as water level and flow rate; The initial state distribution; Discount factor; The upper bound of the allowable long-term average constraint cost is used to characterize the acceptable level of security risk.

[0020] In the specific implementation, state It should include at least the current water level, inflow rate, water forecast for the next few days, dynamic water level upper and lower limits for the corresponding time period, and date code, as shown in the following formula:

[0021]

[0022] in, Current water level (in meters); Current inbound flow rate (unit: m³) / s); For the inflow prediction vector for the next few days; and These are the time-varying lowest and highest operating water levels, respectively. Normalized date encoding, used to characterize seasonality;

[0023] The evolution of reservoir capacity-water level follows the principle of mass conservation, denoted as... For a moment Storage capacity (unit: m) ), Outbound flow rate (unit: m³) / s), Let the scheduling time step be (in seconds), then the water balance equation is:

[0024]

[0025] The monotonic function between reservoir capacity and water level obtained from topographic surveys Association, that is:

[0026]

[0027] in, As the inverse function of the reservoir capacity-water level curve, it ensures that environmental dynamics strictly obey water conservation and topographic conditions, providing a physical basis for subsequent constraint design;

[0028] Under the CMDP framework, strategy The optimization objective is to maximize the cumulative discount reward while satisfying long-term constraints.

[0029]

[0030] in, The length of the round; Cost is constrained by expectations; This indicates the expected value calculation; Indicates the discount factor; For the reward function;

[0031] The safe operation requirements of reservoir scheduling are condensed into an expectation constraint, providing a unified mathematical starting point for subsequent constraint processing mechanisms and Lagrange methods.

[0032] Furthermore, S2 includes at least the following steps:

[0033] S2.1: Determine the probabilistic form of the mask;

[0034] Let the discrete action space be... ,in For the action dimension;

[0035] Given state Define the mask function:

[0036]

[0037] in, Indicates action index In state The following is feasible; This indicates that the action was blocked because it violated current water level, flow rate, or reservoir capacity constraints;

[0038] Let the log-odds ratio of the policy network output without a mask be . Then the logarithmic probability after adding the mask is written as:

[0039]

[0040] Among them, the agreement (In the implementation, a sufficiently small value is used for approximation), so the probability of the masked action being after softmax is strictly zero. Specifically, the strategy after adding the mask is:

[0041]

[0042] This is equivalent to changing the policy definition domain from Shrink to a subset of actionable elements

[0043]

[0044] Thus, the masking mechanism injects physical constraints into the action sampling process in a deterministic manner without changing the policy network structure;

[0045] S2.2: Mapping between action space and flow space;

[0046] In order to express the engineering water level and reservoir capacity constraints in the action mask, it is necessary to establish a mapping relationship between the action index and the physical outflow.

[0047] remember and These are the minimum and maximum feasible outbound flow rates (unit: m³). / s), the mapping function is:

[0048]

[0049] Indexing discrete actions Mapped to physical traffic ;

[0050] Accordingly, define normalized action variables. , used to represent "continuous positions" in the action space, and its relationship with physical flow is written as:

[0051]

[0052] in, For a monotonically increasing function, different For different linear or nonlinear mappings, given Next, define the inverse mapping from traffic to normalized actions:

[0053]

[0054] And further Quantized into discrete action indexes:

[0055]

[0056] in, Indicates rounding;

[0057] Therefore, any given flow constraint interval All can be mapped to action index ranges This is then transformed into a continuous "action window" in the mask vector;

[0058] S2.3: Mask interval calculation based on water balance;

[0059] The key to the masking mechanism lies in how to adjust the masking based on the current state. Calculate the allowed normalized action range This allows us to obtain the corresponding action index range. ;

[0060] Directly utilizing the water balance equation and water level constraints, for the outflow Provide upper and lower bound constraints;

[0061] On the one hand, to prevent the water level from exceeding the daily upper limit. Use the following formula to write out the condition that the final storage capacity does not exceed the limit storage capacity. Conditions:

[0062]

[0063] in, For time; Given the reservoir capacity corresponding to the upper limit water level, the minimum allowable outflow can be obtained from this formula:

[0064]

[0065] when hour, A positive value indicates that a larger outflow is needed to reduce the reservoir capacity; when... hour, If the outflow is negative, the outflow is allowed to be slightly less than the inflow to achieve water storage;

[0066] On the other hand, to prevent the storage capacity from falling below the daily minimum storage capacity. (corresponding to the lowest operating water level) ), needs to meet:

[0067]

[0068] Thus, the maximum allowed outflow is obtained:

[0069]

[0070] The feasible interval for outflow can be written as:

[0071]

[0072] in:

[0073]

[0074] Furthermore, by combining engineering constraints such as minimum ecological flow, shipping flow, and equipment capacity limits, the endpoints of the intermediate zone can be further tightened.

[0075] Finally, and Mapped to normalized action range and action index range and in Only actions within that interval are retained;

[0076] Thus, the masking mechanism provides a feasible domain for actions consistent with physical constraints at each time step based on water balance, achieving a seamless mapping of physical constraints, flow range, and action range.

[0077] Furthermore, S3 includes at least the following steps:

[0078] S3.1: Family of mapping functions and inverse functions;

[0079] Given the size of the discrete action space Record action index The corresponding normalized position is:

[0080]

[0081] Based on this, a class of mappings with a unified form is defined:

[0082]

[0083] in It is a strictly monotonically increasing function. It includes at least self-linear mappings, convex mappings, concave mappings, and logarithmic / exponential mappings;

[0084] For any given ,if only exist If the function is continuous and strictly monotonically increasing, then there exists an inverse function. Thus, the physical flow is expressed through the formula. Mapping and regressing to unify the action space;

[0085] This inverse mapping is particularly important in action masks, where the given upper and lower bounds of the traffic need to be passed through... Transformed into an action range, in Implemented in the middle;

[0086] S3.2: Resolution Analysis;

[0087] To quantitatively analyze the resolution differences of different mapping functions in various flow ranges, the concept of motion density is introduced;

[0088] remember For a given traffic range of interest, the range length is... ;

[0089] After mapping this interval to the normalized space, the number of available actions within the interval is approximately:

[0090]

[0091] in, They are respectively The corresponding normalized position;

[0092] Further define the average motion density:

[0093]

[0094] but The larger the value, the higher the action resolution within that flow range;

[0095] Furthermore, S4 includes at least the following steps:

[0096] Given dual variables In this case, define the modified value objective:

[0097]

[0098] And optimize using any policy gradient-based deep reinforcement learning algorithm (such as PPO). In this case, the policy update only needs to replace the immediate reward with the formula in the equation. The rest of the implementation remains unchanged;

[0099] On the other hand, after several rounds of policy updates, the average constraint cost is estimated based on the empirical trajectory under the current policy. And update the dual variables according to the formula:

[0100]

[0101] in, The dual learning rate; This represents the projection onto a non-negative number axis;

[0102] If the constraint cost of the current strategy is significantly higher than the threshold ,but Increase the size of the constraint to enhance its penalty strength in subsequent policy updates;

[0103] Conversely, if the constraint cost is much lower than the threshold, then Gradually reduce the rewards to free up more optimization space for the main task rewards;

[0104] In engineering implementation, to improve numerical stability, the constraint cost is normalized and the default portion is extracted;

[0105] remember For the environment at all times The original constraint cost is returned. As the normalization factor, If the default threshold is given, then the normalized cost and default metric are written as:

[0106]

[0107] Default metrics are used during training. Replacement original cost The inclusion of reward reconstruction and dual updates reduces the optimization difficulties caused by scale differences while keeping the CMDP objective structure unchanged.

[0108] Compared with the prior art, the beneficial effects of the present invention are:

[0109] 1. The Lagrange dual method of this invention shows significant advantages in handling security constraints. Compared with the corrective and penalized methods, the Lagrange method significantly improves the target optimization performance while ensuring constraint satisfaction by adaptively adjusting the constraint weights. Experimental results show that this method is superior to other methods in terms of constraint satisfaction and power generation, and the degree of constraint violation is reduced by more than 80%.

[0110] 2. This invention employs an action masking mechanism that effectively improves training stability through the synergistic effect of pre-defense and post-optimization. The complete rule mask can accurately define the safety boundary, reducing the violation rate from 21%-79% to zero, while accelerating policy convergence. Research has found that the completeness design of the mask rules is crucial. Basic rules may produce pseudo-feasible region problems in complex periods, while complete rules combined with physical constraints such as water balance can more accurately identify the action space.

[0111] 3. The nonlinear action mapping function used in this invention has a significant impact on policy learning efficiency. Concentrated action density in the low flow range (such as quadratic mapping) can accelerate the learning of key constraints, improve convergence stability, and achieve zero violations. However, excessive concentration (such as cubic mapping) may affect operational flexibility, while high flow density concentration (such as logarithmic mapping) significantly reduces policy performance. Quadratic mapping provides a better balance between accuracy and flexibility.

[0112] In summary, the improved Lagrange algorithm proposed in this invention, combining action masking and nonlinear mapping, provides a systematic solution for safety reinforcement learning in reservoir scheduling. This method, through a two-layer safety framework of "pre-emptive defense + post-emptive optimization," effectively improves scheduling performance while ensuring operational safety, offering a new technical approach to multi-objective water resource management problems. Attached Figure Description

[0113] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0114] Figure 1 This is a flowchart of the two-layer security reinforcement learning method framework of the present invention;

[0115] Figure 2 This is an evolution diagram of performance and constraint satisfaction during the training process of this invention;

[0116] Figure 3 This is a schematic diagram illustrating the internal operating mechanism of the Lagrange mechanism of this invention;

[0117] Figure 4 This is a schematic diagram showing the dynamic comparison of the training of the five nonlinear mapping functions of this invention;

[0118] Figure 5 This is a schematic diagram comparing the time series of water level and flow rate for the five mapping functions of this invention. Detailed Implementation

[0119] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0120] This invention is conducted from three levels:

[0121] First layer: Comparison of constraint handling mechanisms (RQ1). We compare three mainstream methods: (1) Correction-only method – after a violation, it forces correction to the safe boundary but does not provide a learning signal; (2) Penalty-based method – after a violation, it corrects the violation and applies a fixed weight penalty; (3) Theoretical method (PPO-Lagrangian) – based on the CMDP framework, it adaptively adjusts the constraint weights through Lagrangian duality. We conduct quantitative evaluation from four dimensions: safety (violation rate, violation severity), performance (average reward, power generation), training efficiency (number of convergence rounds), and deployability (hyperparameter tuning cost), to provide a basis for selection in engineering practice.

[0122] The second layer: Synergistic effect of the pre-emptive defense mechanism (RQ2). We designed a state-driven action masking mechanism to dynamically shield dangerous actions based on the current water level, flow rate, equipment capacity, and downstream carrying capacity, thus constructing the first line of defense. Ablation experiments (no mask, single-rule mask, and complete mask) were used to evaluate the impact of the mask on the violation rate, exploration efficiency, and convergence speed in the early stages of training. The synergistic effect of the mask and the posterior constraint reinforcement method (Lagrangian) was also analyzed to verify the effectiveness of the "pre-emptive defense + posterior optimization" two-layer safety framework.

[0123] Layer 3: Nonlinear Mapping-Assisted Safety Exploration (RQ3). We compare four action mapping functions (linear, logarithmic, quadratic, and cubic) to evaluate their safety and responsiveness in different flow ranges (ecological baseflow, routine scheduling, and emergency flood discharge), revealing the improvement effect of nonlinear mapping on violation rates in low flow ranges and its potential impact on responsiveness in high flow ranges, providing engineering guidance for action space design.

[0124] Please see Figure 1 This paper presents the proposed two-layer security reinforcement learning framework. The overall approach is as follows: Based on unified CMDP modeling, a physical mechanism-based pre-defense mechanism is constructed to filter and map policy outputs according to engineering constraints such as water level and outflow before action generation, avoiding decisions that significantly violate hydraulic boundary conditions. Simultaneously, a Lagrange duality-based posterior optimization mechanism is introduced to continuously monitor constraint costs and adaptively adjust penalty intensity during long-term training, ensuring that the learned policy meets long-term security requirements while guaranteeing scheduling efficiency. Within this overall framework, the security reinforcement learning method is systematically described from a CMDP perspective.

[0125] Specifically as follows:

[0126] A reinforcement learning method for reservoir scheduling safety based on the Lagrange method includes at least the following steps:

[0127] S1: Treat the reservoir scheduling problem as a sequential decision problem with security constraints, and use the Constrained Markov Decision Process (CMDP) to formally model the problem, representing the reservoir system as a seven-tuple;

[0128] S2: In the strategy sampling phase, a state-driven action mask mechanism is introduced. Before each decision, actions that obviously do not meet the physical constraints are dynamically filtered out based on the current reservoir state. This creates a "front-line defense" outside of strategy learning. The state-driven action mask mechanism calculates feasible outflow intervals based on the water balance equation and physical constraints. It is then transformed into action index intervals through action-flow mapping to filter out illegal actions during the strategy sampling phase.

[0129] S3: Perform nonlinear motion mapping and resolution design, and adopt a nonlinear motion mapping function within a unified framework. By adjusting The shape allows for different motion densities to be allocated across different flow ranges;

[0130] Focusing on constraint handling mechanisms, these mechanisms operate at the policy update level. Constraint information is fed back to the policy "after the fact" through penalties or dual variables. Constraint handling mechanisms include at least corrective methods, penalized methods, and Lagrange dual methods, and they address constraint costs during policy updates. Make different settings;

[0131] Corrective methods ensure safety but can lead to insufficient policy learning.

[0132] Punitive methods are easy to implement, but the choice of fixed punishments can affect learning outcomes.

[0133] The Lagrange dual method, by introducing dynamically adjusted dual variables, makes the handling of constraints more flexible and effective, and is suitable for complex decision-making problems.

[0134] The correction-only approach treats safety constraints as hard boundaries;

[0135] At each decision-making moment, the policy network first outputs a candidate action. The environment then predicts the water level after the action is performed based on system dynamics. ,like Exceeding the current allowable water level range Then, a projection operator is used to correct the actions to the feasible set:

[0136]

[0137] in, This is the mapping function from actions to physical flow; In the state The set of actions that satisfy water level and flow constraints; the actual actions executed are always... ,award The method does not explicitly include constraint penalties. While it guarantees hard safety at each step, the policy struggles to "sense" the location of constraint boundaries because violations are silently corrected, leading to insufficient learning of optimal behavior near the constraint region.

[0138] Penalty-based methods modify the reward function, adding constraint violations to the objective with fixed weights. Given a fixed penalty coefficient... Define the modified instant reward:

[0139]

[0140] The original CMDP problem is then transformed into an unconstrained MDP maximization problem:

[0141]

[0142] The advantage of the penalty-based method is that it can be directly trained using standard reinforcement learning algorithms (such as PPO), and constraint information is injected into the estimation of the value function and advantage function through the penalty term. However, fixed weights... The choice of weights lacks theoretical guidance: too small a weight is insufficient to suppress violations, while too large a weight will make the strategy overly conservative or even cause training failure. This contradiction is particularly prominent in the reservoir scheduling problem with multi-source heterogeneous constraints.

[0143] The Lagrangian-based approach directly introduces learnable dual variables (Lagrangian multipliers) into the dual space of CMDP. Construct the Lagrange function:

[0144]

[0145] Consider the following saddle point problem:

[0146]

[0147] Expanding the above equation into the form of trajectory expectation, we get:

[0148]

[0149] in For the trajectory. It can be seen that when the dual variable... When fixed, maximizing is equivalent to the modified reward:

[0150]

[0151] Solve the unconstrained MDP; while updating when the policy is fixed. This is equivalent to increasing the penalty intensity when constraints are violated and decreasing the penalty intensity when constraints are abundant. In practice, an alternating update strategy is typically used.

[0152]

[0153] in The outer iteration number, For the dual learning rate, For the first The estimated average constraint cost under the round-robin strategy, [·] This represents the projection onto the non-negative semi-axis. Compared to a fixed penalty method, automatic adjustment... Finding a balance between safety and benefits provides a more theoretically sound solution for complex CMDPs such as reservoir scheduling.

[0154] S4: At the policy optimization level, a constraint reinforcement method based on Lagrange duality is adopted to ensure that the policy satisfies the CMDP constraint in the long-term scale, thus transforming the constraint optimization problem of CMDP into a saddle point search problem in the policy space and dual space.

[0155] S1 includes at least the following steps:

[0156] Represent the reservoir system as a 7-tuple:

[0157]

[0158] in: For state space, Indicates time The system status; For the action space, Indicates the agent at time... The selected scheduling decision; As the state transition kernel, it describes the state transition process during the execution of actions. The system after Evolved to The probability of; This is an instant reward function used to characterize the comprehensive benefits of power generation, flood control, ecology, and shipping. The constraint cost function is used to quantify the degree of violation of safety constraints such as water level and flow rate; The initial state distribution; Discount factor; The upper bound of the allowable long-term average constraint cost is used to characterize the acceptable level of security risk.

[0159] In the specific implementation, state It should include at least the current water level, inflow rate, water forecast for the next few days, dynamic water level upper and lower limits for the corresponding time period, and date code, as shown in the following formula:

[0160]

[0161] in, Current water level (in meters); Current inbound flow rate (unit: m³) / s); For the inflow prediction vector for the next few days; and These are the time-varying lowest and highest operating water levels, respectively. Normalized date encoding, used to characterize seasonality;

[0162] The evolution of reservoir capacity-water level follows the principle of mass conservation, denoted as... For a moment Storage capacity (unit: m) ), Outbound flow rate (unit: m³) / s), Let the scheduling time step be (in seconds), then the water balance equation is:

[0163]

[0164] The monotonic function between reservoir capacity and water level obtained from topographic surveys Association, that is:

[0165]

[0166] in, As the inverse function of the reservoir capacity-water level curve, it ensures that environmental dynamics strictly obey water conservation and topographic conditions, providing a physical basis for subsequent constraint design;

[0167] Under the CMDP framework, strategy The optimization objective is to maximize the cumulative discount reward while satisfying long-term constraints.

[0168]

[0169] in, The length of the round; Cost is constrained by expectations; This indicates the expected value calculation; Indicates the discount factor; For the reward function;

[0170] The safe operation requirements of reservoir scheduling are condensed into an expectation constraint, providing a unified mathematical starting point for subsequent constraint processing mechanisms and Lagrange methods.

[0171] S2 includes at least the following steps:

[0172] S2.1: Determine the probabilistic form of the mask;

[0173] Let the discrete action space be... ,in For the action dimension;

[0174] Given state Define the mask function:

[0175]

[0176] in, Indicates action index In state The following is feasible; This indicates that the action was blocked because it violated current water level, flow rate, or reservoir capacity constraints;

[0177] Let the log-probability of the policy network output without a mask be... Then the logarithmic probability after adding the mask is written as:

[0178]

[0179] Among them, the agreement (In the implementation, a sufficiently small value is used for approximation), so the probability of the masked action being after softmax is strictly zero. Specifically, the strategy after adding the mask is:

[0180]

[0181] This is equivalent to changing the policy definition domain from Shrink to a subset of actionable elements

[0182]

[0183] Thus, the masking mechanism injects physical constraints into the action sampling process in a deterministic manner without changing the policy network structure;

[0184] S2.2: Mapping between action space and flow space;

[0185] In order to express the engineering water level and reservoir capacity constraints in the action mask, it is necessary to establish a mapping relationship between the action index and the physical outflow.

[0186] remember and These are the minimum and maximum feasible outbound flow rates (unit: m³). / s), the mapping function is:

[0187]

[0188] Indexing discrete actions Mapped to physical traffic ;

[0189] Accordingly, define normalized action variables. , used to represent "continuous positions" in the action space, and its relationship with physical flow is written as:

[0190]

[0191] in, For a monotonically increasing function, different For different linear or nonlinear mappings, given Next, define the inverse mapping from traffic to normalized actions:

[0192]

[0193] And further Quantized into discrete action indexes:

[0194]

[0195] in, Indicates rounding;

[0196] Therefore, any given flow constraint interval All can be mapped to action index ranges This is then transformed into a continuous "action window" in the mask vector;

[0197] S2.3: Mask interval calculation based on water balance;

[0198] The key to the masking mechanism lies in how to adjust the masking based on the current state. Calculate the allowed normalized action range This allows us to obtain the corresponding action index range. ;

[0199] Directly utilizing the water balance equation and water level constraints, for the outflow Provide upper and lower bound constraints;

[0200] On the one hand, to prevent the water level from exceeding the daily upper limit. Use the following formula to write out the condition that the final storage capacity does not exceed the limit storage capacity. Conditions:

[0201]

[0202] in, For time; Given the reservoir capacity corresponding to the upper limit water level, the minimum allowable outflow can be obtained from this formula:

[0203]

[0204] when hour, A positive value indicates that a larger outflow is needed to reduce the reservoir capacity; when... hour, If the outflow is negative, the outflow is allowed to be slightly less than the inflow to achieve water storage;

[0205] On the other hand, to prevent the storage capacity from falling below the daily minimum storage capacity. (corresponding to the lowest operating water level) ), needs to meet:

[0206]

[0207] Thus, the maximum allowed outflow is obtained:

[0208]

[0209] The feasible interval for outflow can be written as:

[0210]

[0211] in:

[0212]

[0213] Furthermore, by combining engineering constraints such as minimum ecological flow, shipping flow, and equipment capacity limits, the endpoints of the intermediate zone can be further tightened.

[0214] Finally, and Mapped to normalized action range and action index range and in Only actions within that interval are retained;

[0215] Thus, the masking mechanism provides a feasible domain for actions consistent with physical constraints at each time step based on water balance, achieving a seamless mapping of physical constraints, flow range, and action range.

[0216] S3 includes at least the following steps:

[0217] S3.1: Family of mapping functions and inverse functions;

[0218] Given the size of the discrete action space Record action index The corresponding normalized position is:

[0219]

[0220] Based on this, a class of mappings with a unified form is defined:

[0221]

[0222] in It is a strictly monotonically increasing function. It includes at least self-linear mappings, convex mappings, concave mappings, and logarithmic / exponential mappings;

[0223] Linear mapping: This corresponds to a uniform resolution across the entire range;

[0224] Convex mapping: such as or Allocate higher density in low-flow segments and compress resolution in high-flow segments;

[0225] Concave mapping: such as Higher density is allocated in high-flow segments, and resolution is compressed in low-flow segments;

[0226] Logarithmic / exponential mapping: Adjusting the density balance between low and medium flow rates and high flow rates using logarithmic or exponential functions.

[0227] For any given ,if only exist If the function is continuous and strictly monotonically increasing, then there exists an inverse function. Thus, the physical flow is expressed through the formula. Mapping and regressing to unify the action space;

[0228] This inverse mapping is particularly important in action masks, where the given upper and lower bounds of the traffic need to be passed through... Transformed into an action range, in Implemented in the middle;

[0229] S3.2: Resolution Analysis;

[0230] To quantitatively analyze the resolution differences of different mapping functions in various flow ranges, the concept of motion density is introduced;

[0231] remember For a given traffic range of interest, the range length is... ;

[0232] After mapping this interval to the normalized space, the number of available actions within the interval is approximately:

[0233]

[0234] in, They are respectively The corresponding normalized position;

[0235] Further define the average motion density:

[0236]

[0237] but The larger the value, the higher the action resolution within that flow range;

[0238] With low flow segment For example ( (for a preset interval width), different mapping functions correspond to The analytical expression is obtained by differentiation. For linear mappings... ,have and The relationship is linear, and the action density is constant throughout the entire interval; for convex mappings... or , exist When the flow rate is low, the change is slow, corresponding to a higher action density in the low flow rate segment, while the high flow rate segment is compressed; concave mapping exhibits the opposite characteristics. In this way, the trade-off between "low flow rate accuracy – high flow rate accuracy" of different mapping schemes can be quantitatively analyzed during the method design stage, without relying on specific experimental configurations.

[0239] From the perspective of the physical characteristics of reservoir operation, low-flow sections are typically closely related to ecological flow assurance and power generation efficiency during the dry season, while high-flow sections are closely linked to flood control safety. Therefore, a natural design principle is that when the task focuses more on the fine-grained control of low-flow sections, convex mapping should be prioritized to improve efficiency. Action density within a given interval; when the task focuses more on flood peak shaving and emergency flood discharge in high-flow sections, concave or near-linear mappings are used to improve the resolution of high-flow areas; for scenarios that simultaneously consider multiple objectives, segmented or hybrid mappings can be considered, using different mappings in different flow intervals. This allows for a more balanced distribution of actions across the entire flow range.

[0240] S4 includes at least the following steps:

[0241] Given dual variables In this case, define the modified value objective:

[0242]

[0243] And it is optimized using any policy gradient-based deep reinforcement learning algorithm (PPO is used in this embodiment). At this time, the policy update only needs to replace the immediate reward with the formula. The rest of the implementation remains unchanged;

[0244] On the other hand, after several rounds of policy updates, the average constraint cost is estimated based on the empirical trajectory under the current policy. And update the dual variables according to the formula:

[0245]

[0246] in, The dual learning rate; This represents the projection onto a non-negative number axis;

[0247] If the constraint cost of the current strategy is significantly higher than the threshold ,but Increase the size of the constraint to enhance its penalty strength in subsequent policy updates;

[0248] Conversely, if the constraint cost is much lower than the threshold, then Gradually reduce the rewards to free up more optimization space for the main task rewards;

[0249] In engineering implementation, to improve numerical stability, the constraint cost is normalized and the default portion is extracted;

[0250] remember For the environment at all times The original constraint cost is returned. As the normalization factor, If the default threshold is given, then the normalized cost and default metric are written as:

[0251]

[0252] Default metrics are used during training. Replacement original cost The inclusion of reward reconstruction and dual updates reduces the optimization difficulties caused by scale differences while keeping the CMDP objective structure unchanged.

[0253] The subsequent improved algorithm pseudocode, when using PPO, is the improved PPO-Lagrangian algorithm pseudocode;

[0254] To clearly demonstrate how the improved Lagrange duality method is embedded in the PPO training process, the algorithm pseudocode is provided below. This pseudocode is based on the standard PPO main structure, with extensions only in the reward reconstruction and dual variable update sections.

[0255] Algorithm 1: Improved PPO-Lagrangian constrained reinforcement learning algorithm;

[0256] Input: CMDP environment Strategy-value function parameters Initial values ​​of dual variables Constraint threshold Dual learning rate Normalization factor Default threshold Dual update interval Maximum dual upper bound (Optional).

[0257] Initialization: Set the environment to the given initial value and clear the experience buffer.

[0258] Repeat until convergence or the maximum number of iterations is reached:

[0259] Use the current strategy Interact with the environment, collect several trajectories, and obtain a sequence. ,in The constraint cost for each step.

[0260] Cost constraints at each step Perform normalization and default measurement calculations:

[0261]

[0262] Reconstructing Instant Rewards with Default Measurement:

[0263]

[0264] Will As the reward input, a loss function is constructed according to the standard PPO process (advantage function estimation, value function reward, pruning strategy loss, etc.). And on Perform multi-step gradient updates.

[0265] Calculate the average default rate for the current batch. ,in This represents the number of time steps in the batch.

[0266] If the current iteration round satisfies the dual update condition (e.g., every...), (Updated once per cycle), then according to

[0267]

[0268] Perform dual variable updates, and truncate and project as needed:

[0269]

[0270] Record and monitor changes in constraint costs, default measures, and dual variables, such as , and This is used for subsequent analysis and convergence diagnosis.

[0271] Based on the above embodiments, the corresponding experimental verification was carried out:

[0272] This paper addresses the multi-year optimal scheduling problem of a single reservoir by constructing a simulation environment for the Three Gorges Reservoir and designing an experimental scheme. The goal is to maximize the cumulative power generation revenue over multiple years using a reinforcement learning method, while satisfying multiple constraints including operational safety, ecological considerations, and navigation requirements. The scheduling time step is daily, and the decision-making process must simultaneously address the coupling relationship between the annual transition between wet and dry seasons and cross-year water volume management.

[0273] The experiment uses the Three Gorges Reservoir as a case study for modeling and verification. Located on the main stream of the Yangtze River, the Three Gorges Reservoir has an installed capacity of 22,500 MW, a maximum discharge capacity of 31,187 m³ / s, a minimum outflow of 5,500 m³ / s, a dead water level of 145 m, a normal storage water level of 175 m, and a flood control limit water level dynamically adjusted between 145 and 175 m according to operating procedures. The reservoir capacity-water level relationship was modeled using piecewise fitting based on engineering measurement data to ensure monotonic reversibility. Power generation output is determined by both head and outflow; unit efficiency curves and guaranteed output parameters are derived from publicly available literature and engineering data. Hydrological data are sourced from the "Hydrological Yearbook" compiled by the Yangtze River Water Resources Commission, selecting a 60-year daily inflow sequence from 1960 to 2020 to construct a multi-year scheduling environment. All hydrological and operational data underwent quality control and engineering verification.

[0274] In terms of constraint design, this invention characterizes the key operational requirements of the Three Gorges Reservoir within the CMDP framework. Safety constraints mainly include ensuring the water level does not exceed the dynamic operating upper and lower limits and that the outflow does not exceed the flood discharge and generating unit safety capabilities; ecological constraints are addressed by setting a lower limit for the ecological baseflow (6000m³). 3 / s); Shipping constraints are imposed by setting a lower limit for the safe flow of traffic (6500m). 3 / s).

[0275] Experimental Design:

[0276] To systematically evaluate the impact of different constraint handling mechanisms, pre-defense strategies, and action space designs on the long-term scheduling performance of the Three Gorges Reservoir, this invention adopts a phased experimental design. All experiments are conducted in the same Three Gorges Reservoir simulation environment, with consistent environmental dynamics, state space, action space, and constraint definitions. Controlled variable experiments are only conducted on the constraint handling module, action mask, and action mapping methods.

[0277] Data partitioning. Based on the annual flood and drought characteristics of the 60-year historical inflow from 1960 to 2020, each year was divided into five levels: extremely dry years, low-water years, normal-water years, high-water years, and extremely high-water years. Typical years were selected based on 95% extremely dry years, 75% low-water years, 50% normal-water years, 25% high-water years, and 5% extremely high-water years. These years were isolated and formed into an independent evaluation set to test the generalization performance of the strategy; the remaining years constituted the training set for strategy learning. All comparative experiments used consistent inflow sequences and initial water level settings to ensure comparability of results.

[0278] Experimental setup:

[0279] The overall design consists of three main experimental sections:

[0280] Experiment A:

[0281] Comparison of constraint handling mechanisms. Under the same Three Gorges environment and linear action-flow mapping, three typical constraint handling strategies are compared. Specific configurations are as follows:

[0282] A1 (Correction): Corrective approach, which only corrects actions after the fact and does not introduce punishment or dual variables.

[0283] A2 (Penalty): A penalized method that applies a fixed-weight penalty to the reward function. ).

[0284] A3 (Lagrangian): The PPO-Lagrangian method introduces learnable Lagrange multipliers to achieve adaptive constraint handling.

[0285] Experiment B:

[0286] Action masking mechanism evaluation. Under the Lagrangian dual framework (A3), which performed well in Experiment A, state-driven action masks of varying intensities were introduced. The specific configurations are as follows:

[0287] B1 (NoMask): No mask, preserving the complete 99-dimensional motion space as the baseline.

[0288] B2 (BaseMask): Base mask, applying the specific scheduling procedures of the Three Gorges Reservoir from August to October, and does not include general physical constraints.

[0289] B3 (FullMask): A complete mask that overlays six types of general physical constraints on top of B2, including upper / lower reservoir capacity control, water level rise and fall rate constraints, dry season flow multiple constraints, outflow change rate constraints, and water level over-limit protection.

[0290] Experiment C:

[0291] Non-linear action mapping comparison. Within the unified framework of "PPO-Lagrangian + complete rule mask" (A3+B3), different action-traffic mapping functions are replaced. Specific configuration is as follows:

[0292] C1 (Linear): Linear mapping This serves as the baseline configuration.

[0293] C2 (Square): Square Mapping The density is concentrated in the low flow range.

[0294] C3 (Cubic): Cubic mapping This further strengthens low flow density.

[0295] C4 (Exp): Exponential mapping, which is less concentrated in density in low-flow intervals compared to square mapping.

[0296] C5 (Log): Logarithmic mapping, which concentrates the action density in the high-flow range as a benchmark.

[0297] In the three experiments described above, the common hyperparameter settings remained unchanged. Each experimental configuration was run five times with a fixed set of random seeds. After training, the final convergent strategy was validated using a 5-year inflow dataset independent of the training set.

[0298] Experiment A: Comparative Analysis of Constraint Handling Mechanisms

[0299] Figure 2 The study demonstrates the performance and constraint satisfaction evolution of three constraint handling mechanisms during the training process, revealing their distinctly different learning trajectories. Figure 2 (a) shows that the correction mechanism (A1, pink curve) exhibits slow convergence in reward evolution, gradually improving from an initial value of approximately -6000 to a final convergence of approximately -300 (Table 1: -299.46 ± 202.49), but consistently performs the worst throughout the training process. This phenomenon stems from the passive nature of the correction mechanism: when the agent's chosen action triggers a constraint violation, the correction mechanism only truncates the action but does not feed the violation information back into the gradient update of the policy network. Therefore, the agent repeatedly explores near the boundary of the exploration space but fails to learn an effective avoidance strategy from the violation experience. Figure 2 The overall water level constraint violation evolution curve in (c) illustrates this problem more clearly. The violation amount of the A1 method started to improve from about -45, but eventually stabilized at around -25 (Table 1 constraint violation reward: -24.98±3.09), indicating limited improvement. Figure 2 (e) shows that, in terms of air traffic assurance satisfaction, the A1 method has been maintained between -400 and -450 for a long time, and finally reached -384.04±48.57, which shows that the learning efficiency is insufficient, indicating that the correction mechanism has systematic deficiencies in dealing with hard constraints.

[0300] The penalty mechanism attempts to establish a direct link between constraint violation and policy optimization by introducing a penalty term into the reward function. Figure 2 The six subplots show that the overall performance of method A2 (green curve) falls between that of A1 and A3. Figure 2 (a) shows that the reward curve of A2 fluctuates greatly in the early stage of training and eventually converges to about -720 (Table 1: -723.34±283.34), which is worse than A1. This is because the fixed penalty coefficient leads to training instability. Figure 2(c) shows that in the overall water level constraint violation evolution, the A2 method improved from about -45 to about -14 (Table 1 constraint violation reward: -14.34±2.86), which is a significant improvement compared to A1, with an improvement of 42.6%. Figure 2 (e) shows that the air traffic assurance satisfaction improved from -384.04 to -341.50 (an improvement of 11.1%). However, the penalty mechanism faces an inherent trade-off: too small a penalty coefficient cannot effectively suppress constraint violations, while too large a penalty coefficient will lead to an overly conservative strategy, sacrificing the optimization space of the task objective. This instability is reflected in the large standard deviation of the reward in the A2 method (±283.34), reflecting the increased gradient estimation noise caused by the scale distortion of the reward signal after the introduction of the penalty term.

[0301] The Lagrange mechanism transforms a constrained optimization problem into an unconstrained saddle point optimization process by introducing dynamically adjusted Lagrange multipliers λ. Figure 2 The blue curve (A3 method) shows optimal evolutionary characteristics across all six index dimensions. Figure 2 (a) shows that although the reward curve of A3 fluctuates in the early stage of training, it rises continuously from the 100th episode and eventually converges to approximately -294 (Table 1: -293.69±239.69), which is the optimal value among the three methods. Figure 2 (b) shows that, in terms of total power generation, the A3 method ultimately reached a level of approximately 398.52, an improvement of 4.9% over A1 and 3.8% over A2. The most significant advantage lies in constraint satisfaction. Figure 2 (c) shows that the overall water level constraint violation of A3 improved rapidly from -45 and converged to near 0 after the 200th episode (Table 1 constraint violation reward: -2.85±1.68, an improvement of 88.6% over A1 and 80.1% over A2). Figure 2 (d) shows that the power guarantee bonus is -26.28±24.22, which is significantly better than A1's -98.92 and A2's -90.02. Figure 2 (e) shows that the air traffic assurance bonus is -115.50±75.42, which is 69.9% better than A1 (-384.04) and 66.2% better than A2 (-341.50), demonstrating the excellent ability to handle complex dynamic constraints. Figure 2 (f) shows that the ecological guarantee reward is -0.49±0.71, which is also the optimal value.

[0302] Figure 3(a) illustrates the evolution of the λ value, revealing the adaptive nature of the Lagrange mechanism. The λ value starts at approximately 1.8, slowly increasing to around 2.5 during the initial exploration phase (0-50 episodes, yellow area). In the growth phase (50-250 episodes, blue area), the λ value accelerates to approximately 6.0. In the stable phase (250-300 episodes, green area), the λ value stabilizes between 6.0 and 6.5, eventually converging to 6.5316 ± 0.3699 (Table 1). This evolution reflects the system's dynamic equilibrium mechanism—when constraint violations are frequent, the λ value increases to reinforce the importance of constraint satisfaction; when constraints are effectively controlled, the λ value tends to stabilize, and the system enters the fine-tuning phase of policy optimization. Table 1 shows that the A3 method achieves optimal performance across all constraint dimensions, and combined with the constraint cost (0.2124 ± 0.0217) in Table 1, it forms an effective balance between constraint satisfaction and objective optimization.

[0303] Figure 3 The internal workings of the Lagrange mechanism are specifically demonstrated, revealing that its superiority stems from the co-evolution between the dual variable λ and the original strategy. Figure 3 (a) shows that the evolution curve of the λ value exhibits three distinct stages. In the first stage (0-50 episodes, marked in yellow as "Initial Exploration"), the λ value slowly increases from an initial value of approximately 1.8 to around 2.5, with a growth rate of approximately 0.014 / episode. In the second stage (50-250 episodes, marked in blue as "Growth Phase"), the λ value accelerates, climbing from 2.5 to 6.0, with the growth rate increasing to approximately 0.018 / episode. In the third stage (250-300 episodes, marked in green as "Stable Phase"), the λ value fluctuates slightly between 6.0 and 6.5 before stabilizing, converging to a steady-state value of 6.5316 ± 0.3699. This evolutionary trajectory is consistent with... Figure 3 (b) shows a precise correspondence between the changes in constraint cost (Constraint Cost C(π), the solid blue line). In the first 50 episodes of training, the constraint cost steadily decreases from approximately 1.1 but remains at a relatively high level (approximately 1.0), with the slow increase in λ reflecting the system's initial awareness of constraint violations. As training enters the growth phase (50-250 episodes), the constraint cost decreases rapidly to the 0.5-0.6 range, while the rapid increase in λ indicates that the system is strengthening its focus on constraint satisfaction. This creates a negative feedback loop of decreasing constraint cost and increasing λ.

[0304] Figure 3(b) shows that the penalty term λ·c(s,a), represented by the orange dashed line, exhibits an inverted U-shaped curve that first rises and then falls. In the initial exploration phase (0-50 episodes), the penalty term rapidly increases from approximately 1.5. In the growth phase (50-150 episodes), the penalty term reaches a peak of approximately 3.5, reflecting that constraint violations and the λ value are both at high levels, indicating that the system applies the strongest penalty signal to violations. Subsequently (150-300 episodes), as the constraint cost continues to decrease (the blue solid line decreases from 0.8 to 0.3), the penalty term also decreases to around 1.5. This pattern of rising and then falling reveals the core mechanism of the Lagrange method: it does not simply pursue zero violation, but rather seeks the Pareto optimal point between constraint satisfaction and objective optimization by dynamically adjusting the penalty intensity. When the constraint cost is high (e.g., ...), the penalty term increases significantly. Figure 3 (b) As shown in the first 150 episodes), the system strengthens safety constraints by increasing the λ value and penalty term. After the constraint cost decreases (after 150 episodes), although the λ value continues to increase ( Figure 3 (a) shows that λ increases from 4.5 to 6.5), but since c(s,a) itself has decreased significantly, the penalty term λ·c(s,a) decreases instead, allowing the policy network to focus more on optimizing the task objective. This adaptive adjustment mechanism avoids the over-conservatism problem common in penalty mechanisms.

[0305] Starting from the 250th episode, the system enters a stable convergence phase (green area). Figure 3 (b) shows that the constraint cost stabilizes in the range of 0.2-0.3, and the standard deviation decreases significantly, indicating that the agent has established a stable safety margin. The penalty term stabilizes in the range of 1.0-1.5, maintaining a moderate constraint pressure. Figure 3 (a) shows that the λ value continues to be fine-tuned from 6.0 to 6.5 during this stage, and this delayed fine-tuning characteristic demonstrates the advantages of the Lagrange mechanism. The policy network first learns a coarse safety boundary during the growth phase (50-250 episodes), corresponding to... Figure 2 (c) shows a rapid improvement in overall constraint violation (from -40 to -5). Subsequently, the λ value continues to be fine-tuned during the steady-state phase (250-300 episodes) to optimize the accuracy of constraint satisfaction. Figure 2 In (c), the violation curve eventually converges to near zero. This phased learning model significantly reduces training complexity, decouples policy optimization from constraint satisfaction, and avoids the gradient conflict problem common in penalty mechanisms. (Comparison) Figure 3The two curves in (b) show that the decrease of the constraint cost C(π) is monotonically and steadily (from 1.1 to 0.3), while the penalty term λ·c(s,a) goes through a complete cycle of rising-peaking-falling. This difference is a direct reflection of the dynamic adjustment capability of the Lagrange mechanism.

[0306] Table 1 Summary of Training Performance Metrics

[0307]

[0308] Experiment B: Comparative Analysis of Action Masking Mechanisms

[0309] After Experiment A established the superiority of the Lagrange duality method in constraint handling, further improving policy security and training efficiency became the focus of the next research step. Experiment B focused on the Action Mask mechanism—a pre-emptive defense strategy that intelligently prunes the action space during the policy sampling phase. Unlike the post-feedback constraint handling in Experiment A, the Action Mask dynamically filters out actions that clearly violate physical constraints before each decision step using state-driven rules, reducing infeasible explorations during the sampling phase. This experiment systematically evaluates the impact of the masking mechanism on action space constraints, violation prevention, and decision quality at different scheduling stages by comparing three mask configurations—no mask (B1), basic rule mask (B2), and full rule mask (B3). The research objective is to reveal the intrinsic mechanism of the synergistic effect between pre-emptive defense and posterior optimization.

[0310] Experiment B designed three action mask configurations within the Lagrange duality framework. B1 (NoMask) disables the masking mechanism, maintaining the complete 99-dimensional action space, serving as a baseline control. B2 (WaterMask) enables basic masking rules based on the Three Gorges Dam operation procedures, dynamically adjusting the feasible action range according to the current water level and seasonal constraints. B3 (FullMask) superimposes general physical rules on top of B2, including reservoir capacity control based on water balance, low-limit reservoir capacity protection, and water level fluctuation rate constraints, forming the most stringent forward defense system. The three configurations are completely consistent in constraint processing (all using Lagrange duality), training hyperparameters, and evaluation procedures; the only difference lies in the mask strength during the action sampling phase.

[0311] The seasonal nature of reservoir operation determines that the constraints faced at different times vary significantly. This experiment divides the annual operation cycle into three key periods. The drawdown period (May 1st - June 10th, corresponding to periods 10-16) primarily aims to empty the reservoir, requiring a significant outflow to lower the water level from its high point to the flood control limit. During this period, there is a dual constraint of rapid flood discharge and avoiding excessive discharge. The flood season (June 11th - September 10th, corresponding to periods 17-27) is a critical period for flood control, requiring the water level to be maintained near the flood control limit while dealing with potential flood peaks, resulting in the greatest constraint pressure. The storage period (September 11th - October 31st, corresponding to periods 28-36) requires raising the water level from the flood control limit to the normal storage level to ensure power generation during the dry season. During this period, rapid water storage and ensuring downstream flow become the main contradiction.

[0312] Table 2 summarizes the key performance indicators of the three masking configurations at different times. All data is based on actual operation records during the evaluation phase. This table clearly demonstrates how the masking mechanism achieves precise control over violations at different scheduling periods by dynamically adjusting the action space.

[0313] Experimental results show that the intensity of action space constraints exhibits significant dynamic changes with the scheduling period. During the drawdown period, B3 compresses the available actions from 99 dimensions to an average of 18.32 dimensions (compression of 81.5%), during the flood season to 20.66 dimensions (compression of 79.1%), and during the water storage period to 39.98 dimensions (compression of 59.6%). This differentiated compression rate accurately reflects the constraint tension during different hydrological windows. Regarding violation prevention, the three configurations show a clear progressive pattern. The violation rate of B1 increases significantly with the increase in constraint difficulty across the three periods (from 21.46% to 78.82%). Although B2 significantly reduces the violation rate to single digits during the flood season and water storage period, it exhibits an abnormally high violation rate of 60.49% during the drawdown period. This phenomenon reveals the limitations of static procedure masks in dealing with highly dynamic constraints—namely, over-reliance on current state judgments while ignoring temporal water balance, which easily leads to the construction of pseudo-feasible regions. In contrast, B3 achieved zero violations and zero agent-environment mean difference across all periods, demonstrating the absolute security of the complete rule mask in handling physical constraints. Particularly during the most challenging water storage period, B1 suffered from nearly 80% action violations due to its inability to balance rapid water storage with flow assurance. B3, however, successfully guided the strategy to operate safely within a limited but sufficient range of actionable actions by precisely shrinking the action space to 39.98 dimensions, showcasing the core value of a proactive defense mechanism under extreme conditions.

[0314] Table 2 Comparison of Action Space and Violation Characteristics in Different Periods

[0315]

[0316] Experiment C: Analysis of Nonlinear Action Mapping Mechanism

[0317] Building upon Experiment A's establishment of the Lagrange duality mechanism and Experiment B's verification of the effectiveness of the complete rule mask, Experiment C focuses on the third dimension of action space design: the impact of nonlinear action mapping functions. Traditional reinforcement learning methods typically employ linear mapping to convert discrete action indices into physical flows, resulting in a uniform distribution of available actions across the flow space. However, the actual needs of reservoir scheduling exhibit significant asymmetry across different flow ranges. In the low flow range (0-15 km³ / s), precise control is required for ecological baseflow and navigation safety, with even minor deviations potentially leading to constraint violations. In the medium flow range (15-35 km³ / s), power generation scheduling needs to balance reservoir capacity changes with stable output. In the high flow range (>35 km³ / s), rapid response is required for flood control and discharge, but the tolerance for control precision is relatively high. This asymmetry raises research questions regarding nonlinear mapping functions: can the performance of the strategy be further improved while ensuring safety by reallocating action density?

[0318] Experiment C designed five nonlinear mapping functions for comparison with a linear benchmark. C1 uses a linear mapping as the benchmark, with actions uniformly distributed in the flow space. C2 uses a quadratic mapping, concentrating action density in the low flow range to enhance fine-grained control over ecological and shipping constraints. C3 uses a cubic mapping, further strengthening the density concentration effect in the low flow range. C4 uses an exponential mapping, utilizing the rapid growth characteristic of the exponential function to retain more action options in the high flow range. C5 uses a logarithmic mapping, utilizing the compression characteristic of the logarithmic function to achieve an extremely high-density action distribution in the low flow range. All mapping functions are based on a complete rule mask (B3) and a Lagrange duality mechanism (A3) to ensure fairness in the comparison. The experiment was trained using five years of historical inflow data from the Three Gorges Reservoir (2012-2016), with each configuration trained for 300 episodes, and the generalization performance of the final policy was tested on the same evaluation dataset.

[0319] Figure 4 The evolution of the learning curves of five mapping functions during the training process is shown, revealing the mechanism by which mapping shaping affects the convergence properties of the policy. Figure 5The reward curves in (a) show that different mapping functions exhibit distinctly different learning trajectories. C2 (square mapping) and C3 (cubic mapping) show a rapid upward trend in the early stages of training, breaking through the -2000 reward threshold within 50 rounds and stabilizing in the high reward range of -500 to 0 between 100 and 150 rounds. In contrast, the learning curve of C1 (linear mapping) is flatter, with a significantly slower convergence speed than C2 and C3, and the final reward stabilizes at around -1000. C4 (exponential mapping) performs in between, showing a moderate convergence speed. The learning curve of C5 (logarithmic mapping) remains consistently low throughout the training process, hovering in the range of -5000 to -6000, indicating that the agent faces severe learning obstacles under this configuration.

[0320] Table 3 statistically quantifies the average performance in the later stages of training (the last 10 rounds), further confirming the above observations. The Episode Return metric shows that C2 ranks first with a mean of 371.85±50.85, followed closely by C3 (364.05±58.94). The standard deviations of both are significantly smaller than other configurations, indicating high convergence stability. C1's return is 244.49±124.31, with a standard deviation of 124.31, reflecting the high volatility of the strategy under linear mapping. C5's return is only 74.79±99.84, not only having the lowest mean but also a standard deviation close to the mean itself, indicating extreme instability during training. Looking at the sub-indicators, C3 achieves the highest value (413.43±29.80) in power generation reward, indicating that cubic mapping has potential in optimizing economic objectives. However, the penalty for ecological security in C3 (-0.57±1.19) is slightly higher than that in C2 (-0.08±0.14), reflecting the potential side effects of excessive concentration in low-flow intervals. The shipping security indicators show a clear divergence: the penalties for C2 and C3 are -30.14±25.88 and -38.79±32.11, respectively, significantly better than C1's -113.80±80.88 and C5's -221.46±63.67. This indicates that a moderate concentration of low-flow interval density helps meet the minimum shipping flow requirements, but excessive compression (such as logarithmic mapping) hinders the agent from learning an effective balance strategy.

[0321] Table 4 shows the final performance of the five mapping functions on independent evaluation sets, revealing the complexity of translating training advantages into practical applications. In terms of total power generation, C2 ranks first with 420.931 billion kWh, a 0.54% improvement over the linear benchmark C1 (418.676 billion kWh). C3 ranks second with 420.307 billion kWh, only 0.15% lower than C2. C4's power generation (418.443 billion kWh) is close to C1's, while C5 lags significantly behind (411.408 billion kWh), 1.74% lower than C1. This ranking is largely consistent with the reward ranking during the training phase, but the improvement is relatively moderate. This indicates that, with the complete rule mask ensuring zero violations, the gains from mapping optimization are mainly reflected in the fine-tuning of economic objectives. A more crucial finding is the divergence in security metrics. The violation rate data shows that C2, C3, and C4 all achieved zero violations (0.00%), while C1 had a slight violation of 0.05%, and C5 had a significant violation of 0.88%. This reveals an important pattern: extreme density compression of logarithmic maps not only failed to improve security, but also weakened the robustness of the policy under boundary conditions.

[0322] The guarantee rate index further reveals the impact of mapping functions on soft constraint satisfaction. C1, C2, and C4 all reach or approach 100% in the three guarantee rates of power generation, shipping, and ecology, indicating that these three mappings have good generalization ability in multi-objective balance. C3 shows a slight decrease of 0.05% in the shipping guarantee rate (99.95%), but still remains at an extremely high level. In contrast, the guarantee rates of C5 drop to 97.75%, 97.75%, and 99.56%, respectively. Although the absolute levels are still high, the relative decrease is significant, again confirming the negative impact of excessive density compression on policy quality. The average agent action index shows clear differences in mapping characteristics: the mean of C3 is 54.996, C2 is 42.998, C1 is 25.766, C4 is 31.795, while C5 is only 14.538. This value directly reflects the distribution characteristics of different mapping functions in converting action indices into physical flows. Cubic and square mappings tend to output larger action values, corresponding to medium-to-high flow ranges, while logarithmic mappings systematically favor extremely low flow ranges.

[0323] Figure 5The daily water level and flow time-series curves over five years visually demonstrate the differences in scheduling patterns shaped by different mapping functions. Observing the evolution of the water level curves (blue), all five configurations successfully maintained the standard water level rhythm during the drawdown period (January-April), flood season (May-September), and storage period (October-December), with the year-end water level stabilizing at around 175 meters, proving that mapping optimization did not disrupt the basic water level control framework. However, subtle differences are clearly presented in the outflow curves (magenta). The outflow curves of C1, C2, and C3 exhibit similar fluctuation patterns following the inflow (black) during the flood season, but C2 and C3 show smoother responses during low-flow periods (such as the end of the drawdown period and the beginning of the storage period), reducing unnecessary flow fluctuations. The outflow curve of C4 is highly similar to that of C1, indicating that exponential mapping did not bring significant differences in actual scheduling. C5 exhibited a significantly abnormal outflow pattern: during the impoundment period, while other configurations controlled their outflow within the range of 20-30 km³ / s to rapidly raise the water level, C5's outflow remained at a low level, causing a delay in the impoundment process.

[0324] Table 3 Summary of performance statistics during the training phase (mean ± standard deviation of the last 10 rounds)

[0325]

[0326] Table 4 Global Performance Comparison During the Evaluation Phase

[0327]

[0328] In summary:

[0329] This invention, based on the Constrained Markov Decision Process (CMDP) framework, systematically compares three types of constraint handling mechanisms: violation correction, penalty-based, and theorized (PPO-Lagrangian) methods, and proposes a two-layer security framework of "state-driven mask + posterior optimization (Lagrangian dual)". Experimental results in a 4-year long-cycle scheduling scenario of the Three Gorges Reservoir show that:

[0330] (1) The corrective method is safe (0.5% violation rate) but has poor performance (average reward of 0.765).

[0331] (2) The penalty method relies on a lot of weight tuning (tested 15 groups). Under the optimal configuration, the reward reaches 0.812, but the violation rate is still 1.2%.

[0332] (3) PPO-Lagrangian significantly outperforms other methods in both violation rate (0.3%) and performance (0.825) by adaptively adjusting the Lagrange multipliers, and reduces hyperparameter tuning costs by 80%;

[0333] (4) The state-driven mask mechanism can reduce the violation rate from 8.5% to 2.1% in the early stage of training, and improve the convergence speed by 31% when combined with Lagrangian.

[0334] (5) The quadratic nonlinear mapping reduces the violation rate by 76% in the low flow segment (ecological base flow) while maintaining the responsiveness in the high flow segment. The comprehensive optimal solution (mask + Lagrangian + quadratic mapping) reduces the violation rate to 0.1%, increases the average reward to 0.839, and reduces the convergence rounds to 290, providing a systematic solution that combines theoretical guarantee and engineering practice for safe and reliable intelligent decision-making in reservoir scheduling.

[0335] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A reinforcement learning method for reservoir scheduling safety based on an improved Lagrange method, characterized in that: At least the following steps are included: S1: Treat the reservoir scheduling problem as a sequential decision problem with security constraints, and use the Constrained Markov Decision Process (CMDP) to formally model the problem, representing the reservoir system as a seven-tuple; S2: Introduce a state-driven action masking mechanism in the strategy sampling stage. Before each decision, dynamically filter out actions that obviously do not meet the physical constraints based on the current reservoir state. This builds a "front line of defense" outside of strategy learning. The state-driven action masking mechanism calculates the feasible outflow interval based on the water balance equation and physical constraints. It is then transformed into an action index interval through action-flow mapping to filter out illegal actions in the strategy sampling stage. S3: Perform nonlinear motion mapping and resolution design, and adopt a nonlinear motion mapping function within a unified framework. By adjusting The shape allows for different motion densities to be allocated across different flow ranges; S4: At the policy optimization level, a constraint reinforcement method based on Lagrange duality is adopted to ensure that the policy satisfies the CMDP constraint in the long-term scale, thus transforming the constraint optimization problem of CMDP into a saddle point search problem in the policy space and dual space.

2. The reservoir scheduling safety reinforcement learning method based on the Lagrange method as described in claim 1, characterized in that: S1 includes at least the following steps: Represent the reservoir system as a 7-tuple: in: For state space, Indicates time The system status; For the action space, Indicates the agent at time... The selected scheduling decision; As the state transition kernel, it describes the state transition process during the execution of actions. The system after Evolved to The probability of; This is an instant reward function used to characterize the comprehensive benefits of power generation, flood control, ecology, and shipping. The constraint cost function is used to quantify the degree of violation of safety constraints such as water level and flow rate; The initial state distribution; Discount factor; The upper bound of the allowable long-term average constraint cost is used to characterize the acceptable level of security risk. In the specific implementation, state It should include at least the current water level, inflow rate, water forecast for the next few days, dynamic water level upper and lower limits for the corresponding time period, and date code, as shown in the following formula: in, This is the current water level; This represents the current inbound flow. For the inflow prediction vector for the next few days; and These are the time-varying lowest and highest operating water levels, respectively. Normalized date encoding, used to characterize seasonality; The evolution of reservoir capacity-water level follows the principle of mass conservation, denoted as... For a moment Storage capacity, Outbound flow; Given the scheduling time step, the water balance equation is: The monotonic function between reservoir capacity and water level obtained from topographic surveys Association, that is: in, As the inverse function of the reservoir capacity-water level curve, it ensures that environmental dynamics strictly obey water conservation and topographic conditions, providing a physical basis for subsequent constraint design; Under the CMDP framework, strategy The optimization objective is to maximize the cumulative discount reward while satisfying long-term constraints. in, The length of the round; Cost is constrained by expectations; This indicates the expected value calculation; Indicates the discount factor; For the reward function; The safe operation requirements of reservoir scheduling are condensed into an expectation constraint, providing a unified mathematical starting point for subsequent constraint processing mechanisms and Lagrange methods.

3. The reservoir scheduling safety reinforcement learning method based on the Lagrange method as described in claim 2, characterized in that: S2 includes at least the following steps: S2.1: Determine the probabilistic form of the mask; Let the discrete action space be... ,in For the action dimension; Given state Define the mask function: in, Indicates action index In state The following is feasible; This indicates that the action was blocked because it violated current water level, flow rate, or reservoir capacity constraints; Let the log-probability of the policy network output without a mask be... The logarithmic probability after adding the mask is written as: Among them, the agreement Therefore, the probability of the masked action occurring after softmax is strictly zero. Specifically, the masking strategy is as follows: This is equivalent to changing the policy definition domain from Shrink to a subset of actionable elements Thus, the masking mechanism injects physical constraints into the action sampling process in a deterministic manner without changing the policy network structure; S2.2: Mapping between action space and flow space; In order to express the engineering water level and reservoir capacity constraints in the action mask, it is necessary to establish a mapping relationship between the action index and the physical outflow. remember and The minimum and maximum feasible outbound flow rates are represented by the following mapping function: Indexing discrete actions Mapped to physical traffic ; Accordingly, define normalized action variables. , used to represent "continuous positions" in the action space, and its relationship with physical flow is written as: in, For a monotonically increasing function, different For different linear or nonlinear mappings, given Next, define the inverse mapping from traffic to normalized actions: And further Quantized into discrete action indexes: in, Indicates rounding; Therefore, any given flow constraint interval All can be mapped to action index ranges This is then transformed into a continuous "action window" in the mask vector; S2.3: Mask interval calculation based on water balance; The key to the masking mechanism lies in how to adjust the masking based on the current state. Calculate the allowed normalized action range This allows us to obtain the corresponding action index range. ; Directly utilizing the water balance equation and water level constraints, for the outflow Provide upper and lower bound constraints; On the one hand, to prevent the water level from exceeding the daily upper limit. Use the following formula to write out the condition that the final storage capacity does not exceed the limit storage capacity. Conditions: in, Given the reservoir capacity corresponding to the upper limit water level, the minimum allowable outflow can be obtained from this formula: when hour, A positive value indicates that a larger outflow is needed to reduce the reservoir capacity; when... hour, If the outflow is negative, the outflow is allowed to be slightly less than the inflow to achieve water storage; On the other hand, to prevent the storage capacity from falling below the daily minimum storage capacity. It needs to meet the following requirements: Thus, the maximum allowed outflow is obtained: The feasible interval for outflow can be written as: in: Furthermore, by combining engineering constraints such as minimum ecological flow, shipping flow, and equipment capacity limits, the endpoints of the intermediate zone can be further tightened. Finally, and Mapped to normalized action range and action index range and in Only actions within that interval are retained; Thus, the masking mechanism provides a feasible domain for actions consistent with physical constraints at each time step based on water balance, achieving a seamless mapping of physical constraints, flow range, and action range.

4. The reservoir scheduling safety reinforcement learning method based on the Lagrange method as described in claim 4, characterized in that: The S3 includes at least the following steps: S3.1: Family of mapping functions and inverse functions; Given the size of the discrete action space Record action index The corresponding normalized position is: Based on this, a class of mappings with a unified form is defined: in It is a strictly monotonically increasing function. It includes at least self-linear mappings, convex mappings, concave mappings, and logarithmic / exponential mappings; For any given ,if only exist If the function is continuous and strictly monotonically increasing, then there exists an inverse function. Thus, the physical flow is expressed through the formula. Mapping and regressing to unify the action space; This inverse mapping is particularly important in action masks, where the given upper and lower bounds of the traffic need to be passed through... Transformed into an action range, in Implemented in the middle; S3.2: Resolution Analysis; To quantitatively analyze the resolution differences of different mapping functions in various flow ranges, the concept of motion density is introduced; remember For a given traffic range of interest, the range length is... ; After mapping this interval to the normalized space, the number of available actions within the interval is approximately: in, They are respectively The corresponding normalized position; Further define the average motion density: but The larger the value, the higher the action resolution within that flow range.

5. The reservoir scheduling safety reinforcement learning method based on the Lagrange method as described in claim 5, characterized in that: The S4 includes at least the following steps: Given dual variables In this case, define the modified value objective: And it is optimized using any deep reinforcement learning algorithm based on policy gradient. In this case, the policy update only needs to replace the immediate reward with the formula in the equation. The rest of the implementation remains unchanged; On the other hand, after several rounds of policy updates, the average constraint cost is estimated based on the empirical trajectory under the current policy. And update the dual variables according to the formula: in, The dual learning rate; Represents the projection onto a non-negative number axis; If the constraint cost of the current strategy is significantly higher than the threshold ,but Increase the size of the constraint to enhance its penalty strength in subsequent policy updates; Conversely, if the constraint cost is much lower than the threshold, then Gradually reduce the rewards to free up more optimization space for the main task rewards; In engineering implementation, to improve numerical stability, the constraint cost is normalized and the default portion is extracted; remember For the environment at all times The original constraint cost is returned. As the normalization factor, If the default threshold is given, then the normalized cost and default metric are written as: Default metrics are used during training. Replacement original cost The inclusion of reward reconstruction and dual updates reduces the optimization difficulties caused by scale differences while keeping the objective structure of CMDP unchanged.

Images