Reservoir flood control scheduling method based on policy reinforcement learning

Abstract

The application discloses a reservoir flood control scheduling method based on policy reinforcement learning, comprising: obtaining reservoir operation parameters, flood forecasting processes and downstream flood control parameters; constructing a reservoir-river simulation environment based on the above parameters; extracting a state vector containing multi-dimensional hydrological state information in the environment, and determining an action space representing the reservoir discharge flow; inputting the state vector into a pre-configured policy network, calculating a dynamic feasible region by combining the physical and safety boundaries fed back by the simulation environment using the network output layer, and mapping the original output to the feasible region to obtain the discharge flow action; inputting the action into the environment for state deduction to generate a scheduling reward containing downstream excessive penalty, and outputting a scheduling policy based on continuous interactive decision. The application can effectively guarantee the physical executability of the scheduling instruction under extreme flood working conditions, and significantly improve the overall safety and robustness of reservoir flood control scheduling.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of reservoir flood control scheduling and artificial intelligence, and in particular to a reservoir flood control scheduling method based on policy reinforcement learning. Background Technology

[0002] Floods, as destructive natural disasters, pose a serious threat to physical infrastructure within a river basin. Reservoir flood control scheduling is a non-engineering measure to enhance the overall flood control capacity of a river basin and mitigate extreme hydrological events. Scientifically formulating regulation strategies based on dynamically changing hydrological conditions and physical reservoir boundaries is of significant technical importance for suppressing flood peaks, reducing downstream river flow loads, and ensuring the structural safety of water conservancy projects.

[0003] In recent years, with the development of artificial intelligence, many studies have introduced reinforcement learning models into flood control decision-making, aiming to generate scheduling schemes through the interaction between intelligent agents and virtual environments. In conventional implementation schemes, the model is usually trained based on historical flood sequences from external input, and an empirical constant penalty mechanism is set to constrain the model behavior. However, under complex and variable extreme flood conditions, the above schemes are difficult to accurately characterize the spatiotemporal evolution of water flow in natural river channels. Moreover, when the system approaches the engineering physical limits and safety red lines, simple state feedback often fails, causing the output scheduling instructions to easily exceed the execution limits of physical equipment or cause instantaneous flow overload at downstream sections.

[0004] In summary, existing methods still suffer from significant shortcomings in the physical executability of model decisions and their dynamic response capability to the global flood control safety threshold when dealing with extremely complex hydrological processes. Therefore, it is necessary to investigate a scheduling method that can enhance the physical compliance of scheduling instructions under complex operating conditions and effectively improve the robustness of global flood control decisions. Summary of the Invention

[0005] Purpose of the invention: To provide a reservoir flood control scheduling method based on policy reinforcement learning, in order to solve the above-mentioned problems existing in the prior art.

[0006] Technical solution: A reservoir flood control scheduling method based on policy reinforcement learning, comprising:

[0007] Acquire reservoir operating parameters, flood forecasting processes, and downstream flood control parameters;

[0008] Based on the reservoir operating parameters and the downstream flood control parameters, a reservoir-river simulation environment is constructed.

[0009] Based on the reservoir-river simulation environment and the flood forecasting process, a state vector containing multi-dimensional hydrological state information is extracted, and the action space representing the reservoir discharge flow is determined.

[0010] The state vector is input into a pre-configured policy network. The output layer of the policy network is used to calculate the dynamic feasible region by combining the physical boundary and safety boundary fed back from the reservoir-river simulation environment. The original output of the policy network is then mapped into the dynamic feasible region to obtain the discharge flow action.

[0011] The downstream flow action is input into the reservoir-river simulation environment for state deduction, the state vector is updated, and a scheduling reward including downstream exceedance penalty is generated. Based on continuous interactive decision-making, the reservoir flood control scheduling strategy is output.

[0012] Preferably, the dimensionless downstream control station flow exceeding penalty term is configured with a penalty coefficient, which is adaptively adjusted based on the downstream control station flow feedback from the reservoir-river simulation environment. The specific adjustment logic is as follows:

[0013] When the downstream control station traffic is not greater than the pre-configured warning traffic, the penalty coefficient remains a basic constant;

[0014] When the downstream control station traffic exceeds the pre-configured warning traffic, the penalty coefficient increases exponentially with the degree to which the downstream control station traffic exceeds the warning traffic.

[0015] Preferably, the policy network is pre-trained through the following offline training phase:

[0016] Build an experience replay pool with pre-configured capacity, and a pre-built value network for evaluating policy performance;

[0017] The intelligent agent interacts with the reservoir-river simulation environment to collect experience trajectory tuples containing the current state, discharge flow action, scheduling reward and the state at the next moment, and stores the experience trajectory tuples in the experience playback pool.

[0018] The experience trajectory tuples are extracted from the experience replay pool for training, and the value network is updated with the goal of minimizing the temporal difference error. The parameters of the policy network are also updated along the value-increasing direction.

[0019] Preferably, the experience replay pool is a priority experience replay pool, and the step of extracting the experience trajectory tuples from the experience replay pool for training is specifically executed through the following sampling mechanism:

[0020] The sampling priority is calculated based on each of the aforementioned empirical trajectory tuples. The sampling priority is composed of the absolute value of the dimensionless time-series difference error of the empirical trajectory tuple and the additional priority of flood severity determined based on the degree of downstream flow exceeding the warning level.

[0021] The sampling probability of extracting the empirical trajectory tuple is determined based on the sampling priority, and an importance sampling weight related to the sampling probability is applied in the training loss function used to update the network parameters to correct for distribution bias.

[0022] Preferably, the allowable range of opening and closing rates of the gate is composed of a lower limit and an upper limit.

[0023] The lower limit of the amplitude is determined based on the actual discharge flow in the previous period and the pre-configured allowable amplitude of the receding section, while the upper limit of the amplitude is determined based on the actual discharge flow in the previous period and the pre-configured allowable amplitude of the rising section.

[0024] Preferably, during the process where the penalty coefficient increases exponentially with the degree to which the downstream control station's traffic exceeds the warning traffic:

[0025] The difference between the warning flow and the safe discharge of the downstream control station is used as a standardized constraint scale to perform a dimensionless transformation on the excess range of the downstream control station flow, and the surge rate of the penalty coefficient is determined accordingly.

[0026] Preferably, the method is implemented using a rolling update mechanism in actual reservoir flood control operations:

[0027] Using a preset scheduling period as the time step, the updated flood forecast process and reservoir operation parameters are obtained in real time at the beginning of each time step, and the calculation and mapping of the dynamic feasible domain are re-triggered to output the optimal discharge flow action for the current period.

[0028] Preferably, after the reservoir flood control scheduling strategy is output based on continuous interactive decision-making, the system further includes a physical drive execution step:

[0029] Based on the downstream flow action determined in the output reservoir flood control scheduling strategy, a reservoir gate hoist control command is generated, and the control command is sent to the reservoir's underlying equipment control system to physically drive the gate to perform the action.

[0030] Beneficial effects: This invention can effectively ensure the physical executability of dispatching instructions under extreme flood conditions, and significantly improve the overall safety and robustness of reservoir flood control dispatching. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the implementation of a reservoir flood control scheduling method based on policy reinforcement learning according to the present invention.

[0032] Figure 2 This is a diagram illustrating the traditional scheduling rule-based scheduling process of a reservoir flood control scheduling method based on policy reinforcement learning, as described in this invention.

[0033] Figure 3 This is a diagram illustrating the scheduling process of a flood control scheduling intelligent agent in a reservoir flood control scheduling method based on policy reinforcement learning, as described in this invention. Detailed Implementation

[0034] Example 1: This example provides a reservoir flood control scheduling method based on policy reinforcement learning, which includes the following steps:

[0035] Step 101: Obtain reservoir operation parameters, flood forecasting process, and downstream flood control parameters;

[0036] The reservoir operation parameters specifically include basic information that characterizes the physical constraints of the water conservancy project, such as normal storage level, dead storage level, flood control limit level, water level-reservoir capacity curve, and tailrace level-discharge curve. The flood forecasting process provides a time series of the inflow trend over a future period. Downstream flood control parameters mainly refer to the safe discharge threshold that the downstream protected objects can withstand.

[0037] Step 102: Based on the reservoir operation parameters and the downstream flood control parameters, construct a reservoir-river simulation environment;

[0038] Specifically, the simulation environment is constructed by combining the reservoir water balance equation and river evolution mechanism, which are standard in this field. The initial water level is converted into an initial reservoir capacity using the water level-capacity curve. Based on the water balance principle, the reservoir capacity and water level are recursively calculated from the beginning to the end of each time period. Simultaneously, the downstream flow of water to the corresponding downstream control section is extrapolated. Constructing this reservoir-river simulation environment allows the agent to perform high-frequency simulation calculations in a virtual space, free from the risks of physical trial and error, to explore the optimal flood control process.

[0039] Step 103: Based on the reservoir-river simulation environment and the flood forecasting process, extract the state vector containing multi-dimensional hydrological state information, and determine the action space representing the reservoir discharge flow.

[0040] The multidimensional hydrological state information is a dynamic feature set extracted from real-time feedback and external inputs from the reservoir-river simulation environment. It may include information such as the current dam water level, inflow, outflow, and predicted inflow for multiple future times. These physical features are combined to form the state vector. The action space defines the set of decision instructions executable by the scheduling model. In this embodiment, the decision variables of the action space are explicitly limited to the reservoir's outflow during each scheduling period. Extracting the state vector and determining the action space realizes the mathematical modeling transformation from real-world physical scheduling requirements to a standard Markov decision process.

[0041] Step 104: Input the state vector into the pre-configured policy network, use the output layer of the policy network to calculate the dynamic feasible region by combining the physical boundary and safety boundary fed back from the reservoir-river simulation environment, and map the original output of the policy network into the dynamic feasible region to obtain the discharge flow action.

[0042] The policy network employs a pre-trained neural network structure to implicitly establish a nonlinear mapping relationship between state features and optimal scheduling actions. During the network's forward inference phase, physical boundaries such as the maximum discharge capacity corresponding to the current reservoir water level, obtained in real-time from the reservoir-river simulation environment, are combined with safety boundaries ensuring downstream compliance to calculate the upper and lower limits for the current time period, thereby determining the dynamic feasible region. Based on this, the continuous raw output generated by the last layer of the policy network is transformed using a preset activation function and linearly scaled to the upper and lower bounds of the dynamic feasible region. The constraint mapping process ensures that the final generated discharge flow action naturally satisfies the reservoir's engineering safety constraints and the flood control requirements of downstream protected objects, avoiding the generation of physically unexecutable invalid scheduling instructions during exploration or inference.

[0043] Step 105: Input the downstream flow action into the reservoir-river simulation environment for state deduction, update the state vector and generate a scheduling reward that includes downstream exceedance penalty, and output the reservoir flood control scheduling strategy based on continuous interactive decision-making.

[0044] The mapped and legally authorized downstream discharge actions are applied as execution commands to the reservoir-river simulation environment. The environment, based on physical calculation rules, extrapolates to the next time step and outputs updated water level and flow data as observations for that next time step. Simultaneously, the environment quantifies and evaluates the hydrological consequences of this action and provides a calibrated scheduling reward. This reward comprehensively measures the adequacy of the reservoir's flood control capacity and triggers a downstream exceedance penalty when the downstream flow exceeds a preset safety threshold. Through continuous interaction across the entire flood event timeline, continuously acquiring status, outputting actions, and receiving rewards, a reservoir flood control scheduling strategy capable of dynamically adapting to changing floods is ultimately formed.

[0045] Example 2: Based on Example 1 above, this example further details the evolution-driven mechanism for calculating downstream flow in the process of constructing a reservoir-river simulation environment. In one possible implementation, in the reservoir-river simulation environment, the flow at the downstream flood control station is determined in the following way:

[0046] Step 201: Based on the flood forecasting process, extract the inflow between the dam site and the control station;

[0047] The term "interval inflow" specifically refers to the lateral runoff generated by rainfall confluence over the uncontrolled catchment area between the reservoir dam and the downstream flood control station. In practice, this is achieved by analyzing rainfall field data from the flood forecasting process or by using an area analogy algorithm to extract the time series of lateral inflow for this specific river segment. Extracting this interval inflow introduces natural hydrological increments, which are not directly controlled by reservoir scheduling, into the simulation environment, providing fundamental hydrological boundary conditions for subsequently generating accurate total discharge at downstream control sections.

[0048] Step 202: Based on the river channel physical calculation rules, the downstream flow action is converted into river flood evolution components;

[0049] The aforementioned river flood evolution component refers to the flow component of the water discharged from the reservoir dam, after passing through the river channel for storage and displacement, and reaching the downstream control station. The controlled discharge flow action is input into the river physical calculation model for processing, simulating the hysteresis and flattening effects of flood waves propagating within the natural river channel. Physical calculation rules are used to transform the discharge action, enabling the virtual environment to realistically reflect the attenuation and evolution patterns of water flow in the river channel's spatial distribution, replacing the simplistic assumption of equating the discharge flow with the downstream flow.

[0050] Step 203: In the recursive calculation process, the river flood evolution component is determined based on the discharge flow action, the actual discharge flow of the previous time period generated by the simulation environment iteration, and the river flood evolution component of the previous time period.

[0051] In this embodiment, the standard Muskingan model is used to implement the river channel physical calculation rules. During time-step recursion, the current discharge flow is taken as the current inflow, the actual discharge flow of the previous time period is taken as the previous inflow, and the calculated river flood evolution component of the previous time period is taken as the previous outflow. After applying corresponding evolution coefficients to the three known hydrological parameters, a weighted sum is obtained to obtain the river flood evolution component at the current time. By introducing inflow and outflow states from multiple time periods for joint calculation, the continuity and dynamic conservation of the hydrological extrapolation process in the time series are maintained.

[0052] Step 204, wherein the evolution coefficients of the Muskingen evolution equation are set according to the physical constraint that the product of the pre-configured river section storage parameter and the pre-configured flow proportion factor is not less than twice the time step.

[0053] The pre-configured river storage capacity parameter characterizes the propagation time constant of flood waves within a specific river segment, while the pre-configured flow weight factor reflects the relative dependence of channel storage capacity on inflow and outflow within a specific river segment. When calculating the evolution coefficients used in the weighted summation, the system is forced to perform physical boundary checks. Specifically, with a time step of ∆t, a pre-configured river storage capacity parameter of K, and a pre-configured flow weight factor of x, the system determines and restricts the parameter configuration to satisfy ∆t ≥ 2*K*x. This physical constraint restricts parameter values, preventing numerical oscillations caused by negative evolution coefficients during calculation. In a specific parameter configuration example, the pre-configured river storage capacity parameter can be set to 2 hours, and the pre-configured flow weight factor to 0.25. To satisfy the above constraint, 2*2*0.25 = 1 is calculated, thus the simulation environment iteration time step is forcibly set by the system to be no less than 1 hour. Performing this constraint check and parameter setting rule eliminates interference from unreasonable parameter configurations on the reinforcement learning environment feedback.

[0054] Step 205: The river flood evolution component is superimposed with the inflow of the interval to obtain the flow rate of the downstream flood control station.

[0055] Based on the principle of linear superposition of one-dimensional river flow, the system adds the calculated river flood evolution component to the interval inflow extracted at the same time. The sum of the values ​​is output as the flow rate at the downstream flood control station. The calculated flow rate at the downstream flood control station is then used as a feedback variable in the environmental iteration, providing an objective basis for the reinforcement learning model to evaluate whether the current scheduling strategy has exceeded the downstream physical safety limit.

[0056] This embodiment provides another implementation method for calculating the river flood evolution components. A lag algorithm is used instead of the Muskingan equation to calculate the evolution components. The remaining flow determination steps are the same as in Embodiment 2, and can be referred to the foregoing description, so they will not be repeated here. In one possible implementation, the following steps are included:

[0057] Step 301: Based on the pre-configured river channel time delay steps, perform time delay translation processing on the downstream flow action to obtain the river channel flood evolution component.

[0058] The lag algorithm ignores the smoothing and attenuation effects of natural river channels on flood waves, and only calculates the hydrological calculation model of the absolute time lag caused by the propagation of water along the physical river channel. The pre-configured river channel time delay step number represents the number of steps on the discrete time axis of the environmental iteration required for the flood wave to propagate from the reservoir dam outlet section to the downstream flood control section. In specific implementation, the pre-configured river section storage parameters and the simulation environment iteration time step are obtained. The ratio of the pre-configured river section storage parameters to the time step is calculated, and the ratio is rounded to obtain the pre-configured river channel time delay step number in integer form. For example, if the physical propagation time constant of the flood wave in the target river section is 5 hours, and the system-set environmental time step is 2 hours, then 5 / 2 = 2.5 is calculated, and after rounding, the pre-configured river channel time delay step number is 3.

[0059] After determining the pre-configured number of river channel time delay steps, the system establishes a retrospective mapping mechanism for historical actions in the time-series memory. Specifically, starting from the current moment, the system backtracks backward along the historical timeline by the time span corresponding to the pre-configured number of river channel time delay steps, extracts the discharge flow action physically performed by the reservoir at that historical moment, and directly assigns the extracted value to the river flood evolution component at the current moment. The use of time delay translation replaces the complex recursive derivation of fluid dynamics partial differential equations with a pure time-series shift operator, reducing the computational overhead of the reservoir-river simulation environment in a single time step iteration. In steep mountain river sections where flood wave attenuation effects are extremely weak, this processing method can accelerate the interactive sampling efficiency of reinforcement learning while ensuring the correctness of the physical evolution laws.

[0060] In some optional implementations, when the pre-configured river storage capacity parameter is not divisible by the time step, time-delay shifting of the discharge flow action can be achieved through time-series interpolation to suppress waveform distortion errors introduced by discretization rounding. The system extracts the historical discharge flow actions corresponding to two integer time-delay steps adjacent to the actual time-delay parameter, calculates the fractional part of the actual time-delay parameter, uses this fractional part as a weighting coefficient, performs a linear weighted summation on the two historical discharge flow actions, and uses the weighted calculation result as the current river flood evolution component. Using interpolation shifting allows the virtual environment to output a smooth river flood evolution component that conforms to the physical continuity assumption, even with coarse-grained time step settings.

[0061] After completing the time-delay translation processing of the downstream flow action, the corresponding interval inflow is extracted, and the river flood evolution component is superimposed with the interval inflow to output the flow of the downstream flood control station with delayed physical attributes.

[0062] Example 4: Based on Example 1 above, this example further details the multidimensional hydrological state information. In one possible implementation, the following steps are included:

[0063] Step 401: The reservoir water level and inflow rate at the current moment are fed back from the reservoir-river simulation environment.

[0064] The fundamental feature dimensions of the multidimensional hydrological state information originate from the objective physical state of the reservoir area. The system acquires the upstream water level data generated in the simulation environment at the current iteration time step and the current inflow data calculated from the upstream confluence. To eliminate the negative interference of different physical dimensions on the update of strategy network parameters, the system performs normalization processing on the acquired current reservoir water level and current inflow. By extracting and processing real-time physical quantities, the model calculation unit can perceive the current reservoir water storage status and the immediate inflow load it faces.

[0065] Step 402: Forecast the inflow to the reservoir based on the multiple future moments extracted from the flood forecasting process;

[0066] Flood control decision-making models require a forward-looking time perspective. With a total forecast time step of T, the system extracts the inflow time series from the current moment to T future time steps from the flood forecast data received from external sources. This extracted time series is then concatenated into the state vector. Introducing the forward-looking forecast series provides time-series data support for the model to calculate pre-release and reservoir emptying actions in advance.

[0067] Step 403, and the actual discharge flow rate of the previous period fed back from the reservoir-river simulation environment.

[0068] After introducing the river flood evolution mechanism, the current discharge action will have a continuous impact on the flow at downstream flood control stations with time lag effects. If the state space only contains the current reservoir state and forecast information, it will violate the fundamental assumption of no aftereffect in Markov decision processes, causing time-series difference errors to be unable to be correctly backpropagated and attributed to historical actions. To address this, the system establishes a state cache dimension during environment iteration, capturing the actual discharge flow data output and executed in the previous time step, and incorporating it as an extended feature into the state vector. Extracting the actual discharge flow from the previous time period constructs a physical memory closed loop in the state transition process, enabling the value network to complete credit allocation across river flow time lags when evaluating the value of actions.

[0069] In terms of specific data structure settings, the system extracts the current reservoir water level, the current inflow, the actual outflow from the previous period, and the predicted inflow sequence for multiple future periods, and concatenates them into a one-dimensional floating-point tensor according to a preset feature channel order, which serves as the state vector. The specific dimension of this one-dimensional floating-point tensor is determined by the number of single-point hydrological features and the total number of time steps in the forecast period, thereby ensuring that the data format input to the neural network has a definite dimensional space.

[0070] Example 5: Based on Example 1 above, this example further details the steps for calculating the dynamic feasible region and mapping it to obtain the outflow action. In one possible implementation, the steps include:

[0071] Step 501: Extract the pre-configured safe discharge capacity of the downstream control station from the downstream flood control parameters;

[0072] The downstream control station's safe discharge capacity is characterized as the maximum volumetric flow rate of water allowed to pass through the target river channel to maintain the physical stability of the downstream protected object. This physical quantity serves as a rigid flood control boundary, and is read and loaded by the system from a pre-set database of water conservancy project scheduling procedures.

[0073] Step 502: Based on the flood forecasting process, obtain the interval inflow forecast value corresponding to the river channel time lag time;

[0074] For the confluence area between the reservoir dam and the downstream control station, the system indexes and extracts the interval lateral inflow forecast sequence with an advance time step of the time delay parameter based on the flow propagation time delay step parameter calculated by the river evolution model during the flood forecasting process. The inflow hydrological volume at the target time is extracted to provide a lateral flow boundary reference for inferring the available safe discharge index of the reservoir dam at the current time.

[0075] Step 503: Under the conservative estimate of ignoring the attenuation of river flood, subtract the safe discharge of the downstream control station from the predicted inflow value of the interval to obtain the upper limit of the safe forward discharge of the downstream.

[0076] To establish a pre-emptive constraint framework in complex river channels with time lags in water flow, the system advances the future flow boundary at the current decision-making moment. Based on the principle of extreme safety defense, the system does not consider the peak-reduction effect of channel impoundment, extracts the safe discharge of the downstream control station and the predicted inflow value of the interval, and calculates the algebraic difference. The resulting algebraic difference is set as the upper limit of the downstream safe forward discharge. Applying this upper limit constraint means that when the downstream flow reaches the downstream section after a preset time step, the total flow, after being superimposed with the lateral inflow, is controlled within the safety threshold.

[0077] Step 504, the allowable range of the opening and closing rate of the gate is composed of a lower limit and an upper limit.

[0078] The flow range that a reservoir's flood discharge mechanical structure can regulate per unit time is limited by the equipment's physical inertia and actuator limits. The system is constructed with an amplitude range having upper and lower boundaries, forcibly limiting the flow variation amplitude within a time step to within the physical range allowed for safe operation of the equipment.

[0079] Step 505, wherein the lower limit of the amplitude is determined based on the actual discharge flow in the previous period and the pre-configured allowable amplitude of the receding section, and the upper limit of the amplitude is determined based on the actual discharge flow in the previous period and the pre-configured allowable amplitude of the rising section;

[0080] In actual discharge control, opening and closing gates face asymmetrical hydrodynamic constraints. Drastic flow reduction can easily lead to downstream bank slope instability. Therefore, the system calculates the difference between the actual discharge flow in the previous period and the pre-configured allowable amplitude of the receding section to generate the lower limit of the amplitude; and calculates the sum of the actual discharge flow in the previous period and the pre-configured allowable amplitude of the rising section to generate the upper limit of the amplitude. This asymmetrical amplitude calculation method ensures that the action search space aligns with the physical requirements of the underlying hydrodynamics.

[0081] Step 506: The lower and upper bounds of the dynamic feasible domain are determined by the intersection of the upper limit of the physical discharge capacity of the reservoir at the current water level, the allowable range of the gate opening and closing rate, and the upper limit of the downstream safe forward discharge.

[0082] The system extracts the upper limit of the physical discharge capacity corresponding to the current moment through the hydrodynamic query module. The execution logic judgment module performs an intersection set operation on the upper limit of the capacity, the amplitude range, and the upper limit of the downstream safety look-ahead discharge capacity. The intersection operation constructs a triple hard constraint boundary that integrates physical output capacity, temporal execution smoothness, and look-ahead spatial safety, and outputs it as the lower and upper bounds of the dynamic feasible region.

[0083] Step 507: Determine the relationship between the calculated upper limit of the downstream safety forward discharge and the lower limit of the amplitude range.

[0084] In the first execution path, the upper limit of the downstream safety look-ahead discharge is not less than the lower limit of the amplitude range. Using a preset activation function, the original output of the policy network is mapped to a standard range to obtain a mapped value. Based on the lower and upper bounds of the dynamic feasible region, the mapped value is linearly scaled so that the final output discharge flow action is strictly limited to the lower and upper bounds of the dynamic feasible region. Specifically, the last layer neurons of the policy network output continuous real numbers, which are passed as input values ​​to the Sigmoid activation function and converted into a mapping index between 0 and 1. The system obtains the algebraic difference between the upper and lower bounds of the dynamic feasible region, multiplies this algebraic difference by the mapping index, and adds the lower bound to generate the final flow output value. Constructing this constraint mapping layer avoids the computational graph breakage problem caused by forced post-pruning of the network output, preserving complete differentiable gradient information. In some alternative implementations, the original activation mapping mechanism can be replaced by performing algebraic translation and scaling operations on the output range of the hyperbolic tangent function to achieve the same interval projection effect.

[0085] In the second execution path, when extreme flood conditions cause the calculated downstream safe forward discharge upper limit to be less than the lower limit of the amplitude range, the downstream safe forward discharge upper limit constraint is removed; the upper bound of the dynamic feasible region is redefined as the minimum value between the upper limit of the physical discharge capacity and the upper limit of the amplitude range. In such boundary conflict scenarios, if the lateral inflow of the interval is overloaded, even if the control system issues the fastest allowable closure action to the gate, the downstream flow will still exceed the limit. At this time, the system performs an empty set back decision, forcibly stripping the soft upper limit constraint of the downstream safe forward discharge, and only extracting and retaining the hard physical boundary of the reservoir equipment for calculation. The triggered boundary degradation backoff mechanism maintains the operational capability of the deep learning control center to continuously output physically executable instructions under extreme hydrological conditions.

[0086] By combining the reservoir's real-time physical capabilities, the gate's asymmetric amplitude range, and forward-looking safety limits, a three-constraint feasible region for dynamic contraction was constructed. Combined with activation mapping and a physical backoff mechanism under extreme conditions, the discharge actions output by the deep learning model naturally align with the mechanical operating limits of the engineering equipment, avoiding dangerous out-of-bounds scheduling commands generated during virtual exploration or online inference.

[0087] Example 6: Based on Example 5 above, this example further details the process of generating scheduling rewards that include downstream overrun penalties. In one possible implementation, the following steps are included:

[0088] Step 601, the scheduling reward is composed of the reservoir area safety target and the downstream flood control section safety target, specifically including: a dimensionless remaining flood control capacity item characterizing the reservoir's flood control capacity; and a dimensionless downstream control station flow exceeding penalty item used to constrain downstream outflow.

[0089] When calculating the dimensionless remaining flood control capacity term representing the reservoir's flood control capacity, the system obtains the pre-configured normal reservoir water level and flood control limit water level, and converts them into the corresponding total flood control capacity based on the water level-capacity relationship. The system extracts the remaining flood control capacity of the reservoir at the next moment calculated in the current iteration, calculates the ratio of this remaining flood control capacity to the total flood control capacity, and thus obtains the dimensionless remaining flood control capacity term. By performing normalization processing, the scale influence of the absolute value of the specific reservoir capacity on the value network evaluation results is eliminated. When calculating the dimensionless downstream control station flow exceedance penalty term used to constrain downstream outflow evolution, the system extracts the downstream control station flow fed back from the simulation environment, calculates the difference between this flow and the downstream control station's safe discharge, and compares this difference with the downstream control station's safe discharge to obtain the dimensionless exceedance ratio. The basic penalty term is generated after performing a square operation on this exceedance ratio.

[0090] Step 602: Determine the relationship between the downstream control station flow rate fed back from the reservoir-river simulation environment and the pre-configured warning flow rate.

[0091] In the first execution path, when the flow at the downstream control station is not greater than the pre-configured warning flow, the penalty coefficient remains a basic constant. Specifically, when the hydrological condition is within the safe range, the system assigns a fixed algebraic weight to the dimensionless downstream control station flow exceeding penalty term. The pre-configured warning flow can be set to 80% of the safe discharge of the downstream control station. Under this hydrological condition, the basic constant can be set to 10. Maintaining a constant penalty coefficient allows the agent to focus on optimizing the remaining flood control capacity of the reservoir area before approaching the flood control red line.

[0092] In the second execution path, when the downstream control station traffic exceeds the pre-configured warning traffic, the penalty coefficient increases exponentially with the degree to which the downstream control station traffic exceeds the warning traffic. Using the difference between the warning traffic and the downstream control station's safe discharge capacity as a standardized constraint scale, the excess range of the downstream control station traffic is transformed into a dimensionless value, and the increase rate of the penalty coefficient is determined accordingly. Specifically, the system calculates the algebraic difference between the downstream control station traffic and the warning traffic, divides this algebraic difference by the aforementioned standardized constraint scale, and obtains a dimensionless ratio characterizing the degree of exceeding the warning threshold. The system multiplies this dimensionless ratio by a pre-configured penalty growth rate parameter, calculates the natural exponential function value of the result, and finally multiplies it by the fundamental constant to generate the dynamic penalty coefficient at the current moment. The pre-configured penalty growth rate parameter can be set as a floating-point number between 3 and 5. This dynamic surge mechanism constructs a soft security barrier effect, forcing the policy network to actively reduce the discharge traffic when approaching the security boundary.

[0093] In some optional implementations, the dynamic adjustment logic for the weight of the over-limit penalty term can also use a high-order polynomial mapping instead of exponential function operations. After calculating the dimensionless ratio representing the severity of the over-limit, an even-numbered power operation is performed on it, where the even-numbered power can be set to a positive integer not less than 4. The result of the power operation is then multiplied by a fundamental constant to generate a dynamic penalty coefficient. Using high-order power operations can also achieve a penalty adjustment effect that is non-linearly amplified as the over-limit magnitude increases.

[0094] The river channel evolution simulation was incorporated into the underlying feedback loop, and a dynamic composite reward mechanism was designed. When the evolving water flow approaches the warning line of the downstream control station, the system triggers an adaptive penalty that increases exponentially with the magnitude of the exceedance, constructing a flexible soft safety barrier. This forces the decision-making center to proactively reduce the discharge before the boundary is breached, realizing the transformation of downstream flood control section safety from reactive post-event punishment to proactive dynamic collaborative control.

[0095] Example 7: This example provides an implementation method for pre-training a policy network and a value network. In one possible implementation, the policy network is pre-trained through the following offline training phase:

[0096] Step 701: Construct an experience replay pool with pre-configured capacity and a pre-built value network for evaluating policy performance.

[0097] The system initializes an experience replay pool in memory at a specified data storage depth to cache historical hydrological interaction trajectory data, thereby breaking the correlation of deep learning time series. Simultaneously, it constructs a master policy network, a master value network, and corresponding target policy and target value networks based on a multilayer perceptron architecture. Each neural network contains an input layer, two hidden layers, and an output layer. The hidden layers employ rectified linear unit activation functions and are configured with 128 neurons. The parameter matrix of the target network is iteratively approximated by the master network using a soft update algorithm with set tracking coefficients. The set tracking coefficients can be floating-point values ​​between 0.001 and 0.01.

[0098] In terms of network hierarchy and connectivity, the policy network receives data using the state vector as a single input layer; while the main value network and the target value network adopt a multimodal feature fusion input layer structure. The underlying connectivity is as follows: the state vector and the outflow traffic action are received independently, and after dimensional alignment through the first fully connected network, a concatenation operation is performed on the feature dimension. Then, the concatenated features are input to the subsequent hidden layer containing 128 neuron nodes for multi-layer nonlinear mapping, and finally, a scalar value assessment is output through a linear activation function.

[0099] Step 702: The agent interacts with the reservoir-river simulation environment to collect experience trajectory tuples containing the current state, discharge flow action, scheduling reward and the state at the next moment, and stores the experience trajectory tuples in the experience playback pool.

[0100] During the offline interaction phase, the system superimposes exploratory noise onto the defined sequence of outflow actions output by the main strategy network. The superimposed exploratory noise uses a random number sequence following a Gaussian distribution, and the standard deviation of this random sequence decreases linearly with the number of training rounds. After executing the physical action of superimposing the noise, the system obtains hydrological feedback information from the environment, packages and combines it to generate the experience trajectory tuple, and pushes it to the experience replay pool for cyclic writing and storage.

[0101] Step 703: Calculate the sampling priority based on each of the empirical trajectory tuples. The sampling priority is composed of the absolute value of the dimensionless time-series difference error of the empirical trajectory tuple and the additional priority of flood severity determined based on the downstream flow warning level.

[0102] The system extracts empirical trajectory tuples from the pool, calculates the estimated target value of these tuples using the target network, and subtracts the estimated target value from the current value output by the main value network to obtain the time-series difference error. For the added priority of flood severity, the system extracts the downstream control station flow recorded in the tuple, calculates the difference between this flow and the pre-configured warning flow, divides it by the safe discharge capacity to achieve dimensionless calibration, and finally multiplies it by the pre-configured severity weight coefficient to generate the final priority. The system performs an algebraic addition operation on the dimensionless time-series difference error (after extracting the absolute value) and the added priority of flood severity to output the final sampling priority. The use of a two-factor priority allocation mechanism that superimposes physical hydrological severity allows the model to obtain more frequent network parameter correction opportunities when dealing with critical samples where the flood peak approaches the safety red line.

[0103] Step 704: Determine the sampling probability of extracting the empirical trajectory tuple according to the sampling priority, and apply an importance sampling weight related to the sampling probability to correct the distribution bias in the training loss function used to update the network parameters.

[0104] The system calculates the percentage of sampling priority of each empirical trajectory tuple in the full sample set, generating a non-uniform sampling probability distribution. To offset the statistical bias caused by non-uniform sampling on the network parameter convergence plane, the system calculates a corrected weight for each data tuple based on the sampling probability. This corrected weight includes an annealing factor that linearly anneals from the initial parameters to an upper limit of 1.0; the initial parameters can be set to 0.4. The calculated importance sampling weights are directly multiplied into the mean squared error loss function for error scaling.

[0105] Step 705: Extract the experience trajectory tuples from the experience replay pool for training, update the value network with the goal of minimizing the temporal difference error, and update the parameters of the policy network along the value-increasing direction.

[0106] The system randomly selects batch data according to the set batch size parameter and the sampling probability. The selected data clusters are input into the value network, and the backpropagation algorithm is used to calculate the gradient and update the value network weight matrix, making its output approximate the true cumulative reward. Simultaneously, based on the deterministic policy gradient calculation criterion, the system calculates and updates the policy network's weight parameter matrix along the evaluation gradient direction output by the value network, forcing the policy network to output a discharge flow action with higher scheduling rewards under given multi-dimensional hydrological state information.

[0107] In offline training, a two-factor priority experience replay pool was introduced, which superimposes flood severity and temporal difference error, forcing the algorithm's computing power to be tilted and concentrated towards high-risk periods such as flood peak exceeding warning levels. This accelerated the agent's efficiency in absorbing physical experience under extreme flood control scenarios and effectively raised the model's global flood control safety ceiling under complex climate and environmental changes.

[0108] Example 8: Based on Example 1 above, this example further details the physical execution process of the method in a real business system. In one possible implementation, it includes the following steps:

[0109] Step 801: The method is implemented in actual reservoir flood control operations using a rolling update mechanism: with a preset scheduling period as the time step, the updated flood forecast process and reservoir operation parameters are obtained in real time at the beginning of each time step, and the calculation and mapping of the dynamic feasible domain are re-triggered to output the optimal discharge flow action for the current period.

[0110] Flood control scheduling is a typical dynamic stochastic decision-making process. The system accesses the data stream from the meteorological and hydrological forecasting center via a communication interface, acquiring rolling rainfall and runoff forecast sequences according to a set update frequency. The set update frequency is consistent with the telemetry cycle of the hydrological station network. At the beginning of each physical time step, the system discards the long-term forecast decision sequence generated at the previous moment, using only the latest extracted measured hydrological data and the current reservoir water level as input state features. The model loads the trained policy network parameters and re-solves the physical limits, time amplitude, and downstream look-ahead safety boundary of the current reservoir state in a single forward inference calculation, recalculating and outputting an independent action command for the current time period. The rolling time-domain control mechanism is used to execute the scheduling, enabling the network model to promptly absorb the incremental hydrological information brought about by meteorological forecast error correction, suppressing the divergence of state trajectories caused by open-loop long-sequence decision-making.

[0111] Step 802: Based on the downstream flow action determined in the output reservoir flood control scheduling strategy, generate reservoir gate hoist control instructions and send the control instructions to the reservoir's underlying equipment control system to physically drive the gates to perform actions.

[0112] After completing the policy output in cyberspace, the system enters the cyber-physical system fusion execution phase. The system extracts the numerical discharge flow actions generated by the policy network, uses the pre-set gate opening and discharge flow relationship curves of the reservoir for back-interpolation calculation, and converts the flow demand into gate opening parameters characterizing the displacement of mechanical equipment. The system generates corresponding industrial control protocol data packets and sends instructions to programmable logic controllers (PLCs) via the underlying industrial control bus or industrial Ethernet network. The PLC parses the instructions and generates electrical drive signals to drive the dam's winch-type or hydraulic gate hoist to change the gate opening. During this process, displacement sensors and water level sensors at the reservoir's bottom layer collect real-time feedback on mechanical status and water flow, updating the reservoir's operating parameters for the next moment in a closed loop. The execution of the physical drive steps realizes the closed-loop control of the reinforcement learning algorithm from the computing server to the real physical space of the water conservancy infrastructure.

[0113] Example 9: This example provides an implementation method for verifying the scheduling effect of the above-mentioned flood control scheduling method.

[0114] In practical implementation, the system loads measured hydrological process data from typical flood events and performs parallel simulations using both traditional static scheduling rules and the policy-based reinforcement learning method provided in this embodiment. Traditional static scheduling rules look up the corresponding release command based on the scheduling zone where the reservoir's current water level is located and fixed calendar time periods, without incorporating a dynamic response model for future weather forecasts.

[0115] Under multi-peak flood conditions, traditional dispatching rules, which rely solely on current water levels to determine discharge demand, issue relatively small discharge commands when reservoir levels are low before the flood peak arrives. This results in reservoirs failing to create sufficient flood control capacity in advance. When the main flood peak enters the reservoir, the inflow increases dramatically, forcing the reservoir system to passively discharge a large amount of floodwater in a short period. This causes the reservoir level to rapidly approach the maximum flood control level, and downstream channels bear a sudden, instantaneous flow load.

[0116] Using the flood control scheduling method provided in this embodiment, the system captures the boundary conditions for future hydrological load intensification through forward-looking forecast sequences in multi-dimensional hydrological state information before the main flood peak reaches the dam. Under the constraint of the safety red line, the model generates pre-release commands, causing the reservoir water level to enter a downward trajectory in the early stages of inflow, actively releasing physical storage capacity. During the main flood peak passage phase, the model utilizes the triple constraints of the dynamic feasible region and an adaptive penalty weight allocation mechanism to maintain the outflow at a high plateau period that does not exceed the downstream flood control section's safety threshold, performing peak shaving and smoothing calculations on the inflow flood peak. The hydrological state trajectory predicted by the model shows that, under the premise of satisfying all flood control physical and safety constraints, the highest regulating flood level of the reservoir during the flood process is lower than the set highest flood control level, and the slope of the outflow process line is limited within the equipment's amplitude range, without drastic fluctuations. The above-mentioned prediction logic verifies the objective physical effect of the soft and hard collaborative constraint mechanism on the reservoir's scheduling and control capabilities.

[0117] Those skilled in the art should understand that, in practical applications, some of the preset parameters, threshold conditions, and inherent attribute values ​​involved in the embodiments of this invention can be obtained or adjusted by engineers through conventional technical means, experimental statistics, factory calibration, or direct consultation of datasheets and other existing methods. For example, the upper limit of the maximum physical discharge capacity of the reservoir and the allowable range of gate opening and closing rates, and other inherent hardware attribute parameters, can be obtained by directly consulting the factory datasheets of the underlying electromechanical equipment of the water conservancy hub or through conventional mechanical performance calibration tests; the operational tolerances and control thresholds, such as the safe discharge of downstream control stations and the pre-configured warning flow, can be conventionally calculated and adjusted based on national water conservancy industry flood control standards and specifications or historical hydrological frequency statistics; the algorithm hyperparameters, such as the soft update coefficient, annealing factor, and exploration noise attenuation rate, in the pre-training process of the reinforcement learning model can be evaluated and determined based on conventional grid search cross-validation methods in the field of deep learning and engineering parameter tuning experience. The specific values ​​of these parameters will be adaptively adjusted according to changes in hardware selection and application scenarios, and their acquisition process is common knowledge to those skilled in the art and should not be regarded as a limitation on the integrity and scope of protection of the core technical solution of this invention.

[0118] Example 10: A reservoir flood control scheduling method based on policy reinforcement learning, comprising the following steps:

[0119] Step 1: Collect reservoir scheduling and operation parameters, historical flood processes, inflow runoff forecast processes, and control parameters of downstream flood protection targets;

[0120] Step 2: Construct the reservoir simulation scheduling environment. Based on the reservoir scheduling operation parameters and the control parameters of downstream protected objects, a reservoir simulation scheduling environment is constructed using the water balance equation. This simulation environment uses a time step... Iterative calculations are performed when the current water level above the dam is input. Inbound flow and discharge flow Output the water level above the reservoir dam at the next moment. and rewards .

[0121] Step 3: Construct a policy-based reinforcement learning environment for reservoir flood control scheduling. The reservoir flood control scheduling problem will be transformed into a problem consisting of a state space. Action space ,award And a Markov Decision Process (MDP) constructed from state transition functions. The decision-making action of the reservoir management agent regarding the discharge flow. The adjusted status can be updated and fed back by the reservoir simulation scheduling environment. and rewards This guides the reservoir scheduling agent to generate new action decisions based on feedback, forming a continuous decision-making and learning cycle.

[0122] Step 4: Construct the agent based on the Deterministic Policy Gradient (DPG) method. Based on historical flood information, the agent is constructed using the Deterministic Policy Gradient (DPG) method, which consists of a policy network. and value network The value network is used to evaluate the performance of the policy network and helps train the policy network. When the input state... At that time, the policy network outputs a determined discharge flow action. Value network Based on input status and the action of draining flow Output value This reflects the quality of the flow discharge action; if the strategy network... Output discharge flow action The better, the more valuable. The larger it is. After training, the value network... It will no longer be used, by the policy network. The intelligent agent controlling reservoir scheduling formulates reservoir scheduling strategies.

[0123] Step 5: Construct an experience replay pool and train the value network and policy network. The reservoir flood control agent from Step 4 interacts with the reservoir flood control environment from Steps 2 and 3, and the agent's trajectory is organized into... The tuples are then added to the experience replay pool. This is used during the training of the policy network. Only the states are used at that time. Training the value network All four elements are needed: .

[0124] Step 6: Deployment and Operation Evaluation of the Reservoir Flood Control Scheduling Intelligent Agent. The trained flood control scheduling intelligent agent is deployed, and the model is evaluated using a series of flood events.

[0125] Preferably, the reservoir scheduling and operation parameters mentioned in step 1 include normal water level, dead water level, flood limit water level, water level-reservoir capacity curve, tailwater level-discharge curve, and minimum discharge flow; the downstream flood control protection object control parameters include the embankment protection standard and the corresponding safe discharge.

[0126] Preferably, the reservoir water balance equation is used for calculation during the reservoir simulation scheduling environment iteration in step 2, which can be expressed as:

[0127] (1)

[0128] (2)

[0129] In the formula: Represents the reservoir water level and storage capacity curve; Indicates reservoir time period Storage capacity.

[0130] Preferably, the state space described in step 3 Includes the current reservoir water level, current inflow rate, current outflow rate, downstream control station flow rate, and future data. Forecast inflow at each time point.

[0131] Preferably, the action space described in step 3 The decision variable in the data is the reservoir discharge flow. The discharge flow rate needs to take into account the reservoir's discharge capacity at the current water level and the reservoir's minimum discharge flow rate.

[0132] Preferably, the reward function described in step 3 The maximum flood control safety guarantee criterion is adopted to maximize the reserved flood control capacity while ensuring the safety of the downstream flood control section. The calculation method is as follows:

[0133] (1)

[0134] In the formula: Indicates the normal water level of the reservoir, in meters (m); Reward Indicates the reservoir at any time Remaining flood control capacity

[0135] Preferably, the value network described in step 4 Action value function The approximation of the action value function can be expressed as:

[0136] (2);

[0137] (3);

[0138] In the formula: Indicates the first The reward of time It is obtained by summing the cumulative returns from subsequent time points; Indicates the total time period for calculation; for reporting Regarding variables The action value function can be obtained by calculating the conditional expectation. .

[0139] Preferably, the evaluation model described in step 6 requires calculating the remaining flood control capacity of the reservoir and determining whether the flow process at the downstream flood control control station exceeds the flood control standard, which can be expressed as:

[0140] (4);

[0141] (5);

[0142] In the formula: This indicates the remaining capacity of the flood control reservoir during a particular flood event.

[0143] Preferably, in S4, when constructing an agent based on DDPG, in addition to the main policy network... and the main value network In addition, a corresponding target policy network is constructed. and target value network The target network has the same structure and initial parameters as the main network. During training, the target network parameters are updated softly to track the main network.

[0144] ; in This is the soft update coefficient, with a range of values. .

[0145] Both the policy network and the value network employ a fully connected structure. The policy network contains two hidden layers, each with 128 neurons. The activation function for the hidden layers is ReLU. The activation function and constraint mapping method for the output layer are described in the innovative improvement section. The value network inputs the concatenated state vector and action value. It also contains two hidden layers (128 neurons, ReLU activation), and the output layer has linear activation, outputting a scalar Q-value.

[0146] During training, exploratory noise is added to the deterministic actions output by the policy network: ,in , Linear decay occurs during training to facilitate the transition from exploration to exploitation.

[0147] S5 further specifies that the value network is trained with the goal of minimizing the TD error, and the batch size is randomly sampled from the experience replay pool each time. (e.g., 128) tuples, calculate the loss:

[0148] ; Among them, TD target The policy network is updated along the direction of increasing value:

[0149] ;

[0150] Preferably, the reward function in S3 Further:

[0151] Based on the existing remaining flood control capacity, an adaptive penalty term for downstream safety constraints is introduced to construct a composite reward function:

[0152] In the formula: This indicates the total flood control capacity of the reservoir (m³). This is the flood control limit water level; downstream flood control station period Flow rate (m³ / s); This represents the safe discharge capacity (m³ / s) for downstream flood control stations. The first term is the normalized remaining flood control capacity (dimensionless), and the second term is the normalized penalty term for exceeding the standard (dimensionless). Both terms have the same dimensions.

[0153] Penalty coefficient Adaptive adjustment based on how close the downstream flow rate is to the safe threshold:

[0154] ; In the formula: The downstream warning flow rate (m³ / s) can be taken as... (like ); The base penalty coefficient (e.g., take 10); The penalty growth rate parameter (e.g., 3~5).

[0155] When the downstream flow rate is below the warning value, the penalty coefficient remains a small constant. The agent's primary objective is to maximize remaining storage capacity. When downstream flow exceeds a warning threshold, the penalty coefficient increases exponentially with flow, creating a "soft safety barrier" effect that causes the agent to proactively reduce downstream flow when approaching the safety limit. This exponential growth ensures that the penalty increases much faster than linear penalties, effectively preventing the agent from tentatively breaching the safety boundary.

[0156] Preferably, the output layer of the policy network in S4 is further:

[0157] To ensure that the outflow output by the strategy network always meets the physical constraints of the reservoir, the output layer of the strategy network is designed as a constraint mapping structure. The last layer of the strategy network outputs the original values. Mapped to via the Sigmoid activation function After the interval, linearly scale to the legal discharge range corresponding to the current water level:

[0158] In the formula: For the Sigmoid function; This is the minimum discharge flow rate of the reservoir (m³ / s). For the reservoir at the current water level The maximum discharge capacity (m³ / s) is calculated from the tailrace level discharge curve.

[0159] In this embodiment, the network output naturally satisfies... No post-pruning is required, avoiding the gradient vanishing problem caused by pruning; upper limit of leakage. As the water level changes dynamically, the strategy network automatically adapts to the discharge capacity constraints under different water levels; (c) The gradient of the Sigmoid function is the largest in the middle of the output interval and tends to zero near the boundary, which is consistent with the engineering principle of "prioritizing the intermediate discharge and cautiously approaching the extreme discharge" in actual scheduling.

[0160] Preferably, the experience replay pool in S5 is further defined as follows:

[0161] To concentrate training resources on critical safety periods such as flood peaks, the standard uniform sampling experience replay pool was replaced with a priority experience replay pool based on flood severity. Each experience tuple... The sampling priority is defined as follows:

[0162] In the formula: The TD error of this experience is dimensionless, as the reward has been normalized. This is the flood severity weighting coefficient (e.g., 1.0). This represents the downstream control station flow rate (m³ / s) corresponding to this experience. The first item is the TD error priority for the standard priority experience playback, and the second item is the additional priority based on flood severity.

[0163] The sampling probability is:

[0164] In the formula The priority index (e.g., 0.6) controls the degree of interpolation between priority sampling and uniform sampling. To correct for distribution bias introduced by priority sampling, importance sampling weights are applied to the loss function.

[0165] in Linearly anneal from the initial value (e.g., 0.4) to 1.0.

[0166] Preferably, in order to increase the flow of downstream control stations It has a clear physical origin and, in addition to the reservoir's water balance, includes the Muskingum evolution in the dam-station section. This is based on the actual discharge during the previous period. Current leakage The previous period's evolution outflow As input, the outflow component of the river channel is obtained through evolution, and then the inflow component of the interval is superimposed. Get site traffic:

[0167] In the formula: The storage capacity parameter for the river section (h, approximately equal to the flood wave propagation time). As the flow proportion factor, (dimensionless); The inflow (m³ / s) between the dam site and the control station can be estimated by forecasting or area analogy. The parameter values ​​satisfy... To ensure The evolutionary physics is reasonable.

[0168] Preferably, S2 can also be: the above evolution method can also employ a lag-and-route algorithm or a pure lag translation with the same data structure and similar functions. ( (Time delay steps), both inputs ( ) and output ( The interfaces are consistent and interchangeable.

[0169] Preferably, step S4 can also be: first, give the downstream safety forward-looking discharge limit based on the evolution relationship. : Request for urination with time lag After the evolution, the downstream site traffic will not exceed the safe discharge capacity. To ensure a safety margin, a conservative shift estimate that ignores channel attenuation is adopted. Attenuation will only lower the actual peak value, hence this approach is conservative.

[0170] In the formula for The predicted inflow value (m³ / s) for the time interval. Let the lower and upper bounds of the feasible region be:

[0171] Raw output of the last layer of the policy network Mapped to this interval via Sigmoid:

[0172] In the formula: This is the upper limit of the discharge capacity at the current water level (taken from the discharge capacity curve, i.e., the "non-tailwater level discharge curve" mentioned in the previous round, with correction). , Permissible amplitude variation (m³ / s per second) for rising / receding water levels The value is determined by the gate opening and closing rate and the allowable amplitude of the downstream river channel.

[0173] Empty set rollback rule (guaranteeing feasibility under any operating condition and not violating physical hard constraints): when When the downstream soft safety limit is removed because the inflow is too large and even the minimum discharge cannot guarantee that the downstream will not exceed the limit, the downstream soft limit is removed. ,make Unavoidable exceedances are minimized by soft penalties. In this way, the hard feasible region is responsible for "ensuring downstream safety when feasible", while the soft penalty is responsible for "minimizing exceedances when infeasible".

[0174] Preferably, in step S3, the reward is consistent with the evolution, and the time lag is handled by the value network.

[0175] Downstream traffic in compound rewards The model is now generated consistently from the evolutionary model of Improvement Method 0 (of the same origin as the environment and assessment), and is no longer suspended. Due to the time lag in evolution, The downstream consequences are reflected in This is not included in the single-step reward look-ahead calculation. (To avoid violating Markov property), it relies on the bootstrapping backhaul of the DDPG value network to attribute the delayed consequences to the current action; to ensure that the state satisfies Markov property, the actual discharge of the previous period is explicitly stated. (i.e., the current outbound flow in the status) is included in the status, so that A function that becomes the state.

[0176] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A reservoir flood control scheduling method based on policy reinforcement learning, characterized in that, include: Acquire reservoir operating parameters, flood forecasting processes, and downstream flood control parameters; Based on the reservoir operating parameters and the downstream flood control parameters, a reservoir-river simulation environment is constructed. Based on the reservoir-river simulation environment and the flood forecasting process, a state vector containing multi-dimensional hydrological state information is extracted, and the action space representing the reservoir discharge flow is determined. The state vector is input into a pre-configured policy network. The output layer of the policy network is used to calculate the dynamic feasible region by combining the physical boundary and safety boundary fed back from the reservoir-river simulation environment. The original output of the policy network is then mapped into the dynamic feasible region to obtain the discharge flow action. The downstream flow action is input into the reservoir-river simulation environment for state deduction, the state vector is updated, and a scheduling reward including downstream exceedance penalty is generated. Based on continuous interactive decision-making, the reservoir flood control scheduling strategy is output.

2. The method according to claim 1, characterized in that, In the reservoir-river simulation environment, the flow rate at the downstream flood control station is determined in the following way: Based on the aforementioned flood forecasting process, the inflow between the dam site and the control station is extracted; Based on the river channel physical calculation rules, the downstream flow action is converted into river flood evolution components; The flow rate at the downstream flood control station is obtained by superimposing the river flood evolution component with the inflow of the interval.

3. The method according to claim 2, characterized in that, The conversion of the downstream flow action into river flood evolution components is specifically achieved through recursive calculations using the Muskingen evolution equation: During the recursive calculation process, the river flood evolution component is determined based on the discharge flow action, the actual discharge flow of the previous time period generated by the simulation environment iteration, and the river flood evolution component of the previous time period. The evolution coefficients of the Muskingen evolution equation are set according to the physical constraint that the product of the pre-configured river section storage parameter and the pre-configured flow proportion factor has a time step of not less than twice.

4. The method according to claim 2, characterized in that, The conversion of the downstream flow action into river flood evolution components is specifically achieved through a lag algorithm: Based on the pre-configured river channel time delay steps, the downstream flow action is subjected to time delay translation processing to obtain the river channel flood evolution component.

5. The method according to claim 1, characterized in that, The multidimensional hydrological state information specifically includes: The reservoir water level and inflow rate at the current moment are fed back from the reservoir-river simulation environment. Forecast inflow rates at multiple future moments extracted from the aforementioned flood forecasting process; And the actual discharge flow rate of the previous period, fed back from the reservoir-river simulation environment.

6. The method according to claim 1, characterized in that, The dynamic feasible region is defined by its lower bound and upper bound. The lower and upper bounds of the dynamic feasible region are determined by the intersection of the upper limit of the physical discharge capacity of the reservoir at the current water level, the allowable range of the gate opening and closing rate, and the upper limit of downstream safe forward discharge. The downstream safety forward-looking discharge limit is the discharge extreme value pre-calculated to ensure that the flow at the downstream flood control station does not exceed the standard.

7. The method according to claim 6, characterized in that, The downstream safety forward leakage limit is derived in the following way: Extract the pre-configured safe discharge capacity of downstream control stations from the downstream flood control parameters; Based on the flood forecasting process, the interval inflow forecast value corresponding to the river channel time lag time is obtained; Under the conservative estimate of ignoring the attenuation of river floods, the safe discharge capacity of the downstream control station is subtracted from the predicted inflow value of the interval to obtain the upper limit of the safe forward discharge capacity of the downstream.

8. The method according to claim 6, characterized in that, In the process of mapping the original output of the policy network to the dynamic feasible domain to obtain the outflow traffic action, the output layer of the policy network performs the following constraint mapping process: Using a preset activation function, the original output of the policy network is mapped to a standard range to obtain a mapped value; Based on the lower and upper bounds of the dynamic feasible region, the mapping value is linearly scaled so that the final output of the outflow action is strictly limited to the lower and upper bounds of the dynamic feasible region.

9. The method according to claim 6, characterized in that, When determining the lower and upper bounds of the dynamic feasible region, an empty set back-off step is also included: When extreme flood conditions cause the calculated downstream safe forward discharge limit to be less than the lower limit of the amplitude range, the downstream safe forward discharge limit constraint is removed. The upper bound of the dynamic feasible region is redefined as the minimum value between the upper limit of the physical discharge capacity and the upper limit of the amplitude range.

10. The method according to claim 1, characterized in that, The scheduling reward is composed of both reservoir area safety targets and downstream flood control section safety targets, specifically including: The dimensionless residual flood control capacity term characterizing the flood control capacity of a reservoir; In addition, a dimensionless downstream control station flow exceeding penalty item is used to constrain downstream evolution outflow.

Images