A reservoir gate scheduling method and system

Abstract

The present disclosure relates to the technical field of gate scheduling, and provides a reservoir gate scheduling method and system. The method comprises: acquiring reservoir basic information; constructing a target function and constraint conditions of multi-objective optimization scheduling of a reservoir spillway gate based on the reservoir basic information; the target of the target function is to minimize the sum of the maximum reservoir outflow and the average adjustment times of the reservoir spillway gate; a deep reinforcement learning framework is constructed; state variables and action variables of the deep reinforcement learning framework are determined, and a reward function is determined based on the target function and the constraint conditions; the deep reinforcement learning framework is trained by using a proximal policy optimization algorithm, and the trained agent can formulate a reasonable gate scheduling scheme according to a flood process. The method can be used for reservoir flood control and provides a reasonable gate scheduling scheme for dispatch personnel.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of gate scheduling, and particularly relates to a reservoir gate scheduling method and system. BACKGROUND

[0002] Among all natural disasters, the impact of flood disasters is the most serious, and at the same time, the proportion of the global population facing flood risk is increasing year by year. In the face of increasingly severe flood risk, researchers around the world have carried out a large amount of research work, aiming to deeply understand the formation mechanism, prediction method and prevention and control technology of flood, and provide scientific basis for flood risk management and response.

[0003] Flood disasters have always been an important factor restricting the social and economic development of China. According to statistics, the economic loss caused by floods ranks first among all kinds of disasters, which seriously affects the stability of the national economy and the sustainable development of society. The frequent occurrence of flood disasters not only has a great impact on agricultural production, urban construction and infrastructure, but also directly threatens the safety of people's life and property.

[0004] As one of the widely used measures for flood control in China, reservoirs play an important role in realizing safe and efficient utilization of flood resources. Reservoir flood control scheduling adjusts the outflow of the reservoir to realize the regulation and storage of flood, so as to reduce flood damage, protect the safety of protected objects and improve the comprehensive benefits of the reservoir.

[0005] However, in the traditional reservoir flood control scheduling, the decision variable is the outflow, and the flow is not allocated to the gate. Due to the operation rules of the reservoir spillway gate, such as opening and closing sequence and symmetrical opening and closing, the allocation of outflow to the gate will produce errors, and the traditional scheduling scheme has further optimization space. SUMMARY

[0006] To solve the above problems, the present disclosure provides a reservoir gate scheduling method and system, which constructs a deep reinforcement learning model, determines multiple constraint conditions, establishes a reward function using the constraint conditions and the objective function, trains the model to output the gate opening quantity probability distribution, and opens the gate according to the distribution, thereby solving the problem of error in the allocation of outflow to the gate.

[0007] The following is the technical solution of the present disclosure:

[0008] A reservoir gate scheduling method, characterized in that it comprises:

[0009] obtaining reservoir basic information;

[0010] The target function and constraint condition of the multi-objective optimization scheduling of the reservoir spillway gate are constructed based on reservoir basic information; the target of the target function is to minimize the sum of the maximum reservoir outflow and the average adjustment times of the reservoir spillway gate; the constraint conditions include water balance constraint, reservoir water level constraint, reservoir storage curve constraint, maximum discharge capacity constraint, outflow variation amplitude constraint, and spillway discharge constraint;

[0011] A deep reinforcement learning framework is constructed; the state variable of the deep reinforcement learning framework is the reservoir water level at the beginning of the t period, the inflow of the reservoir at the t period, the outflow of the reservoir at the t-1 period, and the gate combination at the t period; the action variable is the gate combination at the t period; the reward function of the deep reinforcement learning framework is constructed based on the target function and the constraint condition;

[0012] The construction of the reward function includes:

[0013] The basic reward function is determined according to the target function, and the basic reward function is:

[0014] An immediate reward is provided in each time step, and the reservoir water level constraint is taken as a penalty term in the reward function, so that the final reward function is:

[0015]

[0016] In the formula, α' and β' are weight coefficients; c1 and c2 are penalty coefficients;

[0017] The deep reinforcement learning framework is trained by using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model; and the trained deep reinforcement learning model is used for reservoir gate scheduling.

[0018] Further,

[0019] The constraint condition of the reservoir spillway gate scheduling is:

[0020] The water balance constraint is:

[0021] V t+1 = V t +(I t -Q t )·Δt

[0022]

[0023] In the formula, V t is the reservoir storage at the beginning of the t period, m 3 ; I t is the reservoir inflow at the t period, m3 / s; Q t is the reservoir outflow at the t period, m 3 / s; At is the duration of the time period t, h; M is the number of gate types; J m is the number of gates of the mth type; q j,m,t is the discharge of the jth gate of the mth type in the time period t, m 3 / s;

[0024] Reservoir water level constraints:

[0025] Z min ≤ Z t ≤ Z max

[0026] In the formula, Z t is the initial reservoir water level in the time period t, m; Z max and Z min are the upper and lower limits of the reservoir water level, respectively, m;

[0027] Reservoir capacity curve constraints:

[0028] Z t = f zv (V t )

[0029] In the formula, f zv (·) is the functional relationship between the reservoir water level and the storage capacity;

[0030] Maximum discharge capacity constraints:

[0031] Q t ≤ Q t,max

[0032] In the formula, Q t,max is the maximum discharge capacity of the reservoir in the time period t, m 3 / s;

[0033] Discharge amplitude constraints:

[0034] |Q t+1 -Q t |≤ ΔQ

[0035] In the formula, ΔQ is the allowed amplitude of the reservoir discharge between adjacent time periods, m 3 / s;

[0036] Spillway discharge constraints:

[0037] q j,m,t = f m,qz (O j,m,t ,Z t )

[0038] In the formula, O j,m,t is the opening of the jth gate of the mth type in the time period t; f m,qz(·) is the discharge of the mth type of gate and the function relationship of gate opening, upstream water level;

[0039] Further,

[0040] The objective function of the multi-objective optimization scheduling of the reservoir spillway gate is:

[0041]

[0042] In the formula, T is the total scheduling time, h; h; t is the current scheduling period, h; G is the average number of gate adjustments; and α and β are weight coefficients.

[0043] Further,

[0044] In the proximal policy optimization algorithm, the objective function of the policy network is:

[0045]

[0046] The value network expression is:

[0047]

[0048] In the formula: π θ (a t ∣s t ) is the policy of the agent; is the estimated value of the advantage function, calculated using the generalized advantage estimation method; ∈ is a hyperparameter for limiting the policy update amplitude; c H is a hyperparameter for controlling the strength of entropy regularization; V φ (s t ) is the estimated value of the value network for state s t ; R t is the actual return starting from state s t ;

[0049] Further,

[0050] The

[0051] The generalized advantage estimation method is used to calculate The expression is:

[0052]

[0053] In the formula: γ is the discount factor; λ is a hyperparameter for controlling bias and variance.

[0054] Further,

[0055] The deep reinforcement learning framework is trained using the proximal policy optimization algorithm to obtain a trained deep reinforcement learning model; comprising:

[0056] Initialize the parameters of the policy network and the value network;

[0057] Obtain the initial environmental state s0 of the reservoir, namely the inflow and water level at the initial moment, as well as the outflow and gate combination at the previous moment.

[0058] The policy network outputs the probability distribution of different gate combinations based on the initial state, and executes the actions of the reservoir spillway gates according to the probability distribution of different gate combinations. t ;

[0059] Perform action a t Afterwards, a reward value r is obtained. t At this point, the environmental state becomes s t+1 ;

[0060] Experience (s) t ,a t ,r t ,s t+1 Store the data in a buffer, and maximize the buffer capacity when the number of experiences reaches that capacity. To update the parameters of the policy network and minimize To update the parameters of the value network;

[0061] Each time the parameters of the policy network and the value network are updated, the current policy π is affected. θ (a t |s t The training deep reinforcement learning model is then evaluated until the cumulative reward value stabilizes.

[0062] A reservoir spillway gate scheduling system, characterized in that it includes:

[0063] The data acquisition module is used to acquire basic information about the reservoir.

[0064] The framework construction module is used to construct the objective function and constraints for multi-objective optimization scheduling of reservoir spillway gates based on reservoir basic information. The objective function aims to minimize the sum of the maximum outflow from the reservoir and the average number of adjustments to the reservoir spillway gates. The constraints include water balance constraints, reservoir water level constraints, reservoir capacity curve constraints, maximum discharge capacity constraints, outflow variation constraints, and spillway discharge constraints.

[0065] The model building module is used to construct a deep reinforcement learning framework; the state variables of the deep reinforcement learning framework are the reservoir water level at the beginning of time period t, the inflow rate at time period t, the outflow rate at time period t-1, and the gate combination; the action variable is the gate combination at time period t; the reward function of the deep reinforcement learning framework is constructed based on the objective function and constraints.

[0066] The construction of the reward function includes:

[0067] The basic reward function is determined based on the objective function. The basic reward function is as follows:

[0068] An immediate reward is provided at each time step, and the reservoir water level constraint is included as a penalty in the reward function, resulting in the final reward function:

[0069]

[0070] In the formula, α' and β' are weighting coefficients; c1 and c2 are penalty coefficients;

[0071] The model training module is used to train a deep reinforcement learning framework using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model.

[0072] The control module is used to schedule reservoir gates using a trained deep reinforcement learning model.

[0073] Compared with the prior art, this disclosure has the following advantages:

[0074] This invention obtains basic information about the reservoir, including the inflow rate, water level and storage capacity curves, and discharge capacity curves of the spillway facilities for a specific time period, to provide a data foundation for subsequent operations. It constructs an objective function that considers both the maximum outflow rate and the number of times the gates are opened and closed, taking into account both flood control safety and gate operating efficiency and lifespan. Then, it uses a deep reinforcement learning model to make automatic decisions on the opening of the gates.

[0075] In the training of the deep reinforcement learning model, a near-end policy optimization algorithm is used, including a policy network and a value network. The policy network outputs the probability distribution of the number of gates to be opened based on the reservoir state, and the value network outputs the value of the current state. Constraints including various actual situations are determined and a reward function is established. The upper and lower limits of water level are used as penalty terms and intermediate rewards are provided to solve the problem of reward sparsity. The model is trained by inputting relevant time period information to output the probability distribution. Finally, the gates are opened according to the probability distribution, thereby realizing the scheduling of reservoir spillway gates under the constraints and solving the problems of traditional scheduling schemes.

[0076] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 A schematic diagram of the method of the present invention is shown;

[0079] Figure 2 The flowchart of the model training process is shown;

[0080] Figure 3 A flowchart of the model testing process is shown;

[0081] Figure 4 The cumulative reward curve is shown;

[0082] Figure 5 The outbound flow rate process curve is shown;

[0083] Figure 6 The diagram shows the water level process curve of the reservoir;

[0084] Figure 7 A schematic diagram showing the number of deep holes opened in each time period is shown. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0086] like Figure 1 This is a schematic diagram of the method of the present invention. Specific implementation details of the present invention include:

[0087] 1. Step 1: Obtain basic information about the reservoir.

[0088] Basic information includes: reservoir inflow, water level and storage capacity curves, and discharge capacity curves of spillway facilities for a specific time period.

[0089] 2. Step Two: Determine the objective function.

[0090] The maximum outflow and the number of gate openings / closings during the scheduling period are considered simultaneously. The maximum peak reduction criterion, commonly used in reservoir flood control scheduling, is adopted, while inflow between sections is ignored. The maximum reservoir outflow is used as the objective function.

[0091] Furthermore, frequent opening and closing of gates during the flood season shortens their lifespan and increases operating costs. Therefore, the average number of times the reservoir spillway gates are adjusted during the flood season is used as another objective function.

[0092] At the same time, the two objectives are integrated into a single overall objective function using the weighting method, as follows:

[0093]

[0094] In the formula, T is the total scheduling duration (h); t is the current scheduling period (h); G is the average number of gate adjustments; and α and β are weighting coefficients.

[0095] 3. Step 3: Determine the constraints.

[0096] The constraints include:

[0097] 1) Water balance constraints:

[0098] V t+1 =V t +(I t -Q t )·Δt

[0099]

[0100] In the formula, V t Let m be the reservoir water storage at the beginning of time period t. 3 ;I t Let m be the reservoir inflow rate during time period t. 3 / s;Q t Let m be the reservoir outflow during time period t. 3 / s; Δt is the duration of time interval t, in hours; M is the number of gate types; J m q represents the number of gates of the m-th type; j,m,t Let m be the discharge flow of the j-th gate of type m during time period t. 3 / s;

[0101] 2) Reservoir water level constraints

[0102] Z min ≤Z t ≤Z max

[0103] In the formula: Z t Z represents the reservoir water level at the beginning of time period t, in meters. max and Z min These represent the upper and lower limits of the reservoir water level, in meters (m).

[0104] 3) Storage capacity curve constraint

[0105] Zt =f zv (V t )

[0106] In the formula: f zv (·) represents the functional relationship between the reservoir water level and the water storage capacity.

[0107] 4) Maximum discharge capacity constraint

[0108] Q t ≤Q t,max

[0109] In the formula: Q t,max Let m be the maximum outflow from the reservoir during time period t. 3 / s.

[0110] 5) Constraints on the variation of outbound flow rate in adjacent time periods

[0111] |Q t+1 -Q t |≤ΔQ

[0112] In the formula: ΔQ is the allowable variation of reservoir outflow between adjacent time periods, m 3 / s.

[0113] 6) Spillway discharge constraints

[0114] q j,m,t =f m,qz (O j,m,t Z t )

[0115] In the formula, O j,m,t f represents the opening degree of the j-th gate of type m during time period t; m,qz (·) represents the functional relationship between the discharge capacity of the m-th type of gate and the gate opening and upstream water level;

[0116] 4. Step Four: Determine the basic elements of the reinforcement learning model, including action variables, state variables, and reward function.

[0117] Specifically, the gate combination is selected as the decision variable, and it is defined as the agent's action variable a. t :

[0118] a t ={N t}

[0119] The reservoir water level at the beginning of time period t, the inflow rate at time period t, the outflow rate at time period t-1, and the combination of the reservoir spillway gates at time period t-1 are selected as the state variables of the environment. State s t Defined as:

[0120] s t={I t Z t Q t-1 N t-1}

[0121] Reward is the most important element in reinforcement learning. A good reward function can guide the agent towards...

[0122] Training in the "correct direction" allows the agent to gradually learn the optimal policy. Conversely, an inappropriate reward function will hinder the agent from learning a reasonable policy. Typically, the reward function is directly related to the objective function. Since agents tend to learn policies that maximize cumulative rewards, the basic reward in this invention...

[0123] The function is described as follows:

[0124]

[0125] In reinforcement learning training, the agent typically does not receive rewards. The environment only provides a reward when a specific goal is achieved. In this invention, according to the aforementioned reward function, the environment only provides a reward to the agent at the end of the scheduling period. This reward reflects the maximum outflow from the reservoir and the number of gate adjustments during the entire scheduling period. This form of reward function makes it difficult for the agent to learn a reasonable strategy, resulting in a reward sparsity problem.

[0126] A common approach to mitigating this problem is reward plasticity, where the environment provides intermediate rewards to the agent at each time step, guiding the agent to learn appropriate policies. These intermediate rewards are typically much smaller than the final reward to emphasize the final reward value. Furthermore, the action chosen by the agent must satisfy the constraints of the reservoir spillway gate. In this invention, the upper and lower limits of the reservoir water level are incorporated as penalty terms into the reward function. In summary, the calculation formula for the reward function of this invention is as follows:

[0127]

[0128] In the formula: α' and β' are weighting coefficients; c1 and c2 are penalty coefficients.

[0129] 5. Step Five: Determine the deep reinforcement learning algorithm and optimize the specific inbound traffic process based on the various elements of reinforcement learning.

[0130] This invention selects the Proximal Policy Optimization (PPO) algorithm to solve the problem.

[0131] The proximal policy optimization algorithm is an improvement on the policy gradient method, primarily addressing the stability and efficiency issues of the policy gradient method during policy updates. The proximal policy optimization algorithm improves training stability by limiting the magnitude of policy updates. It introduces a "shearing" mechanism to prevent policy updates from becoming too rapid, thus avoiding policy collapse. The core idea of ​​the proximal policy optimization algorithm is to use the Trust Region Methods to control the size of each policy update, ensuring that the difference between the old and new policies is not too large.

[0132] The proximal policy optimization algorithm consists of two neural networks: a policy network and a value network. The policy network takes the state as input and outputs the probability of each possible action. The value network takes the state as input and outputs the value of that state, which is the agent's expected cumulative reward in that state.

[0133] The update objective of the policy network is to maximize the shearing function, while the update objective of the value network is to minimize the mean squared error loss. The corresponding functional forms are as follows:

[0134]

[0135] In the formula: π θ (a t |s t () is the agent's strategy; It is an estimate of the advantage function; ∈ is a very small value (e.g., 0.2) used to limit the magnitude of policy updates; V φ (s t ) is the value network for state s t The estimated value of R; t From state s t The initial actual return.

[0136] Calculate using the generalized dominance estimation method The expression is:

[0137]

[0138] In the formula: γ is the discount factor; λ is a hyperparameter used to control bias and variance.

[0139] 6. Step Six: Determine the hyperparameters of the deep reinforcement learning algorithm, including the total number of training iterations, learning rate, batch size, hidden layer size, etc.

[0140] 7. Step Seven: Train the model. The agent continuously interacts with the environment, updating the neural network parameters based on the loss function. Each time the parameters are updated, the current policy is evaluated, and a cumulative reward curve is output.

[0141] Specifically, the training is conducted based on the basic idea of ​​the near-end policy optimization algorithm, and the specific training process is as follows:

[0142]

[0143] During the training phase, the agent calculates the probability distribution of actions based on the policy network and selects actions according to this probability distribution. Considering constraints, when the agent selects an action, it no longer chooses from the overall action space, but rather based on the currently received environmental state s. t Choose an action from the corresponding action space.

[0144] like Figure 2 This is a flowchart of the model training process of the present invention.

[0145] 8. Step 8: Based on the cumulative reward curve in Step 7, select the neural network parameters with the largest cumulative reward.

[0146] 9. Step Nine: Use the network parameters from Step Eight as the network parameters for scheduling to determine the scheduling process.

[0147] The evaluation of the current strategy described in step 7 and the determination of the scheduling process in step 9, the interaction process between the agent and the environment are as follows: Figure 3 As shown, the agent calculates the probability of all actions in the corresponding state based on the policy network, and selects the action with the highest probability from the corresponding action space.

[0148] The following are specific examples of the present invention:

[0149] Taking a certain reservoir as an example, the specific steps include the following:

[0150] Step 1: Input basic reservoir information. Select the period from 00:00 on August 11, 2020 to 24:00 on August 25, 2020 as the start and end time for scheduling, with a time resolution of 2 hours. Import the inflow sequence, water level and storage capacity curve, and discharge capacity curve of the spillway facilities for a certain reservoir. The discharge capacity curve of the spillway facilities is shown in Table 1.

[0151] Table 1 Discharge Capacity of Drainage Facilities

[0152]

[0153] Step 2: Determine the objective function and select appropriate values ​​for α and β.

[0154] Step 3: Determine the constraints. Select appropriate upper and lower limits of reservoir water level, upper limit of outflow, maximum outflow variation in adjacent time periods, initial reservoir water level, initial outflow, and gate combination.

[0155] Step 4: Determine the basic elements of the reinforcement learning model. Select appropriate values ​​for α', β', c1, and c2.

[0156] Step 5: Determine the deep reinforcement learning algorithm. The Proximal Policy Optimization (PPO) algorithm is selected for solving the problem.

[0157] Step 6: Determine the hyperparameters for deep reinforcement learning. The total number of training iterations is set to 2000, the learning rate to 0.0003, the batch size to 1024, and the hidden layer size to 128.

[0158] Step 7: Train the model. The agent continuously interacts with the environment. Each time the parameters are updated, the current policy is evaluated, and a cumulative reward curve is output, in the form of... Figure 4 As shown.

[0159] Step 8: Based on the cumulative reward value obtainable by each strategy in Step 4, select the network parameter with the largest cumulative reward value.

[0160] Step 9: Using the network parameters selected in Step 8, output the scheduling process, such as... Figure 5 , 6 As shown in Figures 7 and 8.

[0161] Table 2 Scheduling Results

[0162]

[0163] During the flood event from August 11 to August 25, 2020, the optimized scheduling reduced the maximum outflow by 4.34% compared to the actual scheduling. The number of deep gates opened under the optimized scheduling scheme changed from 2 at the start of the scheduling to 4, 6, and 8, and then remained unchanged. Compared to the actual scheduling process, the optimized scheduling effectively controlled the maximum outflow and ensured flood control safety. Compared to the actual scheduling process, the optimized scheduling scheme initially deployed a larger number of gates and achieved a larger outflow, keeping the water level at a lower level. Except for the first few scheduling periods, the number of gates opened under the optimized scheduling scheme remained almost constant, and the outflow fluctuated within a small range closely following the water level changes. After the flood peak receded, the outflow exceeded the inflow, and the water level gradually decreased. Comprehensive analysis shows that the optimized scheduling scheme significantly reduced the number of gate openings and closings compared to the actual scheduling, ensuring the safe operation of the gates during the flood season.

[0164] In summary, this invention addresses the engineering problem of large errors in traditional gate optimization scheduling schemes by utilizing deep reinforcement learning technology to optimize gate operation during the scheduling period. Building upon traditional reservoir flood control optimization scheduling, it considers gate constraints, with the objective function simultaneously taking into account flood control targets and the number of gate openings and closings. Using gate combinations as decision variables, and for a known inflow process, it optimizes the gate operation scheme during the scheduling period, ensuring flood control safety while also making gate operation safe and efficient.

[0165] Based on the method of the present invention, this disclosure also provides a system corresponding to the above method, which includes:

[0166] Obtain basic information about the reservoir;

[0167] Based on the basic information of the reservoir, a multi-objective optimization scheduling objective function and constraints for the reservoir spillway gates are constructed. The objective function aims to minimize the sum of the maximum outflow from the reservoir and the average number of adjustments to the reservoir spillway gates. The constraints include water balance constraints, reservoir water level constraints, reservoir capacity curve constraints, maximum discharge capacity constraints, outflow variation constraints, and spillway discharge constraints.

[0168] A deep reinforcement learning framework is constructed; the state variables of the deep reinforcement learning framework are the reservoir water level at the beginning of time period t, the inflow rate at time period t, the outflow rate at time period t-1, and the gate combination; the action variable is the gate combination at time period t; the reward function of the deep reinforcement learning framework is constructed based on the objective function and constraints.

[0169] The construction of the reward function includes:

[0170] The basic reward function is determined based on the objective function. The basic reward function is as follows:

[0171] An immediate reward is provided at each time step, and the reservoir water level constraint is included as a penalty in the reward function, resulting in the final reward function:

[0172]

[0173] In the formula, α' and β' are weighting coefficients; c1 and c2 are penalty coefficients;

[0174] A deep reinforcement learning framework is trained using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model; the trained deep reinforcement learning model is then used for reservoir gate scheduling.

[0175] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for scheduling reservoir gates, characterized in that, include: Obtain basic information about the reservoir; Based on the basic information of the reservoir, a multi-objective optimization scheduling objective function and constraints for the reservoir spillway gates are constructed. The objective function aims to minimize the sum of the maximum outflow from the reservoir and the average number of adjustments to the reservoir spillway gates. The constraints include water balance constraints, reservoir water level constraints, reservoir capacity curve constraints, maximum discharge capacity constraints, outflow variation constraints, and spillway discharge constraints. A deep reinforcement learning framework is constructed; the state variables of the deep reinforcement learning framework are the reservoir water level at the beginning of time period t, the inflow rate at time period t, the outflow rate at time period t-1, and the gate combination; the action variable is the gate combination at time period t. The reward function of the deep reinforcement learning framework is constructed based on the objective function and constraints; The construction of the reward function includes: The basic reward function is determined based on the objective function. The basic reward function is as follows: An immediate reward is provided at each time step, and the reservoir water level constraint is included as a penalty in the reward function, resulting in the final reward function: In the formula, α, β, , These are the weighting coefficients; G represents the maximum outflow rate up to time t; G represents the number of gate adjustments. Let M be the instant reward at step t; M is the number of gate types. Let m be the number of gates of type m; Let be the opening degree of the j-th gate of the m-th class at time t; I is the gate adjustment indication function; This is a water level constraint penalty factor; This is the penalty coefficient, where i = 1 or 2; This is the function to indicate the end of the scheduling process. This is the lower limit of the reservoir water level; The actual water level of the reservoir at step t; The upper limit of the reservoir water level is T; T is the total time step during the scheduling period. A deep reinforcement learning framework is trained using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model; this model is then used for reservoir gate scheduling. The process of training a deep reinforcement learning framework using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model includes: Initialize the parameters of the policy network and the value network; Obtain the initial environmental state s0 of the reservoir, namely the inflow and water level at the initial moment, as well as the outflow and gate combination at the previous moment. The strategy network outputs the probability distribution of different gate combinations based on the initial state, and executes the actions of the reservoir spillway gates according to the probability distribution of different gate combinations. After performing the action "at", a reward value is obtained. At this point, the environment state becomes st+1; Store the experience (st, at, rt, st+1) into a buffer. When the number of experiences reaches the buffer capacity, maximize the buffer. To update the parameters of the policy network and minimize To update the parameters of the value network; Each update to the parameters of the policy network and the value network affects the current policy. The evaluation continues until the accumulated reward value stabilizes, at which point a well-trained deep reinforcement learning model is obtained. in, , , It is the agent's strategy; It is an estimate of the dominance function; It is a hyperparameter used to limit the magnitude of policy updates; It is a hyperparameter used to control the strength of entropy regularization; ; The expected value at time step t; It is a value network for state The estimated value; From state The initial actual return.

2. The reservoir gate scheduling method according to claim 1, characterized in that, The constraints for the operation of the reservoir spillway gates are as follows: Water balance constraints: In the formula, Let t be the reservoir's water storage capacity at the beginning of time period t, in m³. Let t be the reservoir inflow rate in m³ / s. Let t be the reservoir outflow rate during time period t, in m³ / s; t represents the duration of time period t, in hours; M represents the number of gate types. Let m be the number of gates of the m-th type; Let be the discharge capacity of the j-th gate of the m-th type during time period t, in m³ / s; Reservoir water level constraints: In the formula, Let t be the reservoir water level at the beginning of time period t, in meters. and These represent the upper and lower limits of the reservoir water level, in meters (m). Storage capacity curve constraints: In the formula, This represents the functional relationship between the reservoir water level and the water storage capacity. Maximum discharge capacity constraint: In the formula, Let t be the maximum discharge capacity of the reservoir during time period t, in m³ / s; Outbound flow rate variation constraints: In the formula, The allowable variation in reservoir outflow between adjacent time periods, in m³ / s; Spillway discharge constraints: In the formula, Let be the opening degree of the j-th gate of the m-th type during time period t; Let be the functional relationship between the discharge capacity of the m-th type of gate and the gate opening and upstream water level.

3. The reservoir gate scheduling method according to claim 1, characterized in that, The objective function for the multi-objective optimization scheduling of the reservoir spillway gates is: In the formula, T is the total scheduling duration, in hours (h). t represents the current scheduling period, h; G represents the average number of gate adjustments. and These are the weighting coefficients.

4. The reservoir gate scheduling method according to claim 1, characterized in that, In the aforementioned near-end policy optimization algorithm, the objective function of the policy network is: The value network expression is: In the formula: ; It is the agent's strategy; It is an estimate of the dominance function, calculated using the generalized dominance estimation method; It is a hyperparameter used to limit the magnitude of policy updates; It is a hyperparameter used to control the strength of entropy regularization; It is a value network for state The estimated value; From state The initial actual return.

5. A reservoir gate scheduling method according to claim 4, characterized in that, The Calculate using the generalized dominance estimation method The expression is: In the formula: It is a discount factor; It is a hyperparameter used to control bias and variance.

6. A reservoir spillway gate scheduling system, characterized in that, include: The data acquisition module is used to acquire basic information about the reservoir. The framework construction module is used to construct the objective function and constraints for multi-objective optimization scheduling of reservoir spillway gates based on reservoir basic information. The objective function aims to minimize the sum of the maximum outflow from the reservoir and the average number of adjustments to the reservoir spillway gates. The constraints include water balance constraints, reservoir water level constraints, reservoir capacity curve constraints, maximum discharge capacity constraints, outflow variation constraints, and spillway discharge constraints. The model building module is used to build a deep reinforcement learning framework; the state variables of the deep reinforcement learning framework are the reservoir water level at the beginning of time period t, the inflow rate at time period t, the outflow rate at time period t-1, and the gate combination; the action variable is the gate combination at time period t. The reward function of the deep reinforcement learning framework is constructed based on the objective function and constraints; The construction of the reward function includes: The basic reward function is determined based on the objective function. The basic reward function is as follows: An immediate reward is provided at each time step, and the reservoir water level constraint is included as a penalty in the reward function, resulting in the final reward function: In the formula, α, β, , These are the weighting coefficients; G represents the maximum outflow rate up to time t; G represents the number of gate adjustments. Let M be the instant reward at step t; M is the number of gate types. Let m be the number of gates of type m; Let be the opening degree of the j-th gate of the m-th class at time t; I is the gate adjustment indication function; This is a water level constraint penalty factor; This is the penalty coefficient, where i = 1 or 2; This is the function to indicate the end of the scheduling process. This is the lower limit of the reservoir water level; The actual water level of the reservoir at step t; The upper limit of the reservoir water level is T; T is the total time step during the scheduling period. The model training module is used to train a deep reinforcement learning framework using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model. The control module is used to schedule reservoir gates using a trained deep reinforcement learning model. The process of training a deep reinforcement learning framework using a proximal policy optimization algorithm to obtain a trained deep reinforcement learning model includes: Initialize the parameters of the policy network and the value network; Obtain the initial environmental state s0 of the reservoir, namely the inflow and water level at the initial moment, as well as the outflow and gate combination at the previous moment. The strategy network outputs the probability distribution of different gate combinations based on the initial state, and executes the actions of the reservoir spillway gates according to the probability distribution of different gate combinations. After performing the action "at", a reward value is obtained. At this point, the environment state becomes st+1; Store the experience (st, at, rt, st+1) into a buffer. When the number of experiences reaches the buffer capacity, maximize the buffer. To update the parameters of the policy network and minimize To update the parameters of the value network; Each update to the parameters of the policy network and the value network affects the current policy. The evaluation continues until the accumulated reward value stabilizes, at which point a well-trained deep reinforcement learning model is obtained. in, , , It is the agent's strategy; It is an estimate of the dominance function; It is a hyperparameter used to limit the magnitude of policy updates; It is a hyperparameter used to control the strength of entropy regularization; ; The expected value at time step t; It is a value network for state The estimated value; From state The initial actual return.

Images