Multi-target reservoir gate optimal scheduling method based on hierarchical reinforcement learning

Abstract

The invention discloses a hierarchical reinforcement learning-based multi-target reservoir gate optimal scheduling method, which comprises the following steps of: constructing a reservoir drainage facility gate optimal scheduling model, and determining a target function and a constraint condition; establishing a hierarchical reinforcement learning framework, determining options, an option network and an option internal strategy, determining a reservoir water level by the option network according to information such as reservoir inflow to minimize the maximum reservoir outflow of the reservoir, and determining a gate operation scheme by the option internal strategy according to the reservoir water level determined by the option network to minimize gate adjustment times; and solving the layered reinforcement learning framework by using a competitive double-depth Q network algorithm to obtain a reasonable gate operation scheme. According to the method, gate scheduling is decomposed into two levels of decisions, the problems that the number of gate optimization scheduling decision variables is too large and the solution dimension is too large are effectively solved, a reasonable gate operation scheme can be obtained through rapid optimization, and the maximum output flow of a reservoir and the gate adjustment frequency are effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reservoir flood control, and in particular to a multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning. BACKGROUND

[0002] Flood disasters have become the most serious and widespread natural disasters in the world, and at the same time, the economic losses caused by flood disasters around the world are increasing year by year. As one of the widely used measures for flood control in China, reservoirs play an important role in achieving safe and efficient use of flood resources. Reservoir flood control adjusts the outflow of the reservoir to achieve flood regulation and storage, so as to reduce flood damage, protect the safety of protected objects, and improve the overall benefits of the reservoir.

[0003] However, in traditional reservoir flood control, the decision variable is the outflow, and the flow is not allocated to the gate. In fact, the reservoir controls the outflow by adjusting the gate of the discharge facility, and the gate operation needs to meet strict operation rules, such as symmetrical opening and closing, discrete opening, etc. The reservoir flood optimization scheduling scheme that ignores the gate operation rules has further optimization space.

[0004] Due to the large number of gate types and quantities, and the existence of strict operation rules, it is extremely difficult to solve the problem of reservoir gate optimization scheduling; taking Gezhouba Reservoir as an example, the gate opening of the left area, middle area and right area of the Erjiang discharge gate is 0m~12.5m, and the gate opening of the Dajiang sand flushing gate is 0m~19.5m, with a gate opening accuracy of 0.1m. There are about 40 million possible gate operation schemes, which puts high requirements on the optimization algorithm. Therefore, it is urgent to develop a multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning to solve the above problems. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning. Based on the hierarchical reinforcement learning technology, the gate optimization problem is divided into two sub-problems of reservoir water level optimization and gate optimization, so as to realize efficient solution of the gate optimization scheduling problem.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is: A multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning, comprising: Step 1, constructing a reservoir discharge facility gate optimization scheduling model to determine the objective function and constraint conditions; Step 2, establishing a hierarchical reinforcement learning framework, including an option network and an option internal policy; The step 3, both the option network and the option internal policy adopt a competitive double deep Q network, and a hierarchical reinforcement learning framework is solved according to an epsilon-greedy policy to obtain a reasonable gate operation scheme.

[0007] Preferably, the objective function in the step 1 comprises minimizing the maximum outflow of the reservoir and the gate adjustment times of the discharge facility.

[0008] Preferably, the constraint condition in the step 1 comprises a water balance constraint, a reservoir storage capacity constraint, a reservoir water level amplitude constraint, a discharge facility discharge constraint, a hydraulic jump constraint and a gate opening constraint.

[0009] Preferably, the option in the step 2 of the hierarchical reinforcement learning framework is a triple , wherein the option is a reservoir water level at the end of a time period, is an internal policy of the option, is a probability of ending the option , and is a probability of starting the option .

[0010] Preferably, the input of the option network is a reservoir water level at the beginning of a time period, a time period inflow, and the output is an option; the state variable of the internal policy of the option is a time period inflow, a reservoir water level at the beginning of a time period, a reservoir downstream water level at the beginning of a time period, a reservoir water level at the end of a time period, and a gate operation scheme of a previous time period; and the reward function of the option network and the internal policy of the option is constructed based on the objective function.

[0011] Preferably, the construction of the reward function comprises: The objective of the option network is to minimize the maximum outflow of the reservoir, and the basic reward function is: ; An instant reward is provided in each time step, and the final reward function is: ; The objective of the internal policy of the option is to minimize the gate adjustment times, and the reward function is: ; In the formula, is an instant reward of a time period t, T is a total scheduling duration, h; t is a current scheduling time period; is a reservoir outflow of the time period t; M is the number of types of gates; is the number of gates of the mth type; is the opening of the jth gate of the mth type in the time period t.

[0012] Preferably, the objective function of the gate optimization scheduling model of the reservoir discharge facility is: ; In the formula, T is the total scheduling time length, h; t is the current scheduling period; is the reservoir outflow at t period; M is the number of types of gates; is the number of gates of the mth type; is the opening of the jth gate of the mth type at t period.

[0013] Preferably, the constraint condition of the gate optimization scheduling model of the reservoir discharge facility is: Water balance constraint: ; ; In the formula, is the reservoir storage capacity at the beginning of t period; is the reservoir inflow at t period; is the duration of t period; is the discharge of the jth gate of the mth type at t period; Reservoir storage capacity constraint: ; In the formula, and are the upper and lower limits of the reservoir storage capacity respectively; Reservoir water level amplitude constraint: ; In the formula, is the reservoir water level at the beginning of t period; is the allowable amplitude of the reservoir water level of adjacent periods; Discharge facility discharge constraint: ; In the formula, is the reservoir downstream water level at the beginning of t period; is the functional relationship of the discharge of the mth type gate and the gate opening, reservoir upstream water level and downstream water level; Water jump constraint: ; In the formula, is the water jump function; Gate opening constraint: ; In the formula, is the minimum accuracy of the opening of the mth type gate, m; n is a natural number; is the maximum opening of the mth type gate.

[0014] Preferably, the competitive double deep Q network, action value function and state value function are respectively: ; ; In the formula, is the advantage function; is the length of the overall action space; is the network parameter shared by the state value function and the advantage function; , are respectively the parameters independently owned by the state value function and the advantage function; The target value of the neural network is: .

[0015] Preferably, the expression of the epsilon-greedy strategy is: ; ; In the formula, is the initial value of epsilon; is the final value of epsilon; is the decay rate of epsilon; k is the number of training times.

[0016] The beneficial effects of the present application are as follows: The multi-objective reservoir gate optimal scheduling method based on hierarchical reinforcement learning provided by the present application effectively solves the problems of too many decision variable numbers and too large solving dimension of gate optimal scheduling decision by decomposing gate scheduling into two levels of decision, can quickly optimize to obtain a reasonable gate operation scheme according to the flood process, and can ensure that the operation scheme meets various constraint conditions and gate operation rules. The gate operation scheme obtained by optimization can effectively reduce the maximum reservoir outflow, ensure flood control safety, reduce the number of gate adjustments, reduce the work burden of flood season dispatch personnel, and prolong the service life of the gate. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is the model training process flowchart in the embodiment of the present application; Figure 2 is the reservoir outflow process curve diagram in the embodiment of the present application; Figure 3 is the reservoir water level process curve diagram in the embodiment of the present application; Figure 4 is the gate operation scheme schematic diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0018] As Figures 1-4 shown, a multi-objective reservoir gate optimal scheduling method based on hierarchical reinforcement learning includes: Step 1, a reservoir discharge facility gate optimal scheduling model is constructed, and a target function and a constraint condition are determined; Step 2, a hierarchical reinforcement learning framework is established, including an option network and an option internal policy; Step 3, the option network and the option internal policy both adopt a competitive double deep Q network, and the hierarchical reinforcement learning framework is solved according to an ε-greedy strategy to obtain a reasonable gate operation scheme.

[0019] Preferably, the target function in step 1 includes minimizing the maximum outflow of the reservoir and the gate adjustment times of the discharge facility.

[0020] Preferably, the constraint condition in step 1 includes a water balance constraint, a reservoir storage capacity constraint, a reservoir water level amplitude constraint, a discharge facility discharge constraint, a hydraulic jump constraint, and a gate opening constraint.

[0021] Preferably, the option of the hierarchical reinforcement learning framework in step 2 is a triple , wherein the option is the reservoir water level at the end of the period, is the option internal policy, is the probability of ending the option , and is the probability of starting the option .

[0022] Preferably, the input of the option network is the reservoir water level at the beginning of the period and the period inflow, and the output is the option; the state variable of the option internal policy is the period inflow, the reservoir water level at the beginning of the period, the reservoir downstream water level at the beginning of the period, the reservoir water level at the end of the period, and the gate operation scheme of the last period, and the action variable is the gate operation scheme of the current period; the reward function of the option network and the option internal policy is constructed based on the target function.

[0023] Preferably, the construction of the reward function includes: The target of the option network is to minimize the maximum outflow of the reservoir, and the basic reward function is: ; An immediate reward is provided in each time step, and the final reward function is: ; The target of the option internal policy is to minimize the gate adjustment times, and the reward function is: ; In the formula, The instant reward is for time period t, where T is the total scheduling duration (h), and t is the current scheduling time period. t represents the reservoir outflow rate during time period t; M represents the number and types of sluice gates. Let m be the number of gates of the m-th type; Let be the opening degree of the j-th gate of the m-th type during time period t.

[0024] Preferably, the objective function of the reservoir spillway gate optimization scheduling model is: ; In the formula, T is the total scheduling duration (h); t is the current scheduling period. t represents the reservoir outflow rate during time period t; M represents the number and types of sluice gates. Let m be the number of gates of the m-th type; Let be the opening degree of the j-th gate of the m-th type during time period t.

[0025] Preferably, the constraints of the reservoir spillway gate optimization scheduling model are as follows: Water balance constraints: ; ; In the formula, The reservoir's water storage capacity at the beginning of time period t; The inflow rate to the reservoir during time period t; The duration of time interval t; Let be the discharge flow rate of the j-th gate of the m-th type during time period t; Reservoir capacity constraints: ; In the formula, and These are the upper and lower limits of the reservoir's capacity, respectively. Reservoir water level fluctuation constraints: ; In the formula, The reservoir water level at the beginning of time period t; This represents the allowable fluctuation range of reservoir water level between adjacent time periods; Drainage facility discharge constraints: ; In the formula, The downstream water level of the reservoir at the beginning of time period t; Let f(m) be the functional relationship between the discharge capacity of the m-th type of gate and the gate opening, the upstream water level, and the downstream water level of the reservoir. Hydraulic jump constraints: ; In the formula, It is the hydraulic jump function; Gate opening constraints: ; In the formula, Let m be the minimum opening precision of the m-th type of gate, where m is a natural number. is the maximum opening degree of the m-th type of gate.

[0026] Preferably, the competitive dual-depth Q-network has an action value function. and state value function They are respectively: ; ; In the formula, The dominant function; The length of the overall motion space; These are network parameters common to both the state value function and the advantage function; , These are parameters that are independently owned by the state value function and the advantage function, respectively; The target value of the neural network is: .

[0027] Preferably, the expression for the ε-greedy strategy is: ; ; In the formula, Let ε be the initial value; This is the final value of ε; ε represents the decay rate; k represents the number of training iterations.

Claims

1. A multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning, characterized in that, include: Step 1: Construct an optimal scheduling model for the reservoir's spillway gates and determine the objective function and constraints; Step 2: Establish a hierarchical reinforcement learning framework, including the option network and the policy within each option; Step 3: Both the option network and the internal strategy of the option adopt a competitive dual-deep Q network. The hierarchical reinforcement learning framework is solved according to the ε-greedy strategy to obtain a reasonable gate operation scheme.

2. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The objective function described in step 1 includes minimizing the maximum outflow from the reservoir and the number of times the gates of the spillway are adjusted.

3. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 2, characterized in that, The constraints described in step 1 include water balance constraints, reservoir capacity constraints, reservoir water level fluctuation constraints, discharge constraints of spillway facilities, hydraulic jump constraints, and gate opening constraints.

4. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, Options for the hierarchical reinforcement learning framework described in step 2 For triples Among the options The reservoir water level at the end of the time period. For internal strategies of the options, End option The probability, For start options The probability of.

5. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 4, characterized in that, The inputs to the option network are the reservoir water level at the beginning of the time period and the inflow rate during the time period, and the output is the option. The state variables of the option internal policy are the inflow rate during the time period, the reservoir water level at the beginning of the time period, the downstream water level of the reservoir at the beginning of the time period, the reservoir water level at the end of the time period, and the gate operation plan of the previous time period. The action variable is the gate operation plan of the current time period. The reward functions of the option network and the option internal policy are constructed based on the objective function.

6. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 5, characterized in that, The construction of the reward function includes: The goal of the option network is to minimize the maximum outflow from the reservoir, and the basic reward function is: ； Providing immediate rewards at each time step yields the final reward function: ； The objective of the internal strategy is to minimize the number of gate adjustments, and the reward function is: ； In the formula, The instant reward is for time period t, where T is the total scheduling duration (h), and t is the current scheduling time period. t represents the reservoir outflow rate during time period t; M represents the number and types of sluice gates. Let m be the number of gates of the m-th type; Let be the opening degree of the j-th gate of the m-th type during time period t.

7. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The objective function of the optimal scheduling model for the reservoir's spillway gates is: ； In the formula, T is the total scheduling duration, in hours (h). t represents the current scheduling period; t represents the reservoir outflow rate during time period t; M represents the number and types of sluice gates. Let m be the number of gates of the m-th type; Let be the opening degree of the j-th gate of the m-th type during time period t.

8. The multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The constraints of the reservoir spillway gate optimization scheduling model are as follows: Water balance constraints: ； ； In the formula, The reservoir's water storage capacity at the beginning of time period t; The inflow rate to the reservoir during time period t; The duration of time interval t; Let be the discharge flow rate of the j-th gate of the m-th type during time period t; Reservoir capacity constraints: ； In the formula, and These are the upper and lower limits of the reservoir's capacity, respectively. Reservoir water level fluctuation constraints: ； In the formula, The reservoir water level at the beginning of time period t; This represents the allowable fluctuation range of reservoir water level between adjacent time periods; Drainage facility discharge constraints: ； In the formula, The downstream water level of the reservoir at the beginning of time period t; Let f(m) be the functional relationship between the discharge capacity of the m-th type of gate and the gate opening, the upstream water level, and the downstream water level of the reservoir. Hydraulic jump constraints: ； In the formula, It is the hydraulic jump function; Gate opening constraints: ； In the formula, Let m be the minimum opening precision of the m-th type of gate, where m is a natural number. is the maximum opening degree of the m-th type of gate.

9. A multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The competitive dual-depth Q-network, action value function and state value function They are respectively: ； ； In the formula, The dominant function; The length of the overall motion space; These are network parameters common to both the state value function and the advantage function; , These are parameters that are independently owned by the state value function and the advantage function, respectively; The target value of the neural network is: 。 10. A multi-objective reservoir gate optimization scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The expression for the ε-greedy strategy is: ； ； In the formula, Let ε be the initial value; This is the final value of ε; ε represents the decay rate; k represents the number of training iterations.

Images