Reservoir intelligent scheduling method based on near-end strategy optimization algorithm

Abstract

The invention discloses an intelligent reservoir scheduling method based on a near-end strategy optimization algorithm, and relates to the technical field of reservoir application. According to the method, a reservoir is modeled into an intelligent agent, the intelligent agent is composed of two neural networks of the same structure and interacts with a scheduling environment, network parameters are updated by adopting a near-end strategy optimization (PPO) algorithm, simulated reservoir inflow and forecast flow data are generated through an ARMA model according to historical actually-measured runoff data, and the intelligent agent is trained; the real-time scheduling performance of the intelligent agent is verified by using historical actually measured runoff data, behavior actions are expressed in a probability distribution form, the limitation of dimensionality disaster of a traditional algorithm is effectively avoided, and the Agent can more flexibly adjust a scheduling strategy by observing the water level and the incoming water condition in real time, so that the decision process is closer to an actual scheduling scene.

Description

Technical Field

[0001] The present invention relates to the technical field of reservoir application, and in particular to a reservoir intelligent dispatching method based on a proximal strategy optimization algorithm. Background Art

[0002] Real-time decision-making and scheduling of reservoirs is crucial in modern water resources management. Reservoirs not only undertake multiple functions such as flood control, power generation, and irrigation, but also need to cope with complex and changeable hydrological and meteorological conditions. Real-time scheduling refers to the scheduling of reservoirs based on current and predicted hydrological data in a short period of time (usually on a daily or hourly scale) to achieve multi-objective optimization. However, real-time scheduling faces many challenges, such as the need to process a large amount of data and make decisions in a short period of time [1]-[3], and to take into account multiple objectives such as flood control, power generation, shipping, and ecology, which often conflict with each other [4]-[7]. In addition, the high uncertainty of hydrological data further increases the difficulty of real-time scheduling [8]-9].

[0003] With the improvement of computer technology and data processing capabilities, research on real-time reservoir operation has made significant progress. The main research directions include: rule-based operation methods, optimization algorithms, and intelligent operation methods. The rule-based operation method uses pre-set rules or empirical formulas for operation. Its advantages are simplicity and ease, but it lacks flexibility and adaptability and is difficult to cope with complex and dynamically changing hydrological conditions

[10] -

[14] . Optimization algorithms include dynamic programming, genetic algorithms, particle swarm algorithms, etc. Many optimization algorithms require a lot of computing resources and time when solving large-scale, multi-objective reservoir operation problems, and are difficult to meet the requirements of real-time operation

[15] -

[17] . Some data-driven methods have high requirements on data quality and quantity. When data is insufficient or the data quality is poor, it is difficult to obtain good operation results

[18] . By introducing deep reinforcement learning algorithms, the limitations of traditional methods can be overcome to a certain extent, and more flexible and efficient real-time reservoir operation can be achieved

[19] -

[21] .

[0004] In recent years, the application of artificial intelligence and machine learning technologies in reservoir operation has gradually increased. In particular, the reinforcement learning (RL) method learns the operation strategy through interaction with the environment and shows good adaptability and optimization effect

[22] -

[24] . Deep reinforcement learning (DRL) has shown great potential in dealing with complex hydrological conditions. DRL can efficiently solve high-dimensional, nonlinear and time-delay problems by simulating state changes in the environment and dynamically adjusting strategies. Its application in flood forecasting, groundwater flow prediction and other fields has significantly improved the prediction accuracy and adaptability of the model

[32] -

[35] . In addition, DRL can achieve more accurate water resources management through continuous learning and optimization strategies when facing complex and changing hydrological conditions.

[0005] The Proximal Policy Optimization (PPO) algorithm is an advanced algorithm in deep reinforcement learning. It has the remarkable characteristics of a stable and efficient update process. It was proposed by OpenAI in 2017. PPO avoids drastic fluctuations in the policy by limiting the update range of the policy, thereby improving the convergence speed and stability of the algorithm. This feature enables PPO to perform well in real-time decision-making and achieve multi-objective optimization in complex environments. OpenAI successfully used PPO to train agents to master complex games such as Atari games and robot control tasks in simulated environments. By updating the policy in small steps, PPO effectively maintains the stability of the learning process, allowing the agent to gradually improve its performance in the game environment

[36] . In the field of robot control, the PPO algorithm also performs well. It can train robots to complete complex tasks such as path planning and operation control in different physical environments

[37] . By continuously optimizing the policy, PPO enables robots to complete tasks more efficiently. In addition, PPO is also widely used in the field of autonomous driving. It optimizes driving policies through deep reinforcement learning, improving safety and efficiency in actual road environments. Research has shown that the PPO algorithm is highly adaptable and robust in dealing with dynamic environments and complex decisions

[38]

[40] . In the business field, PPO is used for dynamic pricing and inventory management. By simulating different market environments, the PPO algorithm can optimize pricing strategies and inventory management, and improve the adaptability of enterprises under different market conditions

[41] . Overall, the wide application and excellent performance of PPO make it a popular algorithm in the field of deep reinforcement learning.

[0006] Hydropower operation can be modeled as a Markov decision process (MDP)

[42] -

[44] , which is a Markov process with rewards and decisions. MDP describes an environment in reinforcement learning (RL) in which an agent can learn in real time using new data to continuously improve its performance. Real-time reservoir operation faces complex and changeable hydrological conditions. The agent built based on PPO can dynamically adjust the operation strategy to adapt to different hydrological scenarios by continuously interacting with the environment. This makes PPO show obvious advantages in dealing with highly uncertain reservoir operation problems. PPO can consider multiple objectives (such as power generation, shipping, ecology, etc.) at the same time and achieve a balance among various objectives by optimizing the comprehensive reward function, which is particularly important for reservoir systems that need to meet multiple operation requirements;

[0007] However, with the existing technology, there is an urgent need for a model that can adjust the scheduling strategy more flexibly and make the decision-making process closer to the actual scheduling scenario, as well as a method for building the model;

[0008] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0009] The purpose of the present invention is to provide a reservoir intelligent scheduling method based on a proximal strategy optimization algorithm to solve the technical problems raised in the background technology.

[0010] To achieve the above object, the present invention provides the following technical solution: a reservoir intelligent scheduling method based on a proximal strategy optimization algorithm, comprising at least the following steps:

[0011] S1: Building a deep reinforcement learning framework for a reservoir agent, the deep reinforcement learning framework for a reservoir agent includes a collector and a learner, the collector includes at least an agent and a decision environment, the agent and the decision environment interact to generate data, the agent is an agent, and the deep reinforcement learning framework for a reservoir agent also includes a buffer;

[0012] S2: Applying a Markov decision process in a decision environment to define a reservoir environment, wherein the construction of the Markov decision process includes at least a transfer function, a state, an action, a reward, and a value function;

[0013] S3: The PPO deep reinforcement learning algorithm is used in the learner for proximal strategy optimization training, which is used to optimize from two aspects: stability and efficiency. The PPO deep reinforcement learning algorithm is referred to as PPO for short. The optimization process of the PPO includes that the objective function will combine the strategy goal, the entropy reward term and the value function loss;

[0014] S4: Use the Agent after learning optimization for intelligent reservoir scheduling.

[0015] Furthermore, the setting of the reservoir agent deep reinforcement learning framework includes at least the following steps:

[0016] In the collector, the agent interacts with the environment. At time t, the agent observes the state and input in the environment, and then selects an action from the behavior probability distribution;

[0017] Subsequently, the action is sent to the reservoir environment defined by the Markov decision process, the initial reservoir state of the next period is calculated by the transfer function, and the reward value of the action in this period is calculated by the reward function to update the system state and evaluate the reward of the action;

[0018] The states, rewards, and actions are collected into the experience replay buffer, which is part of the deep reinforcement learning framework of the reservoir agent. During the training phase, the experience (state, reward, and action) generated by the interaction between the agent and the environment is collected into the experience replay buffer. When the experience reaches the specified capacity, the collected data is passed to the learner to train the neural network. The collected data is passed to the learner and trained using the PPO deep reinforcement learning algorithm. The probability distribution of the behavior under the corresponding state is adjusted by optimizing the neural network parameters. After the training is completed, the newly updated neural network parameters are passed to the data collector.

[0019] Furthermore, the setting of the collector includes at least the following steps:

[0020] The agent and the decision environment interact continuously to generate data, namely, state s, action a, and reward r. The data generated at each step are strung together in the order of interaction to form a data track for training, namely:

[0021] τ=(<s0,a0,r0,…> ,<s1,c,r1,…> ,…, T-1 ,a T-1 ,r T-1 ,...>)

[0022] Among them, τ represents a trajectory, and T represents the termination time step of the trajectory, that is, the maximum time step of the interaction between the agent and the environment.

[0023] The key to the agent is the exploration and decision-making goals of the action. In the early stage of training, the agent has a higher exploration to make the strategy more random, so as to ensure that the agent correctly explores the action space during training. The exploration slowly decreases with the increase of rewards. The decision-making goal is the effect that the agent is expected to achieve using the latest collected data. From the perspective of trajectory generation, learning is carried out by increasing the probability of high-reward decision trajectories.

[0024] Quantitatively calculate the probability of each trajectory occurring:​

[0025] P θ (τ)=ρ(s0)P θ (a0|s0)P(s1|s0,a0)P θ (a1|s1)P(s2|s1,a1)··

[0026] ·P θ (a t |s t )P(s t+1 |s t ,a t )

[0027]

[0028] The probability of the trajectory occurring includes the rule of the environment P(s t+1 |s t ,a t ), the behavior of the agent P θ (a t |s t ) and the initial state distribution ρ(s0);

[0029] The rule P(s) of the environment t+1 |s t ,a t ) is the rules and logic of the internal state transfer of the environment. Usually, this part cannot be controlled. Instead, the Agent needs to adapt to and understand the rules of the environment through the learning process. Among them, t represents the scheduling period;

[0030] The behavior P of the agent θ (a t |s t ) where θ represents the neural network parameters: this part is controllable by the algorithm. Different neural network parameters determine the corresponding output action when a state is given. By optimizing the neural network parameters, the agent can choose the best action in various states and obtain the highest long-term return.

[0031] The initial state distribution ρ(s0) is the distribution of the initial state of the environment. The collected data trajectories are quantitatively expressed in the corresponding probability form. Then, the expected returns of these trajectories can be obtained by accumulating the immediate reward values. The decision-making goal of the agent is to maximize the probability of the trajectory that produces a high expected return value.

[0032] Further, the transfer function in S2 is: In the reservoir operation problem, the rules of the environment depend on the water balance equation and the environmental constraints;

[0033] The transfer of reservoir status is achieved through water balance, namely:

[0034] V t+1 =V t +(Q in -Q out -Q supply )×Δt

[0035] H=f(V)

[0036] Environmental constraints include water level constraints, flow constraints, output constraints, and water level amplitude constraints, namely:

[0037] H min ≤H≤H max

[0038]

[0039] |H t -H t+1 |≤ΔH m

[0040] Where: t represents the dispatching period; V is the reservoir capacity; H is the reservoir water level; H tail is the tailwater level Q below the dam in ,Q out ,Q supply ,Q spill ,Q power , respectively, inflow, outflow, water supply, abandoned water, and power generation; CE is the power generation efficiency coefficient; P power ,P max is the power generation and maximum output; Δt is the time period.

[0041] Furthermore, the state in S2 is that the agent observes 5 states s in each time period. t : Current date D t , initial water level H t , the outflow Q in the previous period out,t-1 , the same day into the Q in,t and the average inflow for the next 7 days

[0042] The reference formula is:

[0043]

[0044] Furthermore, the action in S2 is that the agent makes a decision based on the state observed during this period, and the decision variable a t It is a value between [0,1] and is converted into outbound flow through the outflow mapping function. The reference formula is:

[0045] Q out,t =a t ×(Q max -Qmin )+Q min .

[0046] Furthermore, the reward in S2 is the reward r generated by the interaction between the decision-making behavior and the environment. t , including: Power generation reward R power , Ecological Reward R Nav , Shipping Rewards R Eco ,Right now:

[0047]

[0048] r t =R power,t +R Nav,t +R Eco,t .

[0049] Furthermore, the value function in S2 is the value function of the numerical reward, which is calculated by the expectation of the discounted sum of the reward. Therefore, the strategy π θ The value function of (a|s) in the environment state s can be defined as:

[0050]

[0051] Where R(t) is the strategy followed θ The discounted sum of rewards at (a|s), γ∈(0,1) is the discount factor, r t is the direct reward at time step t. In state s, the optimal strategy π θ * (a|s) is The largest policy, which meets the RL algorithm’s goal of maximizing expected future rewards.

[0052] Furthermore, the PPO reinforcement learning is obtained through the evolution of Policy gradient theorem, Actor-Critic policy gradient and Trust Region Policy Optimization.

[0053] Furthermore, the objective function of the PPO is as follows:

[0054]

[0055] Among them, L CLIP+VF+S (θ) is the overall objective function of PPO, is the strategy goal, H(πθ(s t )) is the entropy bonus term and is the value function loss, and β is the entropy coefficient.

[0056] The objective function of PPO is designed to limit the step size of policy updates while avoiding the computational complexity of TRPO, which is achieved through an operation called "clipping".

[0057] The strategy objective function L of PPO CLIP (θ) is as follows:

[0058]

[0059] Where ∈ is a small constant used to define the range of clipping, represents the probability of the current strategy selecting action a in state s, represents the probability that the old strategy selects action a in state s, represents the advantage function, and ò represents the clipping range. Value function loss is the mean square error loss, which is used to measure the predicted state value V θ (s t ) and target return R t The difference between The specific formula can be expressed as:

[0060]

[0061] Where: V θ (s t ) is under the policy parameter θ, given the state s t The predicted value function of R t is the target return, which is the return calculated by the advantage function estimation method GAE;

[0062] H(π θ ) is the entropy bonus term:

[0063] L entropy =β·H(π θ )

[0064]

[0065] Among them, L entropy is the loss part of the entropy reward term; β is the entropy coefficient, which is used to adjust the importance of the entropy reward; H(π θ ) is the entropy of the probability distribution of the strategy π under the parameter θ in a given state s. The larger the entropy, the more uniform the behavior probability distribution of the strategy output, which means that the strategy has a stronger tendency to explore.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The present invention models the reservoir as an intelligent agent, which consists of two neural networks with the same structure, interacts with the scheduling environment, and uses the proximal policy optimization (PPO) algorithm to update the network parameters. The intelligent agent is trained by using the historical measured runoff data and the simulated inflow and forecast flow data generated by the ARMA model. The real-time scheduling performance of the intelligent agent is verified by using the historical measured runoff data. The behavior actions are expressed in the form of probability distribution, which effectively avoids the limitation of the dimensionality curse of the traditional algorithm. The agent can adjust the scheduling strategy more flexibly by observing the water level and water inflow in real time, making the decision-making process closer to the actual scheduling scenario.

[0068] In addition, the present invention also constructed an optimization scheduling model based on POA and a real-time scheduling model based on DT as a control. The results show that the real-time scheduling performance of the intelligent agent is better than the DT model and is close to the POA optimization scheduling model. In different scheduling environments, the intelligent agent performs well, especially in extreme years, showing high stability and flexibility. This proves that the real-time scheduling framework of the reservoir intelligent agent has significant advantages in the complex and changeable reservoir scheduling environment, and provides an effective and stable solution for the real-time scheduling of reservoirs.

[0069] Moreover, the ARMA model in the present invention is more stable than the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0071] Figure 1 This is a schematic diagram of the deep reinforcement learning framework of the present invention;

[0072] Figure 2 Schematic diagram of the PPO reinforcement learning algorithm of the present invention;

[0073] Figure 3 This is a schematic diagram of hyperparameter optimization of the present invention;

[0074] Figure 4 This is a schematic diagram of the long sequence scheduling process from 1965 to 2023 of the present invention;

[0075] Figure 5 This is a schematic diagram of the long sequence scheduling benefits of the present invention from 1965 to 2023. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0077] Embodiment 1:

[0078] Embodiment 1 is used to describe in detail the process and specific content of a reservoir intelligent scheduling method based on a proximal strategy optimization algorithm.

[0079] A reservoir intelligent scheduling method based on a proximal strategy optimization algorithm comprises at least the following steps:

[0080] S1: Building a deep reinforcement learning framework for a reservoir agent, the deep reinforcement learning framework for a reservoir agent includes a collector and a learner, the collector includes at least an agent and a decision environment, the agent and the decision environment interact to generate data, the agent is an agent, and the deep reinforcement learning framework for a reservoir agent also includes a buffer;

[0081] S2: Applying a Markov decision process in a decision environment to define a reservoir environment, wherein the construction of the Markov decision process includes at least a transfer function, a state, an action, a reward, and a value function;

[0082] S3: The PPO deep reinforcement learning algorithm is used in the learner for proximal strategy optimization training, which is used to optimize from two aspects: stability and efficiency. The PPO deep reinforcement learning algorithm is referred to as PPO for short. The optimization process of the PPO includes that the objective function will combine the strategy goal, the entropy reward term and the value function loss;

[0083] S4: Use the Agent after learning optimization for intelligent reservoir scheduling.

[0084] The setup of the deep reinforcement learning framework for the reservoir agent includes at least the following steps:

[0085] In the collector, the agent interacts with the environment. At time t, the agent observes the state and input in the environment, and then selects an action from the behavior probability distribution;

[0086] Subsequently, the action is sent to the reservoir environment defined by the Markov decision process, the initial reservoir state of the next period is calculated by the transfer function, and the reward value of the action in this period is calculated by the reward function to update the system state and evaluate the reward of the action;

[0087] The states, rewards, and actions are collected into the experience replay buffer, which is part of the deep reinforcement learning framework of the reservoir agent. During the training phase, the experience (state, reward, and action) generated by the interaction between the agent and the environment is collected into the experience replay buffer. When the experience reaches the specified capacity, the collected data is passed to the learner to train the neural network. The collected data is passed to the learner and trained using the PPO deep reinforcement learning algorithm. The probability distribution of the behavior under the corresponding state is adjusted by optimizing the neural network parameters. After the training is completed, the newly updated neural network parameters are passed to the data collector.

[0088] Setting up a collector consists of at least the following steps:

[0089] The agent and the decision environment interact continuously to generate data, namely, state s, action a, and reward r. The data generated at each step are strung together in the order of interaction to form a data track for training, namely:

[0090] τ=( <s0,a0,r0,…>,<s1,c,r1,…> ,…, T-1 ,a T-1 ,r T-1 ,...>)

[0091] Among them, τ represents a trajectory, and T represents the termination time step of the trajectory, that is, the maximum time step of the interaction between the agent and the environment.

[0092] The key to the agent is the exploration and decision-making goals of the action. In the early stage of training, the agent has a higher exploration ability to make the strategy more random, to ensure that the agent correctly explores the action space during training. The exploration ability slowly decreases as the reward increases. The decision-making goal is to achieve the effect that the agent expects to achieve using the latest collected data. From the perspective of trajectory generation, learning is carried out by increasing the probability of high-reward decision trajectories.

[0093] Quantitatively calculate the probability of each trajectory occurring:

[0094] P θ (τ)=ρ(s0)P θ (a0|s0)P(s1|s0,a0)P θ (a1|s1)P(s2|s1,a1)··

[0095] ·P θ (a t |s t )P(s t+1 |s t ,a t )

[0096] ​

[0097] The probability of a trajectory occurring includes the rule of the environment P(s t+1 |s t ,a t ), the behavior of the agent P θ (a t |s t ) and the initial state distribution ρ(s0);

[0098] Environmental rules P(s t+1 |s t ,a t ) is the rules and logic of the internal state transfer of the environment (transfer of the reservoir state). Usually, this part cannot be controlled. Instead, the Agent needs to adapt to and understand the rules of the environment through the learning process. Where t represents the scheduling period;

[0099] The behavior of the agent P θ (a t |s t ) where θ represents the neural network parameters: this part is controllable by the algorithm. Different neural network parameters determine the corresponding output action when a state is given. By optimizing the neural network parameters, the agent can choose the best action in various states and obtain the highest long-term return.

[0100] The initial state distribution ρ(s0) is the distribution of the initial state of the environment. The collected data trajectories are quantitatively expressed in the corresponding probability form. Then, the expected returns of these trajectories can be obtained by accumulating the immediate reward values. The decision-making goal of the agent is to maximize the probability of the trajectory that produces a high expected return value.

[0101] The reservoir operation problem is systematically described and solved using the Markov decision process (MDP) framework. The Markov decision process is constructed as follows:

[0102] The transfer function is In the reservoir operation problem, the rules of the environment depend on the water balance equation and environmental constraints;

[0103] The transfer of reservoir status is achieved through water balance, namely:

[0104] V t+1 =V t +(Q in -Q out -Q supply )×Δt

[0105] H=f(V)

[0106] Environmental constraints include water level constraints, flow constraints, output constraints, and water level amplitude constraints, namely:

[0107] H min ≤H≤H max

[0108]

[0109] |H t -H t+1 |≤ΔH m

[0110] Where: t represents the dispatching period; V is the reservoir capacity; H is the reservoir water level; H tail is the tailwater level Q below the dam in ,Q out ,Q supply ,Q spill ,Q power , respectively, inflow, outflow, water supply, abandoned water, and power generation; CE is the power generation efficiency coefficient; P power ,P max is the power generation and maximum output; Δt is the time period.

[0111] The state in S2 is that the agent observes 5 states s in each period t : Current date D t , initial water level H t , the outflow Q in the previous period out,t-1 , the same day into the Q in,t and the average inflow for the next 7 days

[0112] The reference formula is:

[0113]

[0114] The action is the decision made by the agent based on the state observed during this period, and the decision variable a t It is a value between [0,1] and is converted into outbound flow through the outflow mapping function. The reference formula is:

[0115] Q out,t =a t ×(Q max -Q min )+Q min .

[0116] Reward is the reward generated by the interaction between decision-making behavior and the environment t , including: Power generation reward R power , Ecological Reward R Nav , Shipping Rewards R Eco ,Right now:

[0117]

[0118] rt =R power,t +R Nav,t +R Eco,t .

[0119] The value function is the value function of the numerical reward, which is calculated by the expectation of the discounted sum of the rewards. Therefore, the strategy π θ The value function of (a|s) in the environment state s can be defined as:

[0120]

[0121] Where R(t) is the strategy followed θ The discounted sum of rewards at (a|s), γ∈(0,1) is the discount factor, r t is the direct reward at time step t. In state s, the optimal strategy π θ * (a|s) is The largest policy, which meets the RL algorithm’s goal of maximizing expected future rewards.

[0122] PPO reinforcement learning is obtained through the evolution of Policy gradient theorem, Actor-Critic policy gradient and TrustRegion Policy Optimization.

[0123] Policy gradient theorem:

[0124] The policy gradient method is a class of optimization algorithms for reinforcement learning

[48] that aims to optimize the policy by gradient ascent or descent to maximize the expected cumulative reward. For any differentiable policy π θ (a|s), whose parameter is θ, and the state-action value function is expressed as

[0125]

[0126] That is, in state s t Next, take action a t , the cumulative expected return when subsequent actions follow policy π, where γ∈(0,1) is the discount factor.

[0127] J(θ) represents the expected value of the cumulative return, then the gradient of the policy can be expressed as the expectation of the state-action value function:

[0128]

[0129] Where τ = [s0, a0, s1, a1, ...] is the trajectory of the state action rolled out according to the strategy π. τ Replaced by sampling N trajectories [τ 1 ,…,τ N ] and define the nth trajectory as The length of the trajectory is T n .in Redefine (REINFORCE: Monte Carlo), and use it to replace, and then replace γ in the formula t Omitting (which is a simpler approximation), we get the actual policy gradient formula:

[0130]

[0131] The policy gradient theorem shows that by collecting θ Based on the samples under the time-series, the gradient of J(θ) can be estimated, and then the gradient ascent method is used to update the policy parameters θ, thereby improving the expectation of the cumulative return. However, it requires a complete sequence of scheduling samples for effective learning, and each update may cause a large change in the policy, thus affecting the stability of the training.

[0132] Actor-Critic Policy Gradient:

[0133] In order to reduce variance and improve learning efficiency, the Actor-Critic method was introduced

[49] . In this method, the "actor" represents the policy function π θ (a|s) is responsible for learning the policy, that is, how to choose an action based on the current state, while the "critic" calculates the state value function or action value function Q φ (s t ,a t ) evaluates the quality of the selected action. This method can update the policy at each step instead of waiting until a complete sequence is completed, which can make more efficient use of data. The value function can be defined as:

[0134]

[0135] That is, in state s t Next, take action a t After that, the cumulative expected return when subsequent actions follow the strategy π.

[0136] By replacing the reward function in the original policy gradient formula with the action value function Q φ (s t ,a t), and then the update formula of the Actor-Critic algorithm is obtained. This method can update the strategy at each step instead of waiting until a complete sequence ends, which can make more efficient use of data and reduce gradient variance.

[0137] Introducing the value function to estimate V φ (s), which is defined as, assuming that the policy is π, it interacts with the environment in a certain state s until the game ends, how much is the expected reward. V φ (s) No action involved, Q φ (s t ,a t ) involves action. φ (s t ) is Q φ (s t ,a t ). In the actor-critic algorithm, only the network V is estimated, and the value of Q is represented by the value of , that is:

[0138] Q φ (s t ,a t )=r t +γV φ (s t+1 )

[0139] Use the temporal difference error (TD Error) to update the cost function parameters. The temporal difference error δ can be calculated as follows:

[0140] δ=r t +γV φ (s t+1 )-V φ (s t )

[0141] Among them, V φ (s t ) is the state s at time step t t The value function estimation under V φ (s t+1 ) is the value function estimate of the next state, and γ is the discount factor.

[0142] The actor ((Actor)) uses the policy gradient theorem to update the policy parameters θ. The policy gradient is obtained by weighting the policy gradient with the time difference error:

[0143]

[0144] Trust Region Policy Optimization:

[0145] Trust Region Policy Optimization (TRPO) is an advanced policy gradient method

[50] that aims to improve the performance of a policy by controlling the step size of the policy update, thereby staying within a "trust region" and avoiding performance degradation due to large policy updates. The core of TRPO is to solve the following optimization problem:

[0146]

[0147] The constraints are:

[0148]

[0149] Among them, δ is the parameter that limits the KL divergence; function D KL Calculate the behavior distance of different θ in the same state s; A π (s,a) is an estimate of the advantage function, which represents the expected benefit of selecting action a relative to the average situation, and uses GAE technology.

[0150] GAE provides a continuous estimate for the advantage function, using the weighted average of the TD (Temporal Difference) residuals with a weighting factor of λ. The calculation formula for GAE is:

[0151]

[0152] Among them, γ is the discount factor; λ is a smoothing parameter used to weigh the bias and variance; is a smoothed estimate; is the TD residual. When (λ=0), GAE degenerates into a one-step TD estimation, and when λ is close to 1, it will try to use more and more steps of TD residuals to estimate the advantage function; r t Represents the immediate reward at time step t.

[0153] Using a perspective similar to trajectory generation, the relationship between the new and old strategies is quantitatively given (the new strategy here refers to the updated strategy, and the old strategy refers to the strategy before the update):

[0154]

[0155] is the probability of the new distribution. As long as the right side of the plus sign in the above formula is greater than 0, a stable strategy improvement can be obtained. However, this item cannot be optimized in practice because it contains the new strategy itself, so it cannot update itself. Therefore, a reasonable way is to use an alternative function approximation, that is:

[0156]

[0157] It is a first-order approximation of η(π), which improves within a certain step size. will make Also improve, is the new strategy expectation, A π (s,a) is the advantage function of the old strategy, which is corrected by IS (Importance Sampling):

[0158]

[0159] IS is the Importance Sampling Ratio, which represents the ratio of the probability of taking an action (a) under the new policy to the probability of taking the same action under the old policy, and is used to sample from a probability distribution different from the desired probability distribution while still getting the correct expected value. It is used to estimate the change in expected return due to a policy change, even if these samples were sampled under the old policy.

[0160] use Interact with the environment to sample a large amount of trace data and put it in the experience replay buffer, whose capacity is buffer_size. At each optimization update, batch_size number of training samples are extracted from the experience replay buffer to update num_epoch is the number of times each batch of data is used to update the policy, that is, the optimizer will traverse the same batch of data multiple times to update the policy until After training to a certain extent and updating many times, Re-sampling fills up the experience replay buffer, which converts the same strategy into a different strategy, greatly improving the efficiency of parameter optimization. Importance sampling can safely use the same data multiple times without causing too much policy distribution shift.

[0161] The TRPO paper gives a quantitative relationship between the new strategy and the alternative function:

[0162]

[0163] KL divergence refers to the distance in action, not the distance in parameters. Because for actors, the change in parameters and the change in action are not necessarily completely consistent. Sometimes the output action may be very different if the parameters change slightly. Or the parameters change a lot, but the output action may not change much.

[0164] Based on the above inequality, the corresponding optimization algorithm can be derived to maintain the stable update of the strategy. Specifically, the optimization direction is given by the formula after the surrogate function approximation, and the optimization step size is constrained by the KL divergence term in the inequality, which is the so-called trust region.

[0165] In practical applications, TRPO is very troublesome to solve constrained optimization problems. The variance of IS calculation is large because when the difference between the new and old policies is large, the variance of the importance sampling ratio may be very large, resulting in unstable estimation performance. To alleviate this problem, OpenAI proposed a simple, practical and computationally efficient policy gradient method, Proximal Policy Optimization (PPO), which aims to improve the complexity and implementation difficulty of TRPO. PPO controls the amount of policy updates to ensure that the new and old policies are not too far apart, thereby reducing the variance of the importance sampling ratio, which can avoid too aggressive policy updates and improve the stability and reliability of learning

[32] .

[0166] In the optimization process of the PPO algorithm, the final objective function will combine the policy objective, entropy reward term and value function loss. The objective function of PPO is as follows:

[0167] The objective function of PPO is as follows:

[0168]

[0169] Among them, L CLIP+VF+S (θ) is the overall objective function of PPO, is the strategy goal, H(πθ(s t )) is the entropy bonus term and is the value function loss, and β is the entropy coefficient.

[0170] The objective function of PPO is designed to limit the step size of policy updates while avoiding the computational complexity of TRPO, which is achieved through an operation called "clipping".

[0171] The strategy objective function L of PPO CLIP (θ) is as follows:

[0172]

[0173] Where ∈ is a small constant (such as 0.1 or 0.2) that defines the range of clipping. represents the probability of the current strategy selecting action a in state s, represents the probability that the old strategy selects action a in state s, represents the advantage function and ò represents the shear range.

[0174] like Figure 2 As shown, It is the green line; It is the blue line; between the green line and the blue line, we want to take the minimum result.

[0175] Value Function Loss is the mean square error loss, which is used to measure the predicted state value V θ (s t ) and target return R t The difference between The specific formula can be expressed as:

[0176]

[0177] Where: V θ (s t ) is under the policy parameter θ, given the state s t The predicted value function of R t is the target return, which is the return calculated by the advantage function estimation method GAE;

[0178] H(π θ ) is the entropy bonus term:

[0179] L entropy =β·H(π θ )

[0180]

[0181] Among them, L entropy is the loss part of the entropy reward term; β is the entropy coefficient, which is used to adjust the importance of the entropy reward; H(π θ ) is the entropy of the probability distribution of the strategy π under the parameter θ in a given state s. The larger the entropy, the more uniform the behavior probability distribution of the strategy output, which means that the strategy has a stronger tendency to explore.

[0182] Embodiment 2:

[0183] Based on the above embodiment 1, the following specific application embodiment is proposed;

[0184] Case Overview

[0185] The Three Gorges Reservoir is located in the middle reaches of the Yangtze River in China and is one of the largest water conservancy projects in the world. Its annual runoff is about 400 billion m 3The runoff during the flood season from May to October accounts for 78% of the annual runoff, while the dry season accounts for only 22%. The total installed capacity of the Three Gorges Power Station is 22,500 MW, and the annual average power generation is about 88,000 GWh, ranking among the top in the world. The Three Gorges ship lock system is equipped with five levels of ship locks and ship lifts, which greatly improves the navigation capacity of the upper reaches of the Yangtze River. The scheduling principle of the Three Gorges Reservoir is to store excess water during the flood season, release water during the dry season to ensure navigation, ecology and water supply needs, and produce as much electricity as possible. The shipping demand is an average daily outflow of more than 6500m 3 / s; the ecological demand is to artificially create peaks from May 1 to June 10 every year to help Yangtze River fish migrate and reproduce, and the average outflow during this period must be greater than 13,000m 3 / s, and the traffic volume exceeds 15000m for at least 7 days 3 / s.

[0186] Dataset

[0187] In order to generate a large training dataset, an autoregressive moving average (ARMA) model was used to simulate the inflow

[45]

[46] . The main dataset is based on 56 years of historical data from 1965 to 2021, including various extreme hydrological events experienced by the basin. The ARMA model was used to fit the historical data to generate simulated data with similar historical statistical characteristics. 200 years of simulated data were generated as a training dataset for training the intelligent agent. The historical data was used as a validation dataset to test the effectiveness of the intelligent agent's scheduling decision.

[0188] The training data set has 200 years and a total of 73,050 scheduling periods, with 20 years as one episode, that is, each episode contains 7,305 steps. During training, a series of episodes are run, and at each step the agent observes the state, makes decisions, calculates the reward value generated by the state change, and optimizes its neural network model.

[0189] Simulation Scheduling

[0190] Three scheduling models are constructed to simulate the scheduling of the Three Gorges Reservoir, namely, the optimization scheduling based on the successive optimization algorithm (POA)

[47] , the decision tree real-time scheduling based on the C4.5 algorithm (DT)

[48]

[50] , and the intelligent agent real-time scheduling based on the PPO algorithm (Agent). The model scheduling configuration is as follows:

[0191] Table 1 Model scheduling configuration

[0192]

[0193] The dispatch models all have the same constraints and dispatch objectives. The model performance is evaluated using observed flows from 1965 to 2023. POA is used as a baseline model using observed flows because in principle it can provide the best solution assuming that future inflows are known during the simulation.

[0194] Based on the above two embodiments, the following result analysis is proposed:

[0195] Hyperparameter Optimization

[0196] In order to optimize the performance of the agent, a random search method is used to select the best hyperparameter combination

[51] . The random search method can efficiently explore possible combinations in a high-dimensional parameter space by randomly sampling in the hyperparameter space. There are 10 hyperparameters that need to be optimized, and the value ranges are shown in the following table.

[0197] Table 2 Hyperparameters and typical ranges

[0198]

[0199] In the present invention, 20 sets of hyperparameter training were performed by random search method. Each set of parameter combination was trained for 30,000,000 steps, and the policy entropy value, policy extrinsic value, and extrinsic reward of the model were recorded, such as Figure 3 Each line represents a set of hyperparameters, with blue representing higher reward values ​​and red representing lower reward values.

[0200] Scheduling process analysis

[0201] Figure 4 The operating process of reservoir water level and outflow flow under three scheduling modes from 1965 to 2023 is shown, where the gray lines represent the process lines of each year, the blue area represents the 25% to 75% confidence interval, and the blue line is the multi-year average process.

[0202] Figure 4 Figures (a) and (d) show that due to the limitation of the curse of dimensionality, the POA optimization algorithm can only use discrete states and actions for scheduling calculations. Discrete actions tend to choose a single outflow value during the dry season, which leads to higher reservoir water levels in wet years, while in dry years, priority is given to meeting shipping needs, resulting in a rapid drop in reservoir water levels.

[0203] Figure 4 Figures (b) and (e) show that the scheduling process of DT in regular years and multi-year averages is basically the same as that of POA, but the fixed rules will produce more water abandonment during the flood season in flood years. During the water storage period from September to October, due to the complexity of flood control and water storage, DT is prone to miss the short water storage opportunity, affecting the reservoir water storage, and thus reducing the benefits of the next year.

[0204] The agent selects the action with the highest reward probability based on the observed state. Since the state reward and action are probability distributions, the agent can decide continuous action values, making scheduling more flexible.

[0205] exist Figure 4 As can be seen from Figures (c) and (f), the Agent generates less water abandonment during the flood season. By observing the water level and water inflow during the same period, the outflow will be increased in the flood year and appropriately reduced in the dry year. In the case of unknown future long-term water inflow, this strategy significantly increases the reservoir water level in the dry year, making the reservoir water level process similar in different years, which is beneficial to reservoir management.

[0206] Comprehensive benefit analysis

[0207] Figure 5 The benefit results of simulated scheduling based on the 1965-2023 validation data set are shown. Each color block represents the data value for one year, and the last color block represents the multi-year average. Figure 5 (a) represents the average annual inflow, Figure 5 (b) represents the annual power generation, Figure 5 (c) represents the annual shipping guarantee rate, Figure 5 (d) represents the annual ecological guarantee rate.

[0208] from Figure 5 (a) and Figure 5 (b) It can be seen that the annual power generation of the reservoir is positively correlated with the inflow of the year. The average power generation of Agent decision scheduling is 0.54% more than that of DT decision scheduling, and only 0.23% different from POA optimization scheduling. In different water years, the power generation of Agent real-time scheduling and POA optimization scheduling is basically the same, while DT decision scheduling will have obvious power generation losses in some years.

[0209] from Figure 5 (c) and Figure 5 (d) It can be seen that the low water inflow in the current year will reduce the shipping and ecological guarantee rate in the next year. This is because the water inflow during the water storage period is insufficient, which makes the reservoir unable to store to the normal water level, thus affecting the shipping and ecological guarantee before the flood season of the next year. POA and DT maintain a 100% guarantee rate in the years that can meet the shipping requirements, while the annual average shipping guarantee rate of Agent is maintained between 99% and 100%. The multi-year shipping guarantee rate of Agent decision scheduling reaches 99.1%, and the multi-year ecological guarantee rate reaches 98.1%, which is almost the same as the POA optimization scheduling results, and shows excellent decision stability in drought years.

[0210] POA represents the best performance that can be achieved by global optimization, which shows that the agent can achieve decision results close to those of POA global optimization with known future inflows when only observing rainfall in the next 7 days. DT decision scheduling relies on scheduling rules extracted from POA optimization results and achieves results similar to POA in most years, but performs poorly in extreme years with fewer samples.

[0211] This paper proposes a multi-objective decision-making and scheduling method for reservoir intelligent agents based on the PPO reinforcement learning algorithm. Taking the Three Gorges Reservoir as an example, a 200-year simulation data set is used for training, and the actual inflow from 1965 to 2023 is used for verification and evaluation. The main research results are as follows:

[0212] (1) This paper preliminarily reveals the influence of PPO hyperparameters on the performance and stability of the model in the real-time reservoir scheduling scenario, and further narrows the typical value range of the hyperparameters.

[0213] (2) The proposed method replaces the empirical knowledge structure with a neural network, and the behavior is expressed in the form of probability distribution, which effectively avoids the limitation of the dimensionality curse of traditional algorithms. The agent can adjust the scheduling strategy more flexibly by observing the water level and water inflow in real time, making the decision-making process closer to the actual scheduling scenario.

[0214] (3) In the simulated scheduling from 1965 to 2023, the average annual power generation of Agent decision scheduling is 0.54% more than DT, and only 0.23% different from the POA optimization scheduling with known future inflows. The Agent's annual average shipping guarantee rate and ecological guarantee rate are close to the POA optimization results. When the 7-day forecast water inflow is known, Agent achieves a scheduling effect similar to that of POA global optimization.

[0215] (4) In typical year scheduling, the Agent was able to maintain a higher reservoir water level in dry years to avoid decision-making errors; faster water storage decisions in normal years led to higher power generation benefits; and in wet years, by increasing the discharge of water from the reservoir, stable water level control was achieved, reducing flood control risks. The Agent performed well in all typical years, especially in extreme years, showing high stability and flexibility.

[0216] In summary, the Agent model showed superior scheduling capabilities in different water inflow years, especially in decision stability in extreme years. In contrast, POA optimization requires known future inflows, which represents the best performance, while DT conventional scheduling has obvious defects in extreme years. Agent achieved a scheduling effect close to global optimization when observing 7-day forecast inflows, proving that it is an effective and stable scheduling strategy.

[0217] References cited based on the prior art presented in the background and description:

[0218] [1]Tao,Y.,Mo,L.,Yang,Y.,Liu,Z.,Liu,Y.,&Liu,T.(2023).OptimizationofCascade Reservoir Operation for Power Generation,Based on an ImprovedLightning Search Algorithm.Water,15(19),3417.

[0219] [2]Sarma, P., Durlofsky, LJ, & Aziz, K. (2008). Computational techniques for closed-loop reservoir modeling with application to a realistic reservoir. Petroleum Science and Technology,

[0220] 26(10-11),1120-1140.

[0221] [3] Yu,

[0223] [4] Yu,

[0225] [5]Diao,Y.,Ma,H.,Wang,H.,Wang,J.,Li,S.,Li,X.,...&Qiu,Q.(2022).Optimalflood-control operation of cascade reservoirs using an improved particleswarm optimization algorithm.

[0226] Water,14(8),1239.

[0227] [6]He,Z.,Zhou,J.,Qin,H.,Jia,B.,&Lu,C.(2019).Long-termjoint schedulingofhydropower station group in the upper reaches of the Yangtze River usingpartition parameter adaptation differential evolution.EngineeringApplicationsofArtificial Intelligence,81,1-13.

[0228] [7]Liu,Y.,Mo,L.,Yang,Y.,&Tao,Y.(2023).Optimal Scheduling ofCascadeReservoirs Based onan IntegratedMultistrategyParticle SwarmAlgorithm.Water,15(14),2593.

[0229] [8]Liu,X.,&Luo,J.(2019).A dynamic multi-objective optimization modelwith interactivity and uncertainty for real-time reservoir flood controloperation.Applied Mathematical Modelling,74,606-620.

[0231] [9]Chen,J.,Zhong,P.A.,Liu,W.,Wan,X.Y.,&Yeh,W.W.G.(2020).A multi-objective risk management model for real-time flood control optimal operationof a parallel reservoir system.Journal ofHydrology,590,125264.

[0232]

[10] Ostadrahimi,L., M.A.,&Afshar,A.(2012).Multi-reservoiroperation rules:

[0233] multi-swarm PSO-based optimization approach.Water resourcesmanagement,26,407-427.

[0234]

[11] Shrestha,B.P.,Duckstein,L.,&Stakhiv,E.Z.(1996).Fuzzy rule-basedmodeling ofreservoir operation.Journal ofwater resources planning andmanagement,122(4),262-269.

[0235]

[12] Moeini,R.,Afshar,A.,&Afshar,M.H.(2011).Fuzzy rule-based model forhydropower reservoirs operation.International Journal of Electrical Power&Energy Systems,33(2),171-178.

[0237]

[13] Kumar,A.S.,Goyal,M.K.,Ojha,C.S.P.,Singh,R.D.,Swamee,P.K.,&Nema,R.K.(2013).

[0238] Application ofANN,fuzzy logic and decision tree algorithms for thedevelopment ofreservoir operating rules.Water resources management,27,911-925.

[0239]

[14] Soentoro,E.A.,&Pebriana,N.(2019).Fuzzy rule-based model tooptimize outflow in single reservoir operation.In MATEC Web ofConferences(Vol.270,p.04015).EDP Sciences.

[0240]

[15] Fang,H.,Li,X.,Shang,W.,&Wang,L.(2022).Improved multi-reservoiroperation rules of water supply system based on target storage curves.WaterSupply,22(6),5753-5770.

[0241]

[16] Rani,D.,&Moreira,M.M.(2010).Simulation-optimization modeling:asurvey and potential application in reservoir systems operation.Waterresources management,24,1107-1138.

[0242]

[17] Maier,H.R.,&Dandy,G.C.(2000).Neural networks for the predictionand forecasting of water resources variables:a review of modelling issues andapplications.Environmental modelling&software,15(1),101-124.

[0243]

[18] Wang,Z.Z.,Zhang,K.,Chen,G.D.,Zhang,J.D.,Wang,W.D.,Wang,H.C.,...&Yao,J.

[0244] (2023).Evolutionary-assisted reinforcement learning for reservoirreal-time production optimization under uncertainty.Petroleum Science,20(1),261-276.

[0245]

[19] Zhang,Z.,Xu,J.,Zhuang,Z.,&Liu,J.(2023).Adaptive Proximal PolicyOptimization with Upper Confidence Bound.arXiv preprint arXiv:2312.07624

[0246]

[20] Borges,A.,&Silva,M.(2020).Kundzewicz,Z.W.,Krysanova,V.,Benestad,R.E.,Hov,

[0247] Piniewski,M.,&Otto,I.M.(2018).

[0248]

[21] Uncertainty in climate change impacts on waterresources.Environmental Science&Policy,79,1-8.

[0250]

[22] Wang,Z.Z.,Zhang,K.,Chen,G.D.,Zhang,J.D.,Wang,W.D.,Wang,H.C.,...&Yao,J.

[0251] (2023).Evolutionary-assisted reinforcement learning for reservoirreal-time production optimization under uncertainty.Petroleum Science,20(1),261-276.

[0252]

[23] Gangrade,S.,Lu,D.,Kao,S.C.,&Painter,S.L.(2022).Machine learningassisted reservoir operation model for long-term water managementsimulation.JAWRA Journal of the American Water Resources Association,58(6),1592-1603.

[0253]

[24] Sadeghi Tabas,S.(2021).Reinforcement Learning Policy GradientMethods for Reservoir Operation Management and Control.

[0254]

[25] Dariane,A.B.,&Moradi,A.M.(2016).Comparative analysis of evolvingartificial neural network and reinforcement learning in stochasticoptimization of multireservoir systems.Hydrological Sciences Journal,61(6),1141-1156.

[0255]

[26] Hakimi-Asiabar,M.,Ghodsypour,S.H.,&Kerachian,R.(2010).Derivingoperating policies for multi-objective reservoir systems:application of self-learning genetic algorithm.Applied Soft Computing,10(4),1151-1163.

[0256]

[27] Kamran,S.,Safavi,H.R.,Golmohammadi,M.H.,Rezaei,F.,Abd Elaziz,M.,Forestiero,A.,

[0257] &Lu,S.(2022).Maximizing sustainability in reservoir operation underclimate change using a novel adaptive accelerated gravitational searchalgorithm.Water,14(6),905.

[0258]

[28] Xu,W.,Meng,F.,Guo,W.,Li,X.,&Fu,G.(2021).Deep reinforcementlearning for optimal hydropower reservoir operation.Journal ofWater ResourcesPlanning and Management,147(8),04021045.

[0260]

[29] Tabas,S.S.,&Samadi,V.(2024).Fill-and-Spill:Deep ReinforcementLearning Policy Gradient Methods for Reservoir Operation Decision andControl.Journal of Water Resources Planning and Management,150(7),04024022.

[0261]

[30] Wu,R.,Wang,R.,Hao,J.,Wu,Q.,&Wang,P.(2024).Multiobjectivemultihydropower reservoir operation optimization with transformer-based deepreinforcement learning.Journal ofHydrology,130904.

[0262]

[31] Xu,W.,Zhang,X.,Peng,A.,&Liang,Y.(2020).Deep reinforcementlearning for cascaded hydropower reservoirs considering inflowforecasts.Water Resources Management,34,3003-3018.

[0264]

[32] Schulman,J.,Wolski,F.,Dhariwal,P.,Radford,A.,&Klimov,O.(2017).Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.

[0265]

[33] Zhan,G.,Zhang,X.,Li,Z.,Xu,L.,Zhou,D.,&Yang,Z.(2022).Multiple-uavreinforcement learning algorithm based on improved ppo in rayframework.Drones,6(7),166.

[0266]

[34] Schulman,J.,Wolski,F.,Dhariwal,P.,Radford,A.,&Klimov,O.(2017).Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.

[0267]

[35] Hsu,C.C.Y.,Mendler-Dünner,C.,&Hardt,M.(2020).Revisiting designchoices in proximal policy optimization.arXiv preprint arXiv:2009.10897.

[0268]

[36] E.,Coates,E.M.,Moe,S.,&Johansen,T.A.(2019,June).Deepreinforcement learning attitude control of fixed-wing UAVs using proximalpolicy optimization.In 2019 international conference on unmanned aircraftsystems(ICUAS)(pp.523-533).IEEE.

[0269]

[37] Wu,C.,Bi,W.,&Liu,H.(2023).Proximal policy optimization algorithmfor dynamic pricing with online reviews.Expert Systems withApplications,213,119191.

[0270]

[38] Lee,J.H.,&Labadie,J.W.(2007).Stochastic optimization ofmultireservoir systems via reinforcement learning.Water resources research,43(11).

[0271]

[39] Xu,W.,Zhang,C.,Peng,Y.,Fu,G.,&Zhou,H.(2014).A two stage Bayesianstochastic optimization model forcascaded hydropower systems consideringvarying uncertainty of flow forecasts.Water Resources Research,50(12),9267-9286.

[0272]

[40] Zhang,X.,Peng,Y.,Xu,W.,&Wang,B.(2019).An optimal operation modelfor hydropower stations considering inflow forecasts with different lead-times.Water resources management,

[0273] 33(1),173-188.

[0274]

[41] McLeod,A.I.,&Li,W.K.(1983).Diagnostic checking ARMA time seriesmodels using squared-residual autocorrelations.Journal oftime seriesanalysis,4(4),269-273.

[0275]

[42] Xu,W.,Meng,F.,Guo,W.,Li,X.,&Fu,G.(2021).Deep reinforcementlearning for optimal hydropower reservoir operation.Journal ofWater ResourcesPlanning and Management,147(8),04021045.

[0277]

[43] Feng,Z.K.,Niu,W.J.,Cheng,C.T.,&Lund,J.R.(2018).Optimizinghydropower reservoirs operation via an orthogonal progressive optimalityalgorithm.Journal of Water Resources Planning and Management,144(3),04018001.

[0278]

[44] Wei,C.C.,&Hsu,N.S.(2008).Derived operating rules for a reservoiroperation system:

[0279] Comparison of decision trees,neural decision trees and fuzzy decisiontrees.Water resources research,44(2).

[0280]

[45] Diao,Y.,Wang,C.,Wang,H.,&Liu,Y.(2021).Construction andapplication ofreservoir flood control operationrules using the decisiontreealgorithm.Water,13(24),3654.

[0281]

[46] Dai,L.,&Luo,G.(2022,August).Research on Reservoir Operation RulesExtraction Based onDecision Tree and Its Ensemble Model.In Journal ofPhysics:Conference Series(Vol.2333,

[0282] No.1,p.012021).IOPPublishing.

[0283]

[47] Bergstra,J.,&Bengio,Y.(2012).Random search for hyper-parameteroptimization.Journal of machine learning research,13(2).

[0284]

[48] Sutton RS,McAllester D,Singh S,et al.Policy gradient methods forreinforcement learning with function approximation[J].Advances in neuralinformationprocessing systems,1999,12.

[49] Crites R,Barto A.An actor / criticalgorithm that is equivalent to Q-learning[J].Advances in Neural InformationProcessing Systems,1994,7.

[0285]

[50] Schulman J, Levine S, Abbeel P, et al.Trust region policyoptimization[C] / / International conference on machine learning.PMLR, 2015:1889-1897.

[0286] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A reservoir intelligent dispatching method based on proximal strategy optimization algorithm, characterized by: At least the following steps are included: S1: Building a deep reinforcement learning framework for a reservoir agent, the deep reinforcement learning framework for a reservoir agent includes a collector and a learner, the collector includes at least an agent and a decision environment, the agent and the decision environment interact to generate data, the agent is an agent, and the deep reinforcement learning framework for a reservoir agent also includes a buffer; S2: Applying a Markov decision process in a decision environment to define a reservoir environment, wherein the construction of the Markov decision process includes at least a transfer function, a state, an action, a reward, and a value function; S3: The PPO deep reinforcement learning algorithm is used in the learner for proximal strategy optimization training, which is used to optimize from two aspects: stability and efficiency. The PPO deep reinforcement learning algorithm is referred to as PPO for short. The optimization process of the PPO includes that the objective function will combine the strategy goal, the entropy reward term and the value function loss; S4: Use the Agent after learning optimization for intelligent reservoir scheduling.

2. The reservoir intelligent dispatching method based on the proximal strategy optimization algorithm according to claim 1 is characterized by: The setting of the reservoir agent deep reinforcement learning framework includes at least the following steps: In the collector, the agent interacts with the environment. At time t, the agent observes the state and input in the environment, and then selects an action from the behavior probability distribution; Subsequently, the action is sent to the reservoir environment defined by the Markov decision process, the initial reservoir state of the next period is calculated by the transfer function, and the reward value of the action in this period is calculated by the reward function to update the system state and evaluate the reward of the action; The states, rewards, and actions are collected into the experience replay buffer, and the collected data is passed to the learner for training using the PPO deep reinforcement learning algorithm. The probability distribution of the behavior under the corresponding state is adjusted by optimizing the neural network parameters. After the training is completed, the latest updated neural network parameters are passed to the data collector.

3. The reservoir intelligent scheduling method based on the proximal strategy optimization algorithm according to claim 1 is characterized by: The setting of the collector comprises at least the following steps: The agent and the decision environment interact continuously to generate data, namely, state s, action a, and reward r. The data generated at each step are strung together in the order of interaction to form a data track for training, namely: τ=(<s0,a0,r0,...> ,<s1,a1,r1,...> ,..., T -1,a T -1,r T -1,...>)​ Among them, τ represents a trajectory, T represents the termination time step of the trajectory, that is, the maximum time step of the interaction between the agent and the environment; The key to the agent is the exploration and decision-making goals of the action. In the early stage of training, the agent has a higher exploration to make the strategy more random, so as to ensure that the agent correctly explores the action space during training. The exploration slowly decreases with the increase of rewards. The decision-making goal is the effect that the agent is expected to achieve using the latest collected data. From the perspective of trajectory generation, learning is carried out by increasing the probability of high-reward decision trajectories. Quantitatively calculate the probability of each trajectory occurring: P θ (τ)=ρ(s0)P θ (a0|s0)P(s1|s0,a0)P θ (a1|s1)P(s2|s1,a1)···P θ (a t |s t )P(s t+1 |s t ,a t ) The probability of the occurrence of the trajectory includes the rule of the environment P(s t+1 |s t , a t ), the behavior of the agent P θ (a t |s t ) and the initial state distribution ρ(s0); The rule P(s) of the environment t+1 |s t , a t ) is the rules and logic of the internal state transfer of the environment. Usually, this part cannot be controlled. Instead, the Agent needs to adapt to and understand the rules of the environment through the learning process. Among them, t represents the scheduling period; The behavior P of the agent θ (a t |s t ) where θ represents the neural network parameters: this part is controllable by the algorithm. Different neural network parameters determine the corresponding output action when a state is given. By optimizing the neural network parameters, the agent can choose the best action in various states and obtain the highest long-term return. The initial state distribution ρ(s0) is the distribution of the initial state of the environment. The collected data trajectories are quantitatively expressed in the corresponding probability form. Then, the expected returns of these trajectories can be obtained by accumulating the immediate reward values. The decision-making goal of the agent is to maximize the probability of the trajectory that produces a high expected return value.

4. The method for intelligent reservoir operation based on proximal strategy optimization algorithm according to claim 1 is characterized in that: The transfer function in S2 is: In the reservoir operation problem, the rules of the environment depend on the water balance equation and environmental constraints; The transfer of reservoir status is achieved through water balance, namely: V t+1 =V t +(Q in -Q out -Q supply )×Δt H=f(V) Environmental constraints include water level constraints, flow constraints, output constraints, and water level amplitude constraints, namely: H min ≤H≤H max |H t -H t+1 |≤ΔH m Where: t represents the dispatching period; V is the reservoir capacity; H is the reservoir water level; H tail is the tailwater level Q below the dam in , Q out , Q supply , Q spill , Q power , respectively, inflow, outflow, water supply, abandoned water, and power generation; CE is the power generation efficiency coefficient; P power , P max is the power generation and maximum output; Δt is the time period.

5. The method for intelligent reservoir operation based on proximal strategy optimization algorithm according to claim 4 is characterized in that: The state in S2 is that the agent observes 5 states s in each period t : Current date D t , initial water level H t , the outflow Q in the previous period out，t-1 , the same day into the Q in，t and the average inflow for the next 7 days The reference formula is:

6. The reservoir intelligent dispatching method based on the proximal strategy optimization algorithm according to claim 5 is characterized by: The action in S2 is the decision made by the agent based on the state observed in this period, and the decision variable a t It is a value between [0, 1] and is converted into outbound flow through the outflow mapping function. The formula is: Q out，t =a t ×(Q max -Q min )+Q min 。 7. The method for intelligent reservoir operation based on proximal strategy optimization algorithm according to claim 6 is characterized by: The reward in S2 is the reward r generated by the interaction between the decision-making behavior and the environment t , including: Power generation reward R power , Ecological Reward R Nav , Shipping Rewards R Eco ,Right now: r t =R power，t +R Nav，t +R Eco，t 。 8. The reservoir intelligent dispatching method based on the proximal strategy optimization algorithm according to claim 7 is characterized by: The value function in S2 is the value function of the numerical reward, which is calculated by the expectation of the discounted sum of the reward. Therefore, the strategy π θ The value function of (a|s) in the environment state s can be defined as: Where R(t) is the strategy followed θ The reward discount sum at (a|s), γ∈(0,1) is the discount factor, r t is the direct reward at time step t. In state s, the optimal strategy yes The largest policy, which meets the RL algorithm’s goal of maximizing expected future rewards.

9. The method for intelligent reservoir operation based on proximal strategy optimization algorithm according to claim 1 is characterized in that: The PPO reinforcement learning is obtained through the evolution of Policy gradient theorem, Actor-Critic policy gradient and TrustRegion Policy Optimization.

10. The reservoir intelligent dispatching method based on the proximal strategy optimization algorithm according to claim 9 is characterized in that: The objective function of the PPO is as follows: Among them, L CLIP+VF+S (θ) is the overall objective function of PPO, is the strategy goal, H(πθ(s t )) is the entropy bonus term and is the value function loss, β is the entropy coefficient; The objective function of PPO is designed to limit the step size of policy updates while avoiding the computational complexity of TRPO, which is achieved through an operation called "clipping". The strategy objective function L of PPO CLIP (θ) is as follows: where ∈ is a small constant used to define the range of clipping, represents the probability of the current strategy selecting action a in state s, represents the probability that the old strategy selects action a in state s, represents the advantage function, ò represents the shear range; Value Function Loss is the mean square error loss, which is used to measure the predicted state value V θ (s t ) and target return R t The difference between The specific formula can be expressed as: Where: V θ (s t ) is under the policy parameter θ, given the state s t The predicted value function of R t is the target return, which is the return calculated by the advantage function estimation method GAE; H(π θ ) is the entropy bonus term: L entropy =β·H(π θ ) Among them, L entropy is the loss part of the entropy reward term; β is the entropy coefficient, which is used to adjust the importance of the entropy reward; H(π θ ) is the entropy of the probability distribution of the strategy π under the parameter θ in a given state s. The larger the entropy, the more uniform the behavior probability distribution of the strategy output, which means that the strategy has a stronger tendency to explore.

Images