Cascade hydropower dispatching method, system and equipment based on TD3 algorithm and storage medium

The cascade hydropower scheduling method based on the TD3 algorithm solves the problems of low computational efficiency and poor reproducibility of results in the scheduling of cascade hydropower stations by using the double-delay-deterministic strategy gradient algorithm TD3. This results in more efficient power generation and more refined scheduling.

CN119026867AActive Publication Date: 2024-11-26HOHAI UNIV +1

Patent Information

Application Number
CN202411199335.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-26
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

Existing technologies are insufficient for achieving precise scheduling in cascade hydropower station operations, resulting in low computational efficiency, high memory consumption, poor reproducibility of results, and traditional machine learning algorithms are prone to getting trapped in local optima, which affects the economic efficiency of hydropower station operation.

Method used

A cascade hydropower scheduling method based on the TD3 algorithm is adopted. By constructing a long-term optimization scheduling model, the scheduling problem is transformed into a Markov decision process, and the TD3 algorithm with double delay-deterministic policy gradient is used to solve it. Combined with the deep deterministic policy gradient algorithm and dual Q-learning, the continuous action space value is output, which reduces network overestimation and error accumulation.

Benefits of technology

It has improved the power generation efficiency of cascade hydropower stations, significantly increased power generation, made the dispatching strategy more refined, and enhanced the economy and reliability of dispatching.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026867A_ABST
    Figure CN119026867A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of hydropower station dispatching, in particular to a cascade hydropower station dispatching method, system and device based on a TD3 algorithm and a storage medium, and the method comprises the steps: constructing a long-term optimization dispatching model based on the basic data and operation conditions of cascade hydropower stations; a scheduling problem in the long-term optimization scheduling model is converted into a Markov decision process; solving the Markov decision process by using a double delay-deterministic strategy gradient algorithm TD3 to obtain a long-term scheduling decision scheme of each power station in the cascade hydropower stations; and outputting a long-term scheduling decision result based on the actual cascade hydropower stations. According to the method, the continuous action space value in the state can be output according to the state quantity, the network over-estimation problem is reduced, the error accumulation problem is solved by adopting delay strategy updating and adding a noise smoothing target strategy, and the power generation benefit of the cascade hydropower station can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of hydropower station dispatching, and in particular to a cascade hydropower dispatching method, system, equipment and storage medium based on a TD3 algorithm. Background Art

[0002] Hydropower station dispatching is an important guarantee for fully utilizing water resources and improving power generation efficiency. In recent years, with the rolling development and centralized commissioning of hydropower bases in southwest my country, the scale of hydropower dispatching has expanded dramatically, and the power grid's requirements for hydropower refinement have increased day by day. The timeliness and availability of results of hydropower dispatching face great challenges.

[0003] The optimal dispatching of cascade hydropower stations in a river basin is a typical discrete, multi-dimensional, nonlinear, large-scale, multi-temporal and multi-space decision-making optimization problem, and its modeling and solution are very difficult. When establishing mathematical models, existing methods are difficult to consider all influencing factors and have poor adaptability to changing environments. The optimization solution methods mainly include traditional mathematical programming methods and traditional machine learning methods. Mathematical programming methods have problems such as time-consuming calculations, excessive memory usage, and low calculation efficiency, while traditional machine learning algorithms are easily limited to local optimality and have poor reproducibility of results.

[0004] At present, with the rapid development of big data and artificial intelligence algorithms, reinforcement learning and even deep reinforcement learning algorithms have been initially applied in the optimization and dispatching of cascade hydropower stations. However, most of the deep reinforcement learning algorithms currently use DQN (Deep Q Network). Since the DQN network outputs discrete action values, it affects the refinement of the dispatching strategy, and thus affects the economic efficiency of the operation of cascade hydropower stations. In this context, in order to further refine the intelligent dispatching decision-making of cascade hydropower stations and improve the economic efficiency of the operation of cascade hydropower stations, it is necessary to study new deep reinforcement learning methods.

[0005] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0006] The present invention provides a cascade hydropower scheduling method, system, device and storage medium based on the TD3 algorithm, thereby effectively solving the problems in the background technology.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a cascade hydropower dispatching method based on TD3 algorithm, comprising the following steps:

[0008] Construct a long-term optimization dispatch model based on the basic data and operation status of cascade hydropower stations;

[0009] Converting the scheduling problem in the long-term optimal scheduling model into a Markov decision process;

[0010] The Markov decision process is solved by using the double-delay-deterministic policy gradient algorithm TD3 to obtain a long-term scheduling decision plan for each power station in the cascade hydropower station;

[0011] Based on actual cascade hydropower stations, long-term scheduling decision results are output.

[0012] Furthermore, in the dual-delay-deterministic policy gradient algorithm TD3:

[0013] Use two valuation networks to evaluate action-values;

[0014] The frequency of updating the valuation function is greater than that of the policy function;

[0015] Soft update of target policy network and target value network parameters;

[0016] Clip the gradient used to update the policy network parameters to a set range;

[0017] During exploration, the policy expectation is smoothed using exploration noise and policy noise.

[0018] Furthermore, the use of the double-delayed-deterministic policy gradient algorithm TD3 to solve the Markov decision process includes the following steps:

[0019] Step 1: Initialize the prediction value network and The network parameters are w1 and w2 respectively;

[0020] Step 2: Initialize the target value network and The network parameters are w1′ and w2′;

[0021] Step 3: Initialize the prediction strategy network μ θ (s) and the target policy network μ θ′ (s), the network parameters are θ and θ′ respectively;

[0022] Step 4: Make the target value network parameters w1′ and w2′ consistent with the predicted value network parameters w1 and w2;

[0023] Step 5: Initialize the experience pool and set its capacity D;

[0024] Step 6: Initialize the environment module, generate the corresponding cascade hydropower station dispatching model, and determine the output initial state s0;

[0025] Step 7: Determine the hyperparameter values, including the total number of iterations M and the discount factor γ;

[0026] Step 8: Perform iterative calculations of the following steps starting from 1 to the final M;

[0027] Step 9: Initialize the environment and determine the output initial state s0;

[0028] Step 10: Input state s0 into the prediction policy network a = μ θ (s), and superimpose the exploration noise N to generate the corresponding action a0 = μ θ (s0)+N;

[0029] Step 11: Input the generated execution action a0 into the cascade hydropower station environment, obtain the reward r0 and the next state s1 data, and store the (s0, a0, r0, s1) data in the experience pool in the form of a unit;

[0030] Step 12: Repeat steps 10 and 11 to generate a series of data groups and save them in the experience pool. When the time step reaches the maximum value of the environment, return to step 9 and continue the operation;

[0031] Step 13: When the number of data groups stored in the loop reaches the set number, n experience transfer samples (s, a, r, s′) are randomly sampled from the experience pool as training data for training calculation:

[0032] Step 13.1: Compute the perturbed target policy network action: It adds a truncation noise to the target action generated by the target policy network;

[0033] Step 13.2: Calculate the updated target: The final target value is calculated by substituting the target strategy action after disturbance and the state at the next moment into two target value networks;

[0034] Step 14: According to the minimized loss function, the two prediction value network parameters w1 and w2 are updated using the calculated target value:

[0035] Step 15: Based on the maximized objective function, use the updated prediction value network Q w1 (s,a) Update the prediction strategy network parameters θ: Among them, the prediction value network parameters are updated twice and then the prediction strategy network parameters are updated once;

[0036] Step 16: Soft update target policy network and target value network parameters: Where τ is a hyperparameter much smaller than 1;

[0037] Step 17: Continue to update the network parameters using steps 13 to 16, and save the network parameter values ​​under the maximum reward until the final M iterations are reached.

[0038] The present invention also includes a cascade hydropower dispatching system based on the TD3 algorithm, using the above method, including:

[0039] Model building unit, used to build a long-term optimization dispatching model based on the basic data and operation status of cascade hydropower stations;

[0040] A conversion unit, used for converting the scheduling problem in the long-term optimization scheduling model into a Markov decision process;

[0041] A solution unit, used to solve the Markov decision process using a double-delayed-deterministic policy gradient algorithm TD3 to obtain a long-term dispatch decision plan for each power station in the cascade hydropower station;

[0042] The application unit is used to output long-term scheduling decision results based on actual cascade hydropower stations.

[0043] The present invention also includes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described above is implemented.

[0044] The present invention also includes a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0045] The beneficial effects of the present invention are as follows: compared with the existing DQN-based deep reinforcement learning algorithm, the present invention can output the continuous action space value under the state according to the state quantity; compared with the existing deep reinforcement learning algorithm based on deep deterministic policy gradient (DDPG), the TD3 algorithm combines the deep deterministic policy gradient algorithm and double Q learning to reduce the network over-estimation problem, adopts delayed policy update and adds noise smoothing target strategy to solve the error accumulation problem, and can effectively improve the power generation efficiency of cascade hydropower stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 It is a structural schematic diagram of the system of the present invention;

[0049] Figure 3 It is a flow chart of the intelligent decision-making method and system for long-term dispatching of cascade hydropower stations based on the double-delay-deterministic policy gradient algorithm;

[0050] Figure 4 The reservoir capacity-water level curves of Jinping I and Ertan hydropower stations;

[0051] Figure 5 It is the TD3 algorithm process;

[0052] Figure 6 This is Jinping Level 1 water level after intelligent scheduling;

[0053] Figure 7 The Jinping Level 1 power generation flow after intelligent scheduling;

[0054] Figure 8 This is the Jinping Level 1 power generation capacity after intelligent dispatching;

[0055] Fig. 9 The water level of Ertan after intelligent regulation;

[0056] Fig.10 is the Ertan power generation flow after intelligent scheduling;

[0057] Fig.11 is the power generation capacity of Ertan after intelligent scheduling;

[0058] Fig.12 A schematic diagram of the structure of a computer device. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0060] Embodiment 1:

[0061] like Figure 1 As shown: A cascade hydropower dispatching method based on TD3 algorithm includes the following steps:

[0062] Construct a long-term optimization dispatch model based on the basic data and operation status of cascade hydropower stations;

[0063] Transform the scheduling problem in the long-term optimal scheduling model into a Markov decision process;

[0064] The double-delayed-deterministic policy gradient algorithm TD3 is used to solve the Markov decision process and obtain the long-term dispatch decision plan for each power station in the cascade hydropower station.

[0065] Based on actual cascade hydropower stations, long-term scheduling decision results are output.

[0066] Compared with the existing DQN-based deep reinforcement learning algorithm, it can output the continuous action space value under the state according to the state quantity; compared with the existing DDPG-based deep reinforcement learning algorithm, the TD3 (Twin Delayed Deep Deterministic policy gradient) algorithm combines the deep deterministic policy gradient algorithm and double Q learning to reduce the network over-estimation problem, and adopts delayed policy update and noise smoothing target strategy to solve the error accumulation problem, which can effectively improve the power generation efficiency of cascade hydropower stations.

[0067] In the double-delayed-deterministic policy gradient algorithm TD3:

[0068] Use two valuation networks to evaluate action-values;

[0069] The frequency of updating the valuation function is greater than the policy function;

[0070] Soft update of target policy network and target value network parameters;

[0071] Clip the gradient used to update the policy network parameters to a set range;

[0072] During exploration, the policy expectation is smoothed using exploration noise and policy noise.

[0073] Among them, the double-delayed-deterministic policy gradient algorithm TD3 is used to solve the Markov decision process, including the following steps:

[0074] Step 1: Initialize the prediction value network Q w1 (s,a) and Q w2 (s,a), the network parameters are w1 and w2 respectively;

[0075] Step 2: Initialize the target value network Q w1′ (s,a) and Q w′2 (s,a), the network parameters are w1′ and w2′ respectively;

[0076] Step 3: Initialize the prediction strategy network μ θ (s) and the target policy network μ θ′ (s), the network parameters are θ and θ′ respectively;

[0077] Step 4: Make the target value network parameters w1′ and w2′ consistent with the predicted value network parameters w1 and w2;

[0078] Step 5: Initialize the experience pool and set its capacity D;

[0079] Step 6: Initialize the environment module, generate the corresponding cascade hydropower station dispatching model, and determine the output initial state s0;

[0080] Step 7: Determine the hyperparameter values, including the total number of iterations M and the discount factor γ;

[0081] Step 8: Perform iterative calculations of the following steps starting from 1 to the final M;

[0082] Step 9: Initialize the environment and determine the output initial state s0;

[0083] Step 10: Input state s0 into the prediction policy network a = μ θ (s), and superimpose the exploration noise N to generate the corresponding action a0 = μ θ (s0)+N;

[0084] Step 11: Input the generated execution action a0 into the cascade hydropower station environment, obtain the reward r0 and the next state s1 data, and store the (s0, a0, r0, s1) data in the experience pool in the form of a unit;

[0085] Step 12: Repeat steps 10 and 11 to generate a series of data groups and save them in the experience pool. When the time step reaches the maximum value of the environment, return to step 9 and continue the operation;

[0086] Step 13: When the number of data groups stored in the loop reaches the set number, n experience transfer samples (s, a, r, s′) are randomly sampled from the experience pool as training data for training calculation:

[0087] Step 13.1: Compute the perturbed target policy network action: It adds a truncation noise to the target action generated by the target policy network;

[0088] Step 13.2: Calculate the updated target: The final target value is calculated by substituting the target strategy action after disturbance and the state at the next moment into two target value networks;

[0089] Step 14: According to the minimized loss function, the two prediction value network parameters w1 and w2 are updated using the calculated target value:

[0090] Step 15: Use the updated prediction value network according to the maximized objective function Update the prediction strategy network parameters θ: Among them, the prediction value network parameters are updated twice and then the prediction strategy network parameters are updated once;

[0091] Step 16: Soft update target policy network and target value network parameters: Where τ is a hyperparameter much smaller than 1;

[0092] Step 17: Continue to update the network parameters using steps 13 to 16, and save the network parameter values ​​under the maximum reward until the final M iterations are reached.

[0093] like Figure 2 As shown, this embodiment also includes a cascade hydropower dispatching system based on the TD3 algorithm, using the above method, including:

[0094] Model building unit, used to build a long-term optimization dispatching model based on the basic data and operation status of cascade hydropower stations;

[0095] A conversion unit, used to convert the scheduling problem in the long-term optimization scheduling model into a Markov decision process;

[0096] A solution unit, used to solve the Markov decision process using a double-delayed-deterministic policy gradient algorithm TD3 to obtain a long-term dispatch decision plan for each power station in the cascade hydropower station;

[0097] The application unit is used to output long-term scheduling decision results based on actual cascade hydropower stations.

[0098] Embodiment 2:

[0099] like Figure 3 As shown, the intelligent decision-making method and system for long-term scheduling of cascade hydropower stations based on the double-delay-deterministic policy gradient algorithm include the following steps:

[0100] S1: Investigate the basic data and operation status of cascade hydropower stations and build a long-term optimization scheduling model for cascade hydropower stations;

[0101] The research object is the cascade hydropower stations in the lower reaches of the Yalong River Basin, mainly including Jinping I, Jinping II, Guandi, Ertan, and Tongzilin. Among them, Jinping I Hydropower Station is annually regulated, Ertan Reservoir is seasonally regulated, Jinping II Reservoir, Guandi Reservoir, and Tongzilin Reservoir are all daily regulated. Therefore, in the long-term scheduling process, Jinping I Hydropower Station and Ertan Hydropower Station are subject to long-term optimized scheduling. The optimized scheduling time is from June 1, 2020 to May 31, 2021, and the time scale is set to one day.

[0102] The basic data of Jinping I and Ertan Hydropower Stations include the highest and lowest water levels, the maximum and minimum reservoir capacities, the maximum power generation flow and the maximum output, as shown in Table 1. The reservoir capacity-water level curves of the two hydropower stations are shown in Figure 4 .

[0103] Table 1 Basic data of Jinping I and Ertan hydropower stations

[0104] Power Station Jinping I Hydropower Station Ertan Hydropower Station Maximum water level (m) 1880 1200 Minimum water level (m) 1800 1155 <![CDATA[Maximum storage capacity (100 million m 3 )]]> 77.6 57.9 <![CDATA[Minimum reservoir capacity (100 million m 3 )]]> 28.5 24.2 <![CDATA[Maximum power generation flow rate (m 3 / s)]]> 2024 2226 Maximum output(MW) 3600 3300

[0105] A long-term optimal dispatching model for cascade hydropower stations is constructed, and the objective function is:

[0106]

[0107] Where, E is the total power generation of the cascade hydropower station in the dispatching cycle, kWh; N is the number of hydropower stations in the cascade basin; T is the number of time periods in a dispatching cycle, here there are 365 days in a year; Δt is the number of hours in a time scale, h; i is the hydropower station number; t is the time period number; P i,t is the average power generation of hydropower station i in period t, kW; k i is the output coefficient of hydropower station i, which is 8.5 here; is the average power generation flow of hydropower station i in period t, m 3 / s;H i,t is the average net water head of hydropower station i in period t, m.

[0108] The constraints include:

[0109] (1) Water balance constraints:

[0110]

[0111] Where V i,t and V i,t+1 are the initial and final storage capacities of hydropower station i in period t, m 3 ; and are the average inflow and outflow of hydropower station i in period t, respectively, 3 / s.

[0112] The water balance constraint between the two hydropower stations is:

[0113]

[0114] In the formula, is the outflow of the upper hydropower station in period t, m 3 / s.

[0115] (2) Reservoir capacity constraints

[0116]

[0117] In the formula, and are the minimum and maximum reservoir capacities of hydropower station i in period t, m 3 .

[0118] (3) Outbound flow constraints

[0119]

[0120] In the formula, and are the minimum and maximum outflows of hydropower station i in period t, respectively. 3 / s.

[0121] (4) Power generation flow constraints

[0122]

[0123] In the formula, and are the minimum and maximum values ​​of the power generation reference flow of hydropower station i in period t, m 3 / s.

[0124] (5) Unit output constraints

[0125]

[0126] In the formula, is the maximum output allowed by hydropower station i in period t, kW.

[0127] (6) Water level restriction

[0128]

[0129] In the formula, and are the minimum and maximum values ​​of the net water head generated by hydropower station i during period t, m.

[0130] (7) Constraints on the relationship between reservoir capacity and water level

[0131] H i,t =f(V i,t ) (9)

[0132] The reservoir capacity and water level curve reflects the functional correspondence between the reservoir capacity and the water head of hydropower station i in period t.

[0133] (8) Initial water level constraint

[0134]

[0135] In the formula, is the initial water head of hydropower station i, m.

[0136] S2, transforming the long-term optimal scheduling problem of cascade hydropower stations into a Markov decision process;

[0137] The Markov decision process defines the state, action, state transfer function and reward function in reinforcement learning according to the characteristics of the long-term scheduling problem of cascade hydropower stations, and constructs a learning process.

[0138] Status t : It is used to describe the status information of the cascade hydropower station at the current moment, and provides reference guidance for the dispatching decision of the intelligent agent through the status information of the cascade hydropower station. It is defined as:

[0139]

[0140] In the formula, s t is the state quantity at time t, which mainly includes time t, as well as the inflow, storage capacity and water level of hydropower station i at that time.

[0141] Action a t : The long-term dispatching of cascade hydropower stations realizes the long-term optimal dispatching decision of cascade hydropower stations by controlling the power generation flow of each hydropower station. Therefore, action a t is the power generation reference flow of each hydropower station, expressed as:

[0142]

[0143] State transfer function: When the upstream water flow is determined at this moment, the reservoir capacity and water level at the next moment are determined by controlling the power generation flow at this moment. Therefore, the operating environment of the cascade hydropower station is also determined, and the state transfer probability of the Markov process is 1, that is:

[0144] p(s′|s,a)=1 (13)

[0145] In the formula, after the agent performs action a in state s, it can obtain a certain environment state s′ at the next moment, and the state transition process satisfies various set constraints.

[0146] Reward function r t (s t ,a t ): In the long-term optimal scheduling of cascade hydropower stations, the reward value is used to reflect the objective function value in a certain period of time, and the sum of all reward values ​​within the scheduling period is further used to represent the objective function within the scheduling period of the cascade hydropower station.

[0147]

[0148] In the formula, during the dispatching process of cascade hydropower stations, the reward value of each time period is the sum of the power generation of each hydropower station in this time period. When the environmental state H of the cascade hydropower station is determined, i,t And the action value After that, the corresponding environmental reward value can be generated for optimal scheduling learning.

[0149] S3, using the TD3 deep reinforcement learning algorithm to solve the Markov decision process and obtain the long-term scheduling decision plan for each power station in the cascade hydropower station;

[0150] For specific steps, see Figure 5 , specifically:

[0151] Step 1: Initialize the prediction value network and The network parameters are w1 and w2 respectively;

[0152] Step 2: Initialize the target value network and The network parameters are w1′ and w2′;

[0153] Step 3: Initialize the prediction strategy network μ θ (s) and the target policy network μ θ′ (s), the network parameters are θ and θ′ respectively;

[0154] Step 4: Make the target value network parameters w1′ and w2′ consistent with the predicted value network parameters w1 and w2;

[0155] Step 5: Initialize the experience pool and set its capacity D;

[0156] Step 6: Initialize the environment module, generate the corresponding cascade hydropower station dispatching model, and determine the output initial state s0;

[0157] Step 7: Determine the hyperparameter values, including the total number of iterations M and the discount factor γ;

[0158] Step 8: Perform iterative calculations of the following steps starting from 1 to the final M;

[0159] Step 9: Initialize the environment and determine the output initial state s0;

[0160] Step 10: Input state s0 into the prediction policy network a = μ θ (s), and superimpose the exploration noise N to generate the corresponding action a0 = μ θ (s0)+N;

[0161] Step 11: Input the generated execution action a0 into the cascade hydropower station environment, obtain the reward r0 and the next state s1 data, and store the (s0, a0, r0, s1) data in the experience pool in the form of a unit;

[0162] Step 12: Repeat steps 10 and 11 to generate a series of data groups and save them in the experience pool. When the time step reaches the maximum value of the environment, return to step 9 and continue the operation;

[0163] Step 13: When the number of data groups stored in the loop reaches the set number, n experience transfer samples (s, a, r, s′) are randomly sampled from the experience pool as training data for training calculation:

[0164] Step 13.1: Compute the perturbed target policy network action: It adds a truncated noise to the target action generated by the target policy network, thereby achieving the purpose of smoothing the target policy;

[0165] Step 13.2: Calculate the updated target: The final target value is calculated by substituting the target strategy action after disturbance and the state at the next moment into two target value networks;

[0166] Step 14: According to the minimized loss function, the two prediction value network parameters w1 and w2 are updated using the calculated target value:

[0167] Step 15: Use the updated prediction value network according to the maximized objective function Update the prediction strategy network parameters θ: Among them, after the prediction value network parameters are updated twice, the prediction strategy network parameters are updated once, thereby achieving the purpose of delaying strategy update;

[0168] Step 16: Soft update target policy network and target value network parameters: Where τ is a hyperparameter much smaller than 1;

[0169] Step 17: Continue to update the network parameters using steps 13 to 16, and save the network parameter values ​​under the maximum reward until the final M iterations are reached.

[0170] S4, using TD3 for long-term dispatching intelligent decision-making, the long-term dispatching intelligent decision-making of Jinping I and Ertan Hydropower Stations on the Yalong River was carried out from June 1, 2020 to May 31, 2021, and the dispatching time scale was set to one day. The hyperparameter settings of TD3 are shown in Table 2.

[0171] Table 2 Main hyper parameters of TD3 algorithm

[0172] sequence Hyperparameters Value Specific description 1 episode num 300 Total time loop times 2 learning rate <![CDATA[3×10 -9 ]]> Learning Rate 3 optimizer Adam Select optimizer 4 <![CDATA[ max_buffer ]]> 40000 <![CDATA[Experience Pool size > 5 batch size 64 Batch sampling sample quantity 6 targetupdaterate 0.005 Target network update rate 7 disdcount factor 0.996 Discount Factor 8 exploration noise N(0,0.1) Exploring Noise 9 policy noise 0.2 Target strategy noise 10 noiseclip 0.5 Target strategy noise limit range 11 policy delay 2 Delayed policy update value

[0173] The dispatch decision results of Jinping I and Ertan Hydropower Stations on the Yalong River for the whole year from June 1, 2020 to May 31, 2021 are shown in Figure 6 and Fig.11 .

[0174] Depend on Figures 6 to 8 It can be seen that in the early stage of scheduling, the power generation flow under TD3 scheduling is larger than the actual power generation flow, and the water level rises more slowly; in the middle stage of scheduling, the power generation flow under TD3 scheduling is less than the actual power generation flow, and the water level remains higher than the actual water level; at the end of scheduling, the TD3 scheduling method quickly increases the power generation flow to release the stored water energy, causing the water level to drop rapidly. From the perspective of power generation, in the early stage of scheduling, the power generation under TD3 scheduling is more stable than the actual power generation, which is closely related to the high value of power generation flow; in the middle stage of scheduling, the power generation under TD3 scheduling begins to decline, and the decline is greater than the actual power generation, thereby ensuring that the water level can remain high; at the end of scheduling, the power generation under TD3 scheduling begins to increase rapidly, quickly converting the stored water energy into power generation.

[0175] Depend on Figures 9 to 11 It can be seen that in the early stage of dispatching, the power generation flow of Ertan Hydropower Station under TD3 dispatching is maintained at a high value, so the corresponding TD3 dispatching water level rise rate is slower than the actual reservoir water level; in the middle stage of dispatching, Ertan Hydropower Station has been operating at the highest water level under TD3 dispatching, and its power generation flow is smaller than the actual power generation flow; at the end of dispatching, TD3 power generation flow increases rapidly and is far higher than the actual power generation flow, and the water level drops rapidly. From the perspective of power generation, in the early stage of dispatching, the power generation power of Ertan Hydropower Station under TD3 dispatching is greater than the actual power generation power, and reaches the maximum power generation power after a period of time; in the middle stage of dispatching, the power generation power and actual power generation power of Ertan Hydropower Station under TD3 dispatching begin to decline, and the power generation power under TD3 dispatching is maintained at a low level, thereby ensuring that the water level is at a high level; at the end of dispatching, the power generation power of TD3 dispatching begins to rise and exceeds the actual power generation, thereby increasing the power generation as much as possible before the end of the dispatching cycle.

[0176] The DQN method and DDPG method are further used to make long-term intelligent scheduling decisions for the Yalong River cascade hydropower stations. The power generation of the cascade hydropower stations under each method is shown in Table 3.

[0177] Table 3 Power generation of cascade hydropower stations under actual scheduling, DQN and DDPG algorithms (unit: billion kWh)

[0178] Power Station Actual scheduling DQN Scheduling DDPG Scheduling TD3 Scheduling Jinping Level 1 188.85 190.05 195.23 195.51 Ertan 158.20 192.08 193.62 194.06 Total of steps 347.05 382.13 388.85 389.57

[0179] As can be seen from Table 3, in the intelligent dispatching decision based on the TD3 deep reinforcement learning algorithm, the total annual power generation of the cascade hydropower stations is 389.57 kWh, which is 4.252 billion kWh higher than the historical actual dispatching decision, 744 million kWh higher than the DQN dispatching decision, and 72 million kWh higher than the DDPG dispatching decision. Compared with the actual dispatching decision, the method of the present invention can significantly improve the power generation efficiency of the cascade hydropower station; at the same time, compared with the existing DQN and DDPG deep reinforcement learning dispatching decisions, the present invention adopts continuous action values ​​and combines deep deterministic policy gradient algorithm and dual Q learning, so that the dispatching strategy is more refined and has better power generation efficiency.

[0180] See also Fig.12 A computer device 400 provided in an embodiment of the present application includes: a processor 410 and a memory 420, wherein the memory 420 stores a computer program executable by the processor 410, and when the computer program is executed by the processor 410, the above method is executed.

[0181] The embodiment of the present application further provides a storage medium 430 on which a computer program is stored. When the computer program is run by the processor 410, the above method is executed.

[0182] Among them, the storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0183] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0184] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0185] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0186] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0187] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0188] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0189] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0190] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A cascade hydropower dispatching method based on TD3 algorithm, characterized in that: The steps include: Construct a long-term optimization dispatch model based on the basic data and operation status of cascade hydropower stations; Converting the scheduling problem in the long-term optimal scheduling model into a Markov decision process; The Markov decision process is solved by using the double-delay-deterministic policy gradient algorithm TD3 to obtain a long-term scheduling decision plan for each power station in the cascade hydropower station; Based on actual cascade hydropower stations, long-term scheduling decision results are output.

2. The cascade hydropower dispatching method based on the TD3 algorithm according to claim 1 is characterized in that: In the dual-delay-deterministic policy gradient algorithm TD3: Use two valuation networks to evaluate action-values; The frequency of updating the valuation function is greater than the policy function; Soft update of target policy network and target value network parameters; Clip the gradient used to update the policy network parameters to a set range; During exploration, the policy expectation is smoothed using exploration noise and policy noise.

3. The cascade hydropower dispatching method based on the TD3 algorithm according to claim 2 is characterized in that: The method of solving the Markov decision process using the double-delayed-deterministic policy gradient algorithm TD3 comprises the following steps: Step 1: Initialize the prediction value network and The network parameters are w1 and w2 respectively; Step 2: Initialize the target value network and The network parameters are w1′ and w2′; Step 3: Initialize the prediction strategy network μ θ (s) and the target policy network μ θ′ (s), the network parameters are θ and θ′ respectively; Step 4: Make the target value network parameters w1′ and w2′ consistent with the predicted value network parameters w1 and w2; Step 5: Initialize the experience pool and set its capacity D; Step 6: Initialize the environment module, generate the corresponding cascade hydropower station dispatching model, and determine the output initial state s0; Step 7: Determine the hyperparameter values, including the total number of iterations M and the discount factor γ; Step 8: Perform iterative calculations of the following steps starting from 1 to the final M; Step 9: Initialize the environment and determine the output initial state s0; Step 10: Input state s0 into the prediction policy network a = μ θ (s), and superimpose the exploration noise N to generate the corresponding action a0 = μ θ (s0)+N; Step 11: Input the generated execution action a0 into the cascade hydropower station environment, obtain the reward r0 and the next state s1 data, and store the (s0, a0, r0, s1) data in the experience pool in the form of a unit; Step 12: Repeat steps 10 and 11 to generate a series of data groups and save them in the experience pool. When the time step reaches the maximum value of the environment, return to step 9 and continue the operation; Step 13: When the number of data groups stored in the loop reaches the set number, n experience transfer samples (s, a, r, s′) are randomly sampled from the experience pool as training data for training calculation: Step 13.1: Compute the perturbed target policy network action: It adds a truncation noise to the target action generated by the target policy network; Step 13.2: Calculate the updated target: The final target value is calculated by substituting the target strategy action after disturbance and the state at the next moment into two target value networks; Step 14: According to the minimized loss function, the two prediction value network parameters w1 and w2 are updated using the calculated target value: Step 15: Use the updated prediction value network according to the maximized objective function Update the prediction strategy network parameters θ: Among them, the prediction value network parameters are updated twice and then the prediction strategy network parameters are updated once; Step 16: Soft update target policy network and target value network parameters: Where τ is a hyperparameter much smaller than 1; Step 17: Continue to update the network parameters using steps 13 to 16, and save the network parameter values ​​under the maximum reward until the final M iterations are reached.

4. A cascade hydropower dispatching system based on TD3 algorithm, characterized in that: Use of the method according to any one of claims 1 to 3, comprising: Model building unit, used to build a long-term optimization dispatching model based on the basic data and operation status of cascade hydropower stations; A conversion unit, used for converting the scheduling problem in the long-term optimization scheduling model into a Markov decision process; A solution unit, used to solve the Markov decision process using a double-delayed-deterministic policy gradient algorithm TD3 to obtain a long-term dispatch decision plan for each power station in the cascade hydropower station; The application unit is used to output long-term scheduling decision results based on actual cascade hydropower stations.

5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Cascade reservoir random optimization scheduling method based on deep Q learning

    CN110930016A

  • Intelligent scheduling method for wind-light-water complementary system based on deep reinforcement learning

    CN116345450A

  • Reactive power optimization method based on double-delay depth deterministic strategy gradient

    CN116468159A

  • Method and apparatus for constructing dispatching model of integrated energy system, medium, and electronic device

    WO2022160705A1

Cited By

  • Flood scheduling method suitable for Fucheng river basin based on Xinanjiang river model

    CN120355151A

  • Multi-agent reinforcement learning driven cascade hydropower random intelligent scheduling method

    CN120931430A