A real-time load distribution method for hydropower stations based on deep reinforcement learning

Abstract

The present invention discloses a real-time load distribution method for a hydropower station based on deep reinforcement learning, comprising: constructing a load optimization distribution model based on deep reinforcement learning, using the power station operation history data to train the load optimization distribution model based on deep reinforcement learning; after the power station receives the load instruction, the power station's startup unit combination is updated in combination with the current load instruction, and on this basis, the remaining feasible startup unit combinations are further set, and the load optimization distribution model based on deep reinforcement learning is used to carry out fast rolling calculations on the forecast information of the flow and load instructions to optimize the startup unit combination; the startup unit combination of the power station is updated, and the load optimization distribution model based on deep reinforcement learning is used to solve the optimal distribution scheme of the power grid load instruction between the startup units. The present invention is applicable to the real-time load distribution of a hydropower station, and can be used to guide the real-time operation of the hydropower station units.

Description

Technical Field

[0001] The present invention relates to a power station optimization dispatching technology, and in particular to a real-time load distribution method for a hydropower station based on deep reinforcement learning. Background Art

[0002] At present, the algorithms for solving the real-time load distribution problem in hydropower stations include traditional optimization algorithms and intelligent algorithms. Traditional optimization algorithms mainly include equal incremental rate method, dynamic programming method and load distribution table method, and intelligent algorithms mainly include genetic algorithm and particle swarm algorithm. However, these methods are difficult to apply to actual real-time scheduling decision guidance, mainly including the following problems:

[0003] (1) In traditional optimization algorithms, the equal incremental rate method requires that the flow characteristic curve of the unit is continuously differentiable and the incremental rate is monotonically increasing. However, the unit has a vibration zone, and most power stations are difficult to meet this condition. The dynamic programming method and the load distribution table method are prone to the curse of dimensionality and are difficult to deal with the problem of multi-unit, high-precision in-plant economic operation.

[0004] (2) Intelligent algorithms have the problem of unstable decision-making results, which can easily produce solutions with poor optimization effects or infeasible solutions. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a real-time load distribution method for a hydropower station based on deep reinforcement learning.

[0006] Technical solution: A real-time load distribution method for a hydropower station based on deep reinforcement learning of the present invention comprises the following steps:

[0007] S1. Construct a load optimization allocation model based on deep reinforcement learning, and train the load optimization allocation model based on deep reinforcement learning in combination with historical operation data; the load optimization allocation model includes state, action, reward and constraint, with all factors that will affect the load allocation decision of the power station as the state s; design actions based on the idea of ​​load adjustment, convert the allocation of a load instruction into a process of gradual adjustment of the load allocation plan, including two types of actions: load adjustment and plan output; each action will have an immediate reward r; the constraints of the load optimization allocation model based on deep reinforcement learning include the constraints of the power station operation;

[0008] S2, the power station receives the load instruction of the power grid, calculates the minimum output and maximum output of each unit in the current combination of the starting units, adds the minimum output of each unit to obtain the lower boundary of the output range of the power station, and adds the maximum output of each unit to obtain the upper boundary of the output range of the power station; based on the value of the load instruction, it is determined whether the current combination of the starting units can complete the load instruction; if the load instruction can be completed, the current combination of the starting units is kept unchanged and step S4 is executed; otherwise, step S3 is executed;

[0009] S3. If the grid load instruction is greater than the upper boundary of the power station's output range, the unit with the longest shutdown time is turned on, and the start and stop status and load range of the power station units are updated until the grid load instruction is between the lower boundary of the power station's output range and the upper boundary of the power station's output range;

[0010] If the grid load instruction is less than the lower boundary of the power plant's output range, the unit with the longest current startup time is shut down, and the startup and shutdown status and load range of the power plant units are updated until the grid load instruction is between the lower boundary of the power plant's output range and the upper boundary of the power plant's output range;

[0011] S4, determine whether there are other feasible startup unit combinations, if there are no other feasible startup unit combinations, directly go to step S6, otherwise, go to step S5;

[0012] S5. Combine the rationality constraints of the start and stop of the units to generate the remaining feasible combinations of the startup units. Combine the load and flow forecast information for a period of time in the future, and use the load optimization allocation model based on deep reinforcement learning to roll-calculate the load allocation schemes in the station and the objective function values ​​that can be obtained for all feasible startup unit combination schemes within the calculation period; calculate the total objective function value that can be obtained for each scheme within the forecast period, select the optimal startup unit combination based on the total objective function value, and update the startup unit combination of the power station;

[0013] S6. Use the load optimization distribution model based on deep reinforcement learning to generate an optimized load distribution plan among the operating units.

[0014] Furthermore, the state in step S1 is a one-dimensional tensor [Q in ,Q flood ,Q generate ,Q g,t ,Z,N in-plant ], where Q in is the inflow flow, Q flood is the flood discharge, Q generate is the power generation flow, Q g,t is the power generation flow required to achieve the water dispatch target of the power station at time t, Z is the upstream water level, N in-plant is the unit load;

[0015] The calculation formula of instant reward r is:

[0016]

[0017] Among them, r is the immediate reward of the deep reinforcement learning model; n is the total number of units; s j and j+1 They are the states before and after the action is performed; and are the output of unit i before and after the action is performed; H is the current water head of the power station; and The units are respectively and The power generation flow when the water head is H;

[0018] The constraints include the constraints of power station operation: unit operation constraints, water balance constraints, water level constraints, water level-water storage relationship constraints and tailwater level-outflow relationship constraints.

[0019] Furthermore, by calculating the power generation flow Q required to achieve the water scheduling target of the power station g,t To optimize the power generation flow of the power station, when the power station needs to store water, Q g,t It can be taken as the minimum downstream flow of the power station. When the power station has a water level control demand, Q g,t The calculation formula is:

[0020] Q g,t =Q in,t -Q flood,t +(V(Z t )-V(Z g,t )) / △t

[0021] Among them, Q in,t is the inflow flow of the power station at time t, Q flood,t is the flood discharge of the power station at time t, V(Z t ) and V(Z g,t ) are water levels Z t and Z g,t When the water storage capacity of the power station reservoir is t is the water level of the power station at time t, Z g,t is the target water level of the power station at time t, and △t is the time interval between load instructions.

[0022] Furthermore, the specific steps of the training process of the load optimization distribution model based on deep reinforcement learning in step S1 include:

[0023] (1) Randomly generate an initial value function Q(s,a), set counter i = 0, counter j = 0, set the number of training rounds to I, the number of samples to J, and the probability of randomly selecting an action to P r , generate a sample library that is identical to the power plant's historical operation database;

[0024] (2) In the sample library, a piece of operating data is randomly selected by sampling without replacement, and a load instruction N is randomly generated while keeping the current combination of the operating units unchanged. c Feasible load distribution scheme Nin-plant , combining the operation data and the feasible load distribution scheme to construct the current state s=[Q in ,Q flood ,Q generate ,Q g,t ,Z,N in-plant ], if all the data in the sample library are sampled, a sample library identical to the power plant historical operation database is regenerated, and i=i+1;

[0025] (3) Input the current state s into the load optimization allocation model, judge and shield the invalid actions that will cause the unit load to fall into the vibration zone or become negative, and generate a random number p between [0,1]. If p≤P r , then randomly select an action from the optional actions as the action a at the current moment, otherwise, select the action that can obtain the maximum value function value from the optional actions as the action a at the current moment;

[0026] (4) Execute action a at the current moment and update N in-plant To obtain the state s' at the next moment, calculate the immediate reward r that can be obtained by the action, store (s, a, r, s') in the experience pool, let j = j + 1, if j ≥ J, then according to the PER method, extract J samples from the experience pool, and update the value function Q (s, a) of the agent based on the samples. The calculation formula is:

[0027] Q'(s,a)=Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)]

[0028] Among them, Q(s,a) and Q'(s,a) are the value functions of the state (s,a) before and after the update, respectively. During the initial learning, Q(s,a) is the initial value function, Q(s',a') is the value function of the state (s',a'), a and a' are the actions at the current moment and the next moment, respectively, α is the step size factor, r is the immediate reward obtained at the current moment, and γ is the decay rate;

[0029] (5) If the current action a is a load adjustment action, set s = s' and return to step (3); if the current action a is a scheme output action, go to step (6);

[0030] (6) If i ≥ I, the training is completed; otherwise, return to step (2).

[0031] Furthermore, in step S5, the method for generating other feasible combinations of starting units in combination with the rationality constraints of unit start-up and shutdown is:

[0032] Under the premise of being able to complete the load command, the power station can start or shut down at most one unit. To ensure the rationality of the unit start and stop, the following three constraints for the unit start and stop are set, and their calculation formulas are:

[0033] The number of units in operation and the number of units in shutdown should be within a reasonable range:

[0034] n on +n off =n

[0035] Among them, n on is the number of operating units in the power station, n on >0,n off is the number of shut-down units in the power station, n off >0, n is the number of units in the power station;

[0036] When the load of the power station increases, the number of units in operation cannot be reduced, and when the load of the power station decreases, the number of units in operation cannot be increased:

[0037]

[0038] Among them, n on,t+1 is the number of units in operation at time t+1, n on,t is the number of units in operation at time t, N c is the grid load instruction, is the current total output of the power station;

[0039] Avoid startup and shutdown operations that are contrary to future load trends:

[0040]

[0041] Among them, N on is the minimum output of the power station after the unit is turned on, N c is the minimum output in load forecasting, is the maximum output of the power station after shutting down the unit, is the maximum output in the load forecast;

[0042] The feasible startup unit combination schemes that can meet the above constraints are retained and used as the remaining feasible startup unit combination schemes except maintaining the status quo.

[0043] Furthermore, in step S5, the load optimization distribution model based on deep reinforcement learning is used to roll-calculate the station load distribution schemes and the objective function values ​​that can be obtained for all feasible startup unit combination schemes within the calculation period. The specific steps are as follows:

[0044] S521, read the load and flow prediction sequence of the power plant in the future period, and read the predicted values ​​of the load and flow in the prediction sequence in order; calculate whether the load prediction value is within the load range of the current startup unit combination, if not, update the startup unit combination of the power plant in combination with the calculation method of step S3 until the load instruction can be completed;

[0045] S522. Use the load optimization distribution model based on deep reinforcement learning to calculate the load distribution plan within the power station under the current combination of start-up and shutdown units and the other feasible combination of start-up and shutdown units, and calculate the objective function value that can be obtained by the load distribution plan in combination with the objective function. The objective function value calculation formula is as follows:

[0046]

[0047] Among them, f t is the objective function value that can be obtained by the load distribution scheme within the power station, Q g,t The power generation flow that the power station needs to achieve at the current moment to achieve the water scheduling target, n is the number of units in the power station, Q(N i,t ,H t ) is the function of power generation flow, unit output and water head, N i,t is the output of unit i at time t, H t is the real-time water head of the hydropower station, c is the number of starts and stops, Q p The penalty flow for opening and shutting down;

[0048] S523. Read the next load and flow prediction value in sequence, and return to step S512 until all load and flow prediction data are read and calculated.

[0049] Furthermore, in step S5, the objective function values ​​obtained by each scheme of the load distribution scheme within the station during the forecast period are summed to obtain the total objective function value obtained by each scheme during the forecast period; the startup unit combination scheme with the largest total objective function value is selected as the optimal startup unit combination scheme, and the start-up and shutdown status of the power station is updated according to the scheme.

[0050] In another embodiment of the present invention, a real-time load distribution system for a hydropower station based on deep reinforcement learning includes:

[0051] The model building and training module is used to build a load optimization allocation model based on deep reinforcement learning, and train the load optimization allocation model based on deep reinforcement learning in combination with historical operation data; the load optimization allocation model includes state, action, immediate reward and constraint, with all factors that will affect the load allocation decision of the power station as the state s; the action is designed with the idea of ​​load adjustment, and the allocation of a load instruction is converted into a process of gradual adjustment of the load allocation plan, including two types of actions: load adjustment and plan output; each action will have an immediate reward r; the constraints of the load optimization allocation model based on deep reinforcement learning include the constraints of the power station operation;

[0052] The data calculation module is used to receive the grid load instruction, calculate the minimum output and maximum output of each unit in the current startup unit combination, add the minimum output of each unit to obtain the lower boundary of the power station output range, and add the maximum output of each unit to obtain the upper boundary of the power station output range;

[0053] A judgment execution module is used to judge whether the current combination of startup units can complete the load instruction in combination with the value of the load instruction; if the load instruction can be completed, the current combination of startup units is kept unchanged, and it is judged whether there are other feasible combinations of startup units. If there are no other feasible combinations of startup units, the load optimization distribution model based on deep reinforcement learning is used to generate an optimized load distribution plan between the startup units. If there are other feasible combinations of startup units, the startup unit combination of the power station is updated according to the unit update module; if the load instruction cannot be completed, if the power grid load instruction is greater than the upper boundary of the power station's output range, the unit with the longest current shutdown time is turned on, and the start-stop state and load range of the power station unit are updated until the power grid load instruction is between the lower boundary of the power station's output range and the upper boundary of the power station's output range; if the power grid load instruction is less than the lower boundary of the power station's output range, the unit with the longest current startup time is turned off, and the start-stop state and load range of the power station unit are updated until the power grid load instruction is between the lower boundary of the power station's output range and the upper boundary of the power station's output range;

[0054] The unit update module is used to generate other feasible startup unit combinations based on the rationality constraints of unit start-up and shutdown, and to use the load optimization distribution model based on deep reinforcement learning to rollingly calculate the station load distribution scheme and the objective function value that can be obtained for all feasible startup unit combination schemes within the calculation period in combination with the load and flow forecast information in the future; calculate the total objective function value that can be obtained for each scheme within the forecast period, select the optimal startup unit combination based on the total objective function value, and update the startup unit combination of the power station.

[0055] In yet another embodiment of the present invention, a device includes a memory and a processor, wherein:

[0056] A memory for storing computer programs that can be run on the processor;

[0057] A processor is used to execute the steps of the above-mentioned method for real-time load distribution of a hydropower station based on deep reinforcement learning when running the computer program.

[0058] A storage medium of the present invention stores a computer program, which, when executed by at least one processor, implements the steps of the above-mentioned method for real-time load distribution of a hydropower station based on deep reinforcement learning.

[0059] Beneficial effects: Compared with the prior art, the significant technical effects of the present invention are:

[0060] The method of the present invention uses a deep reinforcement learning algorithm to construct a load optimization distribution model based on deep reinforcement learning, uses the power station operation history data to train the load optimization distribution model based on deep reinforcement learning and thereby realizes the rapid distribution of real-time load instructions for the hydropower station; after the power station receives the load instruction, the method first updates the power station's startup unit combination in combination with the current load instruction, and then optimizes the startup unit combination in combination with the rolling calculation of the load and flow forecast information, and finally completes the distribution of the load instruction on the startup unit combination. The method proposed by the present invention takes into account the water scheduling requirements of the power station in the objective function, so that when the method makes a load distribution plan, it can further optimize the unit start-stop plan and adjust the unit load distribution plan on the premise of completing the power generation scheduling requirements, and adjust the power generation flow of the power station in accordance with the water scheduling requirements of the power station. At the same time, the method uses deep reinforcement learning to realize load distribution, which can alleviate the dimensionality disaster problem and realize high-precision and high-efficiency real-time load distribution decision. Compared with the commonly used methods in the current real-time load distribution in the station, the method proposed by the present invention can take into account the power generation scheduling requirements and water scheduling requirements of the power station, and can take into account the decision-making accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the method of the present invention;

[0062] Figure 2 It is a schematic diagram of the load adjustment action;

[0063] Figure 3 This is the overall architecture diagram of the deep reinforcement learning D3QN algorithm;

[0064] Figure 4 It is a curve chart of the cumulative rewards obtained during the learning process of the deep reinforcement learning D3QN algorithm;

[0065] Figure 5 It is the water level process curve of Shenxigou Power Station in two test periods;

[0066] Figure 6 This is the unit operation process curve of Shenxigou Power Station during the dry season test period;

[0067] Figure 7 It is the unit operation process curve of Shenxigou Power Station during the flood season test period. DETAILED DESCRIPTION

[0068] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0069] The present invention aims to solve the problem that when existing small and medium-sized hydropower stations undertake peak-shaving and frequency-regulating tasks, load instruction allocation only considers the requirements of power generation scheduling and ignores the requirements of water volume scheduling, which is prone to large fluctuations in water level and flow and frequent adjustments of flood discharge facilities. The present invention provides a real-time load allocation method for hydropower stations based on deep reinforcement learning, which can guide the real-time load allocation of power stations and solve the problems that it is difficult to consider the requirements of water volume scheduling in the real-time load allocation decision of current power stations and it is difficult to balance the decision-making accuracy and efficiency.

[0070] The present invention provides a real-time load distribution method for a hydropower station based on deep reinforcement learning, which can be used to efficiently solve the real-time load distribution problem within the hydropower station, including: sorting out the operation requirements of the power station, constructing a load optimization distribution model based on deep reinforcement learning, and using the power station operation history data to train the load optimization distribution model based on deep reinforcement learning; after the power station receives the load instruction, it updates the startup unit combination of the power station in combination with the current load instruction, and further sets the remaining feasible startup unit combinations on this basis, and uses the load optimization distribution model based on deep reinforcement learning to perform rapid rolling calculations on the forecast information of the flow and load instructions to optimize the startup unit combination; the startup unit combination of the power station is updated, and the load optimization distribution model based on deep reinforcement learning is used to solve the optimal distribution plan of the power grid load instructions among the startup units. The present invention is suitable for real-time load distribution of hydropower stations and can be used to guide the real-time operation of hydropower station units. Figure 1 As shown, the specific steps include:

[0071] S1. Build a load optimization distribution model based on deep reinforcement learning, and train the load optimization distribution model based on deep reinforcement learning in combination with historical operation data. Use the deep reinforcement learning D3QN algorithm to build a load optimization distribution model based on deep reinforcement learning to achieve load optimization distribution between fixed startup units. The specific steps are as follows:

[0072] S11. The deep reinforcement learning model consists of two parts: the agent and the environment. The agent obtains immediate rewards by interacting with the environment to guide actions, and its learning goal is to maximize the cumulative rewards. Among them, the environmental model needs to simulate the computing environment of the load distribution problem within the power station, including the state, immediate rewards and constraints of the power station operation. The agent continuously obtains feedback information in the environment by executing actions, thereby continuously improving the accuracy of its value function to assist the agent to better judge the value of each action in a certain state, including action and agent learning algorithms.

[0073] (1) Status

[0074] Taking all factors that will affect the load distribution decision of the power station as the state s, the present invention selects the real-time inflow flow Q of the power station in , flood discharge Q flood , power generation flow Q generate , the power generation flow Q required to achieve the water dispatch target of the power station g,t , upstream water level Z, unit load N in-plant Construct a one-dimensional tensor [Q in ,Q flood ,Q generate ,Q g,t ,Z,N in-plant ] as the status.

[0075] (2) Constraints

[0076] Corresponding constraints need to be set in the environmental model to ensure that its output results can meet the power plant operation requirements. The constraints of this model take into account the various operating constraints that need to be met during the actual operation of the power plant, as follows:

[0077] Unit operation constraints:

[0078] N i,t ≤N i,t,max

[0079]

[0080]

[0081] Among them, N i,t is the output of unit i at time t, N i,t,max is the maximum output that unit i can achieve at time t, n is the total number of units in the power station, N c is the load instruction, is the lower boundary of the vibration zone of unit i at time t, is the upper boundary of the vibration zone of unit i at time t.

[0082] Water balance constraints:

[0083]

[0084] Among them, V t+1 is the water storage capacity of the power station at time t+1, V t is the water storage capacity of the power station at time t, Q in,t is the inflow flow of the power station at time t; Q flood,t is the flood discharge of the power station at time t, and △t is the time interval between time t+1 and time t.

[0085] Water level constraints:

[0086] Z min ≤Z t ≤Z max

[0087] Among them, Z max and Z min They are the normal water storage level and dead water level of the power station respectively.

[0088] Water level-storage relationship constraints:

[0089] Z=f(V)

[0090] Where Z is the upstream water level of the power station, V is the water storage capacity of the power station, and f(V) is the function calculation formula of the upstream water level of the power station when the water storage capacity of the power station is V.

[0091] Tailwater level-outflow relationship constraints:

[0092] Z tail =g(Q out )

[0093] Among them, Z tail is the tailwater level of the power station, Q out is the outflow flow of the power station, g(Q out ) is the outflow flow of the power station, Q out Function calculation formula for the tailwater level of the power station.

[0094] (3) Action

[0095] The load optimization distribution model based on deep reinforcement learning converts the distribution of a load instruction into a process of gradually adjusting the load distribution plan, which includes two types of actions: load adjustment and plan output. The model sets a fixed unit load according to the required load discrete accuracy. The load adjustment action can adjust a unit load between two running units to form a new load distribution plan. The plan output action means ending the optimization of the load distribution plan and outputting the optimized load distribution plan.

[0096] When selecting actions, the load optimization allocation model based on deep reinforcement learning needs to shield actions that will cause the unit load to fall into the vibration zone and the unit load to be negative.

[0097] Since the load optimization distribution model based on deep reinforcement learning in the present invention has a fixed combination of operating units when making decisions, the load distribution only needs to quickly solve the optimal load distribution plan under the fixed combination of operating units according to the current load instruction when the water head H is determined. When the water head is fixed at H, taking a hydropower station with two operating units as an example, the relationship between the load and power generation flow of the operating units is as follows: Figure 2 As shown in (a), N a Indicates the load of the starting unit 1, N b Indicates the load of the startup unit 2. When the total load N is given, N a +N b = N is the cross section, and the relationship curve between the load and power generation flow between the starting units can be obtained when the total load is fixed at N, such as Figure 2 (b) The task of this model is to quickly search for the power generation flow closest to Q on this curve. g,t load distribution scheme.

[0098] If the load value that the unit needs to bear is directly used as the action, a large number of actions will be generated, which will increase the learning difficulty of the intelligent agent. Considering that the combination of powered-on units is already fixed, combined with the analysis in the previous paragraph, in order to reduce the number of actions and improve the decision-making effect of the model, the present invention adopts the idea of ​​load adjustment to design actions. This idea converts the allocation of a load instruction into a process of gradual adjustment of the load distribution plan, which includes two types of actions: load adjustment and plan output. The model sets a fixed unit load △N according to the required load discrete accuracy. The load adjustment action can adjust a unit load between two powered-on units to form a new load distribution plan, such as Figure 2 As shown in (b), after a unit load is adjusted from the No. 2 operating unit to the No. 1 operating unit, the initial solution That is, it changes to in, This means that the unit load distribution plan is changed from Changes to The power generation flow of the power station is Q 0 Change to Q 1 . The plan output action means ending the optimization of the load distribution plan and outputting the plan. To ensure the accuracy of learning effect and decision results, when the intelligent agent selects actions, some unreasonable actions can be avoided through simple judgment: (1) endless cycle load adjustment actions, for example, after transferring the load from unit 1, the load is transferred back to the unit; (2) actions that will produce infeasible solutions, for example, actions that will cause the unit load to fall into the vibration zone and cause the unit load to be negative.

[0099] (4) Instant Rewards

[0100] The calculation formula for the immediate reward r of each action is:

[0101]

[0102] Among them, r is the immediate reward of the deep reinforcement learning model, s j and j+1 are the states before and after the action is executed, respectively. and are the output of unit i before and after the action is executed, respectively, H is the current water head of the power station, and The units are respectively and The power generation flow when the water head is H, Q g,t is the power generation flow required to complete the water dispatching target of the power station at time t. When the power station needs to store water, Q g,t It can be taken as the minimum downstream flow of the power station. When the power station has a water level control demand, Q g,t The calculation formula is:

[0103] Q g,t =Q in,t -Q flood,t +(V(Z t )-V(Z g,t )) / △t

[0104] Among them, Q in,t is the inflow flow of the power station at time t, Q flood,t is the flood discharge of the power station at time t, V(Z t ) and V(Z g,t ) are water levels Z t and Z g,t When the water storage capacity of the power station reservoir is t is the water level of the power station at time t, Z g,t is the target water level of the power station at time t, and △t is the time interval between load instructions.

[0105] S12. The load optimization distribution model based on deep reinforcement learning makes decisions through an intelligent agent, which is a neural network containing a large number of parameters. The neural network needs to use a large amount of data to learn and update its own parameters. The present invention uses the inflow flow Q in the actual dispatching operation of the power station in recent years. in , flood discharge Q flood , power generation flow Q generate , the power generation flow Q required to achieve the water dispatch target of the power station g,t, upstream water level Z, unit load N in-plant The power plant operation data in the historical operation database is used to build a historical operation database of the power plant. The model is trained using the power plant operation data in the historical operation database. The specific training steps are as follows:

[0106] Step 1: Randomly generate an initial value function Q(s,a), set counter i=0, counter j=0, set the number of training rounds to I, the number of samples to J, and the probability of randomly selecting an action to P r , generate a sample library that is identical to the power plant's historical operation database;

[0107] Step 2: In the sample library, a piece of operation data is randomly selected by sampling without replacement, and a load instruction N is randomly generated while keeping the current combination of the starting units unchanged. c The feasible load distribution scheme N in-plant , combining the operation data and the feasible load distribution plan to construct the current state s = [Q in ,Q flood ,Q generate ,Q g,t ,Z,N in-plant ], if all the data in the sample library are sampled, a sample library identical to the power plant historical operation database is regenerated, and i=i+1;

[0108] Step 3: Input the current s into the load optimization distribution model, judge and shield the invalid actions that will cause the unit load to fall into the vibration zone or become negative, and generate a random number p between [0,1]. If p≤P r , then randomly select an action from the optional actions as the action a at the current moment, otherwise, select the action that can obtain the maximum value function value from the optional actions as the action a at the current moment;

[0109] Step 4: Execute action a at the current moment and update N in-plant To obtain the state s' at the next moment, calculate the immediate reward r that can be obtained by the action, store (s, a, r, s') in the experience pool, let j = j + 1, if j ≥ J, then according to the PER method, extract J samples from the experience pool, and update the value function Q (s, a) of the agent based on the samples. The calculation formula is:

[0110] Q'(s,a)=Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)]

[0111] Among them, Q(s,a) and Q'(s,a) are the value functions of the state (s,a) before and after update respectively. When learning for the first time, Q(s,a) is the initial value function, Q(s',a') is the value function of the state (s',a'), s and s' are the states at the current moment and the next moment respectively, a and a' are the actions at the current moment and the next moment respectively, α is the step size factor, r is the immediate reward obtained at the current moment, and γ is the decay rate.

[0112] Step 5: If the action a at the current moment is a load adjustment action, let s = s' and return to Step 3. If the action a at the current moment is a scheme output action, go to Step 6;

[0113] Step 6: If i ≥ I, the training is completed. Otherwise, return to Step 2.

[0114] After the construction and training of the deep reinforcement learning model are completed, the model can be applied to the decision-making of the real-time load distribution scheme. The process is as Figure 3 shown. After fixing the unit combination of the power station in operation, input the initial load distribution scheme into the model, and the model can output the optimized load distribution scheme.

[0115] S2. After the power station receives the grid load, it combines the value of the load instruction to judge whether the current unit combination in operation can complete the load instruction. The specific steps are as follows: To ensure the safe operation of the grid, after receiving the load instruction, it is first necessary to ensure that the units in operation of the current power station can complete the current load instruction. Therefore, it is necessary to read the start-stop status, real-time water head H, and load instruction N of each unit in the current power station c , and calculate the minimum output and maximum output of each unit according to H; adding the minimum outputs of each unit can obtain the lower boundary N of the output range of the power station, and adding the maximum outputs of each unit can obtain the upper boundary of the output range of the power station Thus, the load range that the power station can bear under the current unit combination in operation can be obtained If then keep the current unit combination in operation unchanged and enter Step S4. Otherwise, enter Step S3.

[0116] S3. If then a new unit needs to be started to complete the load instruction. To balance the start-up time of each unit and extend the service life of the unit, start the unit with the longest shutdown time at present. If N c < N, then shut down the unit with the longest start-up time at present; update the start-stop status and load range of the power station units, and repeat the above judgment until the load instruction is within the load range that the power station can bear.

[0117] S4, determine whether there are other feasible startup unit combinations, if there are no other feasible startup unit combinations, directly go to step S6, otherwise, go to step S5;

[0118] S5. Combine the rationality constraints of the start and stop of the units to generate the remaining feasible combinations of the startup units. Combine the load and flow forecast information for a period of time in the future, and use the load optimization allocation model based on deep reinforcement learning to roll-calculate the load allocation schemes in the station and the objective function values ​​that can be obtained for all feasible startup unit combination schemes within the calculation period; calculate the total objective function value that can be obtained for each scheme within the forecast period, select the optimal startup unit combination based on the total objective function value, and update the startup unit combination of the power station;

[0119] S51. Generate other feasible startup unit combinations based on the rationality constraints of unit startup and shutdown. The specific steps are as follows:

[0120] In order to avoid large-scale start and stop of power plant units, under the premise that the current combination of starting units can complete the load instruction, the present invention assumes that the power plant can only start or shut down one more unit at most. Therefore, in the optimization calculation of the combination of starting units, based on the current start and stop status of the power plant, two combination schemes of starting one unit and shutting down one unit are further set.

[0121] Read the load command forecast data for a period of time in the future, and determine whether the above two startup unit combination schemes are reasonable based on the following constraints:

[0122] The number of units in operation and the number of units in shutdown should be within a reasonable range:

[0123] n on +n off =n

[0124] Among them, n on is the number of operating units in the power station, n on >0,n off is the number of shut-down units in the power station, n off >0, n is the number of units in the power station.

[0125] When the load of the power station increases, the number of units in operation cannot be reduced, and when the load of the power station decreases, the number of units in operation cannot be increased:

[0126]

[0127] Among them, n on,t+1 is the number of units in operation at time t+1, n on,t is the number of units in operation at time t, N c is the grid load instruction, is the current total output of the power station.

[0128] Avoid startup and shutdown operations that are contrary to future load trends. For example, after starting the unit, if the minimum output of the power plant is higher than the minimum load in the forecast period, the power plant will be shut down again. After shutting down the unit, if the maximum output of the power plant is lower than the maximum load in the forecast period, the power plant will be started again:

[0129]

[0130] Among them, N on is the minimum output of the power station after the unit is turned on, N c is the minimum output in load forecasting, is the maximum output of the power station after shutting down the unit, It is the maximum output in load forecast.

[0131] The feasible startup unit combination schemes that can meet the above constraints are retained and used as the remaining feasible startup unit combination schemes except maintaining the status quo.

[0132] When turning on or off a unit, the unit with the longest shutdown time will be turned on, and the unit with the longest startup time will be turned off.

[0133] S52. Use the load optimization distribution model based on deep reinforcement learning to calculate the station load distribution schemes and the objective function values ​​that can be obtained for all feasible startup unit combination schemes within the calculation period. The specific steps are as follows:

[0134] S521. Read the load and flow forecast sequence of the power plant in the future, and read the predicted values ​​of the load and flow in the forecast sequence in order; calculate whether the load forecast value is within the load range of the current startup unit combination. If not, update the startup unit combination of the power plant in combination with the calculation method of step S3 until the load instruction can be completed.

[0135] S522. Use the load optimization distribution model based on deep reinforcement learning to calculate the load distribution plan within the power station under the current combination of start-up and shutdown units and the other feasible combination of start-up and shutdown units, and calculate the objective function value that can be obtained by the load distribution plan in combination with the objective function. The objective function value calculation formula is as follows:

[0136]

[0137] Among them, f t is the objective function value that can be obtained by the load distribution scheme within the power station, Q g,t The power generation flow that the power station needs to achieve at the current moment to achieve the water scheduling target, n is the number of units in the power station, Q(N i,t ,H t ) is the function of power generation flow, unit output and water head, Ni,t is the output of unit i at time t, H t is the real-time water head of the hydropower station, c is the number of starts and stops, Q p Penalty flow for opening and closing.

[0138] S523. Read the next load and flow prediction value in sequence, and return to step S522 until all load and flow prediction data are read and calculated.

[0139] S53. Calculate the total objective function value that can be obtained by each scheme within the forecast period, select the optimal startup unit combination based on this value, and update the startup unit combination of the power station. The specific steps are as follows: sum up the objective function values ​​obtained by each scheme of each load distribution scheme within the station within the forecast period to obtain the total objective function value obtained by each scheme within the forecast period; select the startup unit combination scheme with the largest total objective function value as the optimal startup unit combination scheme, and update the start-up and shutdown status of the power station according to this scheme.

[0140] S6. Combined with the current start-up and shutdown status of the power station, the load optimization distribution model based on deep reinforcement learning is used to generate a real-time load distribution plan under the combination of the start-up units. The calculation process is as follows: Figure 3 As shown, after fixing the combination of startup units, the initial load distribution plan is input into the model, and the model can output the optimized load distribution plan.

[0141] The present invention is specifically described by the real-time load distribution of the Shenxigou Power Station in the Dadu River Basin. The Dadu River is located in the central and western part of Sichuan Province, China, and is the largest tributary of the Minjiang River. The main stream of the Dadu River is 1,062 km long and has a drainage area of ​​77,400 km. 2 , with an annual runoff of 47 billion m 3 , rich in hydropower resources, is the fifth largest hydropower base in China. At present, 22 hydropower stations have been built on the Dadu River. Shenxigou Hydropower Station is the 18th hydropower station on the mainstream of the Dadu River. The power station is equipped with 4 axial-flow units with a single unit capacity of 165,000 kW and a total installed capacity of 660,000 kW. Shenxigou Hydropower Station is one of the hydropower stations that undertakes the peak-shaving and frequency-regulating tasks of the Sichuan power grid. Its load instructions fluctuate greatly, and the response speed requirements are high. Since Shenxigou Power Station is a daily regulation power station, its reservoir capacity is small. Affected by the peak-shaving and frequency-regulating requirements, there are problems such as insufficient power generation benefits in the dry season and large fluctuations in water level and flow in the flood season. The present invention applies the above-mentioned real-time load distribution method for hydropower stations based on deep reinforcement learning to the solution of the real-time load distribution problem within the Shenxigou Power Station:

[0142] Since the operating objectives of power stations in the dry season and flood season are different, it is usually necessary to reduce the power generation flow in the dry season to improve the power generation efficiency. In the flood season, the power station usually has flood discharge flow, and the units need to regulate the power generation flow in accordance with the water level control objectives of the power station. Therefore, a historical operation database of the power station is constructed for the dry season and flood season respectively, and two load optimization distribution models based on deep reinforcement learning are constructed.

[0143] The five-minute operation data of the Shenxigou Power Station from 2017 to January 2019 was used to construct a historical dispatching and operation database of the power station during the dry season. The five-minute operation data of the Shenxigou Power Station from 2017 to August 2019 was used to construct a historical dispatching and operation database of the power station during the flood season.

[0144] Combined with the unit settings of Shenxigou Power Station, two types of actions, load adjustment and scheme output, are designed based on load adjustment. The scheme output action is taken as action No. 0. In addition, the load adjustment action includes 12 possible load adjustment situations between the four units, recorded as actions No. 1-12. The unit load that can be adjusted between the starting units in each action is 1MW. The unit numbers corresponding to each action are shown in Table 1.

[0145] Table 1

[0146]

[0147]

[0148] The target power generation flow Q of the power station is set according to the water dispatching demand of Shenxigou Power Station in the dry season and flood season respectively. g,t Combining the reward function of the load optimization distribution model based on deep reinforcement learning, it can be seen that Q g,t When Q is set to a fixed and sufficiently small value, the power generation flow of the power station can be close to this value to achieve the effect of minimizing water consumption. Therefore, the load optimization distribution model based on deep reinforcement learning for the dry season sets Q g,t Fixed as the ecological flow of Shenxigou Power Station, 327m 3 / s, the power station has a water level control target during the flood season. Since the regulating reservoir capacity of Shenxigou Power Station is small, the water level is prone to exceed the limit during the flood season. The present invention fixes the water level target of the power station during the flood season to the middle water level of the power station, 657.5m, and uses this water level as the target water level to calculate Q g,t The parameters of the two models are shown in Table 2.

[0149] Table 2

[0150] parameter Low water season model Flood season model Network structure 128×1024×512 256×1024×512 Memory pool capacity 1,000,000 1,000,000 Number of samples 64 128 Iteration training times 200 200 Discount Factor 0.9 0.9 Learning Rate 0.001 0.0005

[0151] The power plant operation data in the historical operation database is used to train the model, and the completion of random sampling learning of all data in the training set is regarded as one round. The power plant operation data in January 2017, January 2018, and January 1 to 20, 2019 are used as the training set for real-time load distribution in the station during the dry season, and the power plant operation data in August 2017, August 2018, and August 1 to 20, 2019 are used as the training set for real-time load distribution in the station during the flood season. The intelligent agent is trained on the two training sets for 100 rounds respectively, and the cumulative rewards obtained by the intelligent agent during this period are as follows: Figure 4 As shown, Figure 4 (a) shows the cumulative rewards obtained by the agent after 100 rounds of training on the real-time load distribution training set in the dry season. Figure 4 (b) shows the cumulative reward obtained by the agent after 100 rounds of training on the real-time load distribution training set in the flood season. The learning effect of the agent in the two trainings is similar. The cumulative reward value obtained in the first 10 rounds of learning is greatly improved, but the cumulative reward value in the first 50 rounds of learning has a large fluctuation. This is mainly because the agent has a high probability of taking random actions in the early exploration process. In the 50th to 80th round of learning, the cumulative reward value still has a small increase. After 80 rounds of learning, the fluctuation of the cumulative reward value gradually decreases and tends to converge, which means that the model has learned a better load distribution strategy.

[0152] The trained model is used in the real-time dispatching decision-making of the power station. The actual five-minute operation data of Shenxigou Power Station from January 21 to 23, 2019 is used as the test data for real-time load distribution in the station during the dry season, and the total power generation water consumption is used as its evaluation index. The power station operation data of Shenxigou Power Station from August 21 to 23 is used as the test data for real-time load distribution in the station during the flood season, and the cumulative deviation between the operating water level and the target water level is used as its evaluation index.

[0153] In order to verify the effectiveness of the model proposed in this invention, a load distribution master table with a load discrete accuracy of 2MW is used to make real-time load distribution decisions within the station and compare them. In order to verify whether rolling calculations considering forecast information can improve the start and stop plans of the units, comparison plans considering forecast information and not considering forecast information are set up respectively, and the load distribution master table is used to implement the load distribution calculations of the two plans. In order to verify whether the D3QN algorithm can take into account both decision-making efficiency and accuracy, the D3QN algorithm and the load distribution master table are used to calculate the real-time load distribution plan within the station. Combined with the above analysis, the following three comparison plans are set up:

[0154] Solution 1: Pre-build the load distribution table and use it to calculate the load distribution plan. When making decisions on the combination of startup units, only the real-time load instructions are considered, and flow penalties are set for startup and shutdown.

[0155] Solution 2: Pre-build the load distribution table, consider the rolling calculation of forecast information, and use the load distribution table to calculate the load distribution plan. The forecast data uses the five-minute load and flow forecast data for the next two hours.

[0156] Solution 3: Use historical data to train the deep reinforcement learning model in advance, consider the rolling calculation of forecast information, and use the trained load optimization distribution model based on deep reinforcement learning to calculate the load distribution plan. The forecast data uses the load and flow forecast data within the next two hours.

[0157] Schemes 1, 2 and 3 are used to distribute the load within the power station in the dry season test set and the flood season test set. The indicators and unit start and stop times related to the water volume dispatching target of each scheme are shown in Table 3. The operating water level of the power station in the dry season test set is shown in Table 3. Figure 5 As shown in (a), the operating water level of the power station in the flood season test set is Figure 5 (b) shows the change of the number of units started for each scheme in the dry season test set. Figure 6 As shown in (a), the unit load process of Scheme 1 in the dry season test set is as follows Figure 6 As shown in (b), the unit load process of Scheme 2 in the dry season test set is as follows Figure 6 As shown in (c), the unit load process of Scheme 3 in the dry season test set is as follows Figure 6 (d) shows the changes in the number of units started for each scheme in the flood season test set. Figure 7 As shown in (a), the unit load process of Scheme 1 in the flood season test concentration is as follows Figure 7 As shown in (b), the unit load process of the flood season test concentration scheme 2 is as follows Figure 7 As shown in (c), the unit load process of Scheme 3 in the flood season test is as follows: Figure 7 (d) as shown.

[0158] Table 3

[0159]

[0160]

[0161] The advantages of the method proposed in the present invention are compared from three aspects: unit operation strategy, optimization effect and calculation time consumption:

[0162] In terms of unit operation strategy, both Scheme 1 and Scheme 2 use the load distribution master table to calculate the load distribution plan, but Scheme 2 adds a rolling calculation for forecast data. Compared with Scheme 1, after considering the forecast information, Scheme 2 has achieved better results in the evaluation indicators of water scheduling, although the number of unit starts and stops in the minimum water consumption case and the water level control case has increased by one. This verifies that the rolling calculation combined with forecast information can effectively optimize the start and stop strategy of the unit according to the future load and flow trends of the power station and the water scheduling objectives of the power station, so that the power station can achieve better water scheduling effects.

[0163] In terms of optimization effect, both Scheme 2 and Scheme 3 take forecast information into consideration. However, when calculating the load distribution plan, Scheme 2 uses a load distribution table with a load discrete accuracy of 2MW, and Scheme 3 uses a load optimization distribution model based on deep reinforcement learning with a load discrete accuracy of 1MW. Compared with Scheme 2, Scheme 3 has achieved better results in the evaluation indicators of water volume scheduling, which verifies that the load optimization distribution model based on deep reinforcement learning can effectively optimize the real-time load distribution plan within the power station.

[0164] The calculation time of each solution is shown in Table 4.

[0165] Table 4

[0166] Dry Season Case(s) Flood season case(s) Solution 1 168.27 174.64 Solution 2 2132.63 1845.30 Solution 3 201.75 278.69

[0167] Since the rolling calculation of forecast information greatly increases the amount of calculation of the model, in the dry season case and the flood season case, the average time for a single decision in Scheme 2 is 4.94s and 2.14s, respectively, which are about 13 times and 11 times that of Scheme 1, respectively, and it is difficult to meet the time requirements of real-time load distribution. Both Scheme 2 and Scheme 3 consider forecast information, and the load accuracy of Scheme 3 is higher than that of Scheme 2, but the average time for a single decision in Scheme 3 in the dry season case is 0.23s, which is only 4.66% of Scheme 2, and the average time for a single decision in the flood season case is 0.32s, which is only 14.95% of Scheme 2. This verifies that the method proposed in the present invention can take into account the decision-making accuracy and speed of load distribution, and can effectively improve the response effect and response speed of the power station to the power grid load command.

Claims

1. A real-time load distribution method for a hydropower station based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Construct a load optimization allocation model based on deep reinforcement learning, and train the load optimization allocation model based on deep reinforcement learning in combination with historical operation data; the load optimization allocation model includes states, actions, rewards and constraints, with all factors that will affect the load allocation decision of the power station as states s; The action is designed based on the idea of ​​load adjustment, and the distribution of a load instruction is converted into a process of gradual adjustment of the load distribution plan, which includes two types of actions: load adjustment and plan output. Each action will have an immediate reward r. The constraints of the load optimization distribution model based on deep reinforcement learning include the constraints of power station operation. S2, the power station receives the load instruction of the power grid, calculates the minimum output and maximum output of each unit in the current combination of the starting units, adds the minimum output of each unit to obtain the lower boundary of the output range of the power station, and adds the maximum output of each unit to obtain the upper boundary of the output range of the power station; based on the value of the load instruction, it is determined whether the current combination of the starting units can complete the load instruction; if the load instruction can be completed, the current combination of the starting units is kept unchanged and step S4 is executed; otherwise, step S3 is executed; S3. If the grid load instruction is greater than the upper boundary of the power station's output range, the unit with the longest shutdown time is turned on, and the start and stop status and load range of the power station units are updated until the grid load instruction is between the lower boundary of the power station's output range and the upper boundary of the power station's output range; If the grid load instruction is less than the lower boundary of the power plant's output range, the unit with the longest current startup time is shut down, and the startup and shutdown status and load range of the power plant units are updated until the grid load instruction is between the lower boundary of the power plant's output range and the upper boundary of the power plant's output range; S4, determine whether there are other feasible startup unit combinations, if there are no other feasible startup unit combinations, directly go to step S6, otherwise, go to step S5; S5. Combine the rationality constraints of unit start-up and shutdown to generate other feasible combinations of starting units. Combine the load and flow forecast information for a period of time in the future, and use the load optimization allocation model based on deep reinforcement learning to roll-calculate the station load allocation schemes and the objective function values ​​that can be obtained for all feasible combinations of starting units within the calculation period. Calculate the total objective function value that can be achieved by each scheme within the forecast period, select the optimal combination of startup units based on the total objective function value, and update the startup unit combination of the power station; The method of generating other feasible startup unit combinations by combining the rationality constraints of unit startup and shutdown is as follows: Under the premise of being able to complete the load command, the power station can start or shut down at most one unit. To ensure the rationality of the unit start and stop, the following three constraints for the unit start and stop are set, and their calculation formulas are: The number of units in operation and the number of units in shutdown should be within a reasonable range: n on +n off =n Among them, n on is the number of operating units in the power station, n on >0,n off is the number of shut-down units in the power station, n off >0, n is the number of units in the power station; When the load of the power station increases, the number of units in operation cannot be reduced, and when the load of the power station decreases, the number of units in operation cannot be increased: Among them, n on,t+1 is the number of units in operation at time t+1, n on,t is the number of units in operation at time t, N c is the grid load instruction, is the current total output of the power station; Avoid startup and shutdown operations that are contrary to future load trends: Among them, N on is the minimum output of the power station after the unit is turned on, N c is the minimum output in load forecasting, is the maximum output of the power station after shutting down the unit, is the maximum output in the load forecast; The feasible startup unit combination scheme that can meet the above constraints is retained and used as the remaining feasible startup unit combination scheme except maintaining the status quo; The load optimization distribution model based on deep reinforcement learning is used to roll out the calculation of the station load distribution schemes and the objective function values ​​that can be obtained for all feasible startup unit combination schemes during the calculation period. The specific steps are as follows: S521, read the load and flow prediction sequence of the power plant in the future period, and read the predicted values ​​of the load and flow in the prediction sequence in order; calculate whether the load prediction value is within the load range of the current startup unit combination, if not, update the startup unit combination of the power plant in combination with the calculation method of step S3 until the load instruction can be completed; S522. Use the load optimization distribution model based on deep reinforcement learning to calculate the load distribution plan within the power station under the current combination of start-up and shutdown units and the other feasible combination of start-up and shutdown units, and calculate the objective function value that can be obtained by the load distribution plan in combination with the objective function. The objective function value calculation formula is as follows: Among them, f t is the objective function value that can be obtained by the load distribution scheme within the power station, Q g,t The power generation flow that the power station needs to achieve at the current moment to achieve the water scheduling target, n is the number of units in the power station, Q(N i,t ,H t ) is the function of power generation flow, unit output and water head, N i,t is the output of unit i at time t, H t is the real-time water head of the hydropower station, c is the number of starts and stops, Q p The penalty flow for opening and shutting down; S523, read the next load and flow prediction value in sequence, and return to step S512 until all load and flow prediction data are read and calculated; The objective function values ​​of each scheme obtained by each load distribution scheme in the forecast period are summed up to obtain the total objective function value of each scheme in the forecast period; the combination scheme of the starting units with the largest total objective function value is selected as the optimal combination scheme of the starting units, and the start and stop status of the power station is updated according to the scheme; S6. Use the load optimization distribution model based on deep reinforcement learning to generate an optimized load distribution plan among the operating units.

2. A real-time load distribution method for a hydropower station based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the state is a one-dimensional tensor [Q in ,Q flood ,Q generate ,Q g,t ,Z,N in-plant ], where Q in is the inflow flow, Q flood is the flood discharge, Q generate is the power generation flow, Q g,t is the power generation flow required to achieve the water dispatch target of the power station at time t, Z is the upstream water level, N in-plant is the unit load; The calculation formula of instant reward r is: Among them, r is the immediate reward of the deep reinforcement learning model; n is the total number of units; s j and j+1 They are the states before and after the action is performed; and are the output of unit i before and after the action is performed; H is the current water head of the power station; and The units are respectively and The power generation flow when the water head is H; The constraints include the constraints of power station operation: unit operation constraints, water balance constraints, water level constraints, water level-water storage relationship constraints and tailwater level-outflow relationship constraints.

3. A real-time load distribution method for a hydropower station based on deep reinforcement learning according to claim 2, characterized in that: By calculating the power generation flow Q required to achieve the water dispatch target of the power station g,t To optimize the power generation flow of the power station, when the power station needs to store water, Q g,t It can be taken as the minimum downstream flow of the power station. When the power station has a water level control demand, Q g,t The calculation formula is: Q g,t =Q in,t -Q flood,t +(V(Z t )-V(Z g,t )) / △t Among them, Q in,t is the inflow flow of the power station at time t, Q flood,t is the flood discharge of the power station at time t, V(Z t ) and V(Z g,t ) are water levels Z t and Z g,t When the water storage capacity of the power station reservoir is t is the water level of the power station at time t, Z g,t is the target water level of the power station at time t, and △t is the time interval between load instructions.

4. A real-time load distribution method for a hydropower station based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of the training process of the load optimization distribution model based on deep reinforcement learning in step S1 include: (1) Randomly generate an initial value function Q(s,a), set counter i = 0, counter j = 0, set the number of training rounds to I, the number of samples to J, and the probability of randomly selecting an action to P r , generate a sample library that is identical to the power plant's historical operation database; (2) In the sample library, a piece of operating data is randomly selected by sampling without replacement, and a load instruction N is randomly generated while keeping the current combination of the operating units unchanged. c The feasible load distribution scheme N in-plant , combining the operation data and the feasible load distribution scheme to construct the current state s=[Q in ,Q flood ,Q generate ,Q g,t ,Z,N in-plant ], if all the data in the sample library are sampled, a sample library identical to the power plant historical operation database is regenerated, and i=i+1; (3) Input the current state s into the load optimization allocation model, judge and shield the invalid actions that will cause the unit load to fall into the vibration zone or become negative, and generate a random number p between [0,1]. If p≤P r , then randomly select an action from the optional actions as the action a at the current moment, otherwise, select the action that can obtain the maximum value function value from the optional actions as the action a at the current moment; (4) Execute action a at the current moment and update N in-plant To obtain the state s' at the next moment, calculate the immediate reward r that can be obtained by the action, store (s, a, r, s') in the experience pool, let j = j + 1, if j ≥ J, then according to the PER method, extract J samples from the experience pool, and update the value function Q (s, a) of the agent based on the samples. The calculation formula is: Q'(s,a)=Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)] Among them, Q(s,a) and Q'(s,a) are the value functions of the state (s,a) before and after the update, respectively. During the initial learning, Q(s,a) is the initial value function, Q(s',a') is the value function of the state (s',a'), a and a' are the actions at the current moment and the next moment, respectively, α is the step size factor, r is the immediate reward obtained at the current moment, and γ is the decay rate; (5) If the current action a is a load adjustment action, set s = s' and return to step (3); if the current action a is a scheme output action, go to step (6); (6) If i ≥ I, the training is completed; otherwise, return to step (2).

5. A real-time load distribution system for a hydropower station based on deep reinforcement learning, characterized in that: include: Model building and training module, used to build a load optimization allocation model based on deep reinforcement learning, and train the load optimization allocation model based on deep reinforcement learning in combination with historical operation data; the load optimization allocation model includes state, action, immediate reward and constraint, with all factors that will affect the load allocation decision of the power station as state s; The action is designed based on the idea of ​​load adjustment, and the distribution of a load instruction is converted into a process of gradual adjustment of the load distribution plan, which includes two types of actions: load adjustment and plan output. Each action will have an immediate reward r. The constraints of the load optimization distribution model based on deep reinforcement learning include the constraints of power station operation. The data calculation module is used to receive the grid load instruction, calculate the minimum output and maximum output of each unit in the current startup unit combination, add the minimum output of each unit to obtain the lower boundary of the power station output range, and add the maximum output of each unit to obtain the upper boundary of the power station output range; A judgment execution module is used to judge whether the current combination of startup units can complete the load instruction in combination with the value of the load instruction; if the load instruction can be completed, the current combination of startup units is kept unchanged, and it is judged whether there are other feasible combinations of startup units. If there are no other feasible combinations of startup units, the load optimization distribution model based on deep reinforcement learning is used to generate an optimized load distribution plan between the startup units. If there are other feasible combinations of startup units, the startup unit combination of the power station is updated according to the unit update module; if the load instruction cannot be completed, if the power grid load instruction is greater than the upper boundary of the power station's output range, the unit with the longest current shutdown time is turned on, and the start-stop state and load range of the power station unit are updated until the power grid load instruction is between the lower boundary of the power station's output range and the upper boundary of the power station's output range; if the power grid load instruction is less than the lower boundary of the power station's output range, the unit with the longest current startup time is turned off, and the start-stop state and load range of the power station unit are updated until the power grid load instruction is between the lower boundary of the power station's output range and the upper boundary of the power station's output range; The unit update module is used to generate other feasible combinations of starting units in combination with the rationality constraints of unit start-up and shutdown, and to use the load optimization allocation model based on deep reinforcement learning to rollingly calculate the station load allocation schemes and the objective function values ​​that can be obtained for all feasible combinations of starting units within the calculation period in combination with the load and flow forecast information in the future; Calculate the total objective function value that can be achieved by each scheme within the forecast period, select the optimal combination of startup units based on the total objective function value, and update the startup unit combination of the power station; The method of generating other feasible startup unit combinations by combining the rationality constraints of unit startup and shutdown is as follows: Under the premise of being able to complete the load command, the power station can start or shut down at most one unit. To ensure the rationality of the unit start and stop, the following three constraints for the unit start and stop are set, and their calculation formulas are: The number of units in operation and the number of units in shutdown should be within a reasonable range: n on +n off =n Among them, n on is the number of operating units in the power station, n on >0,n off is the number of shut-down units in the power station, n off >0, n is the number of units in the power station; When the load of the power station increases, the number of units in operation cannot be reduced, and when the load of the power station decreases, the number of units in operation cannot be increased: Among them, n on,t+1 is the number of units in operation at time t+1, n on,t is the number of units in operation at time t, N c is the grid load instruction, is the current total output of the power station; Avoid startup and shutdown operations that are contrary to future load trends: Among them, N on is the minimum output of the power station after the unit is turned on, N c is the minimum output in load forecasting, is the maximum output of the power station after shutting down the unit, is the maximum output in the load forecast; The feasible startup unit combination scheme that can meet the above constraints is retained and used as the remaining feasible startup unit combination scheme except maintaining the status quo; The load optimization distribution model based on deep reinforcement learning is used to roll out the calculation of the station load distribution schemes and the objective function values ​​that can be obtained for all feasible startup unit combination schemes during the calculation period. The specific steps are as follows: S521, read the load and flow prediction sequence of the power plant in the future period, and read the predicted values ​​of the load and flow in the prediction sequence in order; calculate whether the load prediction value is within the load range of the current startup unit combination, if not, update the startup unit combination of the power plant in combination with the calculation method of step S3 until the load instruction can be completed; S522. Use the load optimization distribution model based on deep reinforcement learning to calculate the load distribution plan within the power station under the current combination of start-up and shutdown units and the other feasible combination of start-up and shutdown units, and calculate the objective function value that can be obtained by the load distribution plan in combination with the objective function. The objective function value calculation formula is as follows: Among them, f t is the objective function value that can be obtained by the load distribution scheme within the power station, Q g,t The power generation flow that the power station needs to achieve at the current moment to achieve the water scheduling target, n is the number of units in the power station, Q(N i,t ,H t ) is the function of power generation flow, unit output and water head, N i,t is the output of unit i at time t, H t is the real-time water head of the hydropower station, c is the number of starts and stops, Q p The penalty flow for opening and shutting down; S523, read the next load and flow prediction value in sequence, and return to step S512 until all load and flow prediction data are read and calculated; The objective function values ​​of each scheme obtained by each load distribution scheme within the forecast period are summed up to obtain the total objective function value of each scheme within the forecast period; the startup unit combination scheme with the largest total objective function value is selected as the optimal startup unit combination scheme, and the start and shutdown status of the power station is updated according to the scheme.

6. A device, characterized in that: comprising a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor is used to execute the steps of a real-time load distribution method for a hydropower station based on deep reinforcement learning as described in any one of claims 1 to 4 when running the computer program.

7. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of a real-time load distribution method for a hydropower station based on deep reinforcement learning as described in any one of claims 1 to 4.

Images