Anti-interference multi-stage linkage rapid real-time optimization method and system for cascade hydropower stations

Abstract

The present invention provides an interference-resistant cascade hydropower station multi-level linkage rapid real-time optimization method and system, which relates to the field of cascade hydropower station group scheduling. The method comprises: establishing an initial cascade hydropower station scheduling model, wherein the input of the cascade hydropower station scheduling model comprises the planned loads and the inflow flow of the cascade hydropower station at multiple continuous scheduling time points included in a scheduling cycle; converting the initial cascade hydropower station scheduling model into a model-free Markov decision model with a continuous state space and a continuous action space; training the Markov decision model to obtain the cascade hydropower station scheduling model; obtaining a current scheduling task, wherein the current scheduling task comprises the planned loads and the inflow flow of the cascade hydropower station at multiple continuous future scheduling time points included in a target future scheduling cycle; scheduling the cascade hydropower station group based on the current scheduling task through the cascade hydropower station scheduling model, which has the advantage of improving the effectiveness of the scheduling of the cascade hydropower station group.

Description

Technical Field

[0001] The invention relates to the field of cascade hydropower station group dispatching, and in particular to an interference-resistant multi-level linkage rapid real-time optimization method and system for cascade hydropower stations. Background Art

[0002] In the practice of joint real-time dispatching of cascade hydropower stations, many variable and intertwined influencing factors make real-time dispatching extremely easy to be disturbed, and the dispatching model is difficult to play the role of water resource optimization. The real-time dispatching of cascade hydropower stations not only needs to strictly follow the instructions of the dispatching center, but also needs to deeply consider the ever-changing hydrological conditions in the basin. These factors are often difficult to accurately predict, which adds many challenges to the formulation of dispatching strategies. At the actual operation level, although the flexibility of dispatching instructions is intended to respond to the immediate needs of power grid users, the frequent fluctuations in load demand and the urgent requirements of peak and frequency regulation make it inevitable that there will be deviations between real-time load planning and actual execution. If the deviation is controlled and the power load demand is responded to in a timely manner, it will bring great challenges to the joint dispatching of cascade hydropower stations.

[0003] In addition, the current dispatching system often focuses on the execution of a single task, ignoring the inherent multi-task parallel characteristics of cascade power station dispatching. In the real-time dispatching scenario, the coordination between multiple tasks becomes particularly critical, because there may be direct conflicts of interest between them. The pursuit of water level stability may sacrifice the immediate satisfaction of load demand. On the contrary, excessive satisfaction of load demand may cause unnecessary drastic fluctuations in water level, affecting the safe and stable operation of the power station. The dispatching model needs to solve the problem of balancing multiple objectives and tasks in dispatching.

[0004] In summary, in order to solve the multi-factor interference problem faced by the dispatching of cascade hydropower stations, a more comprehensive and flexible dispatching method and system is urgently needed. The system should be able to deeply integrate predictive information and real-time data, effectively balance the conflicts among multiple tasks, and improve the foresight and adaptability of dispatching decisions, thereby significantly improving the overall efficiency and stability of the dispatching of cascade hydropower stations. Summary of the invention

[0005] The present invention provides an interference-resistant cascade hydropower station multi-level linkage rapid real-time optimization method, comprising: establishing an initial cascade power station scheduling model, wherein the input of the cascade power station scheduling model includes the planned load and the cascade power station inflow at multiple continuous scheduling time points included in a scheduling cycle; converting the initial cascade power station scheduling model into a model-free Markov decision model with a continuous state space and a continuous action space; training the Markov decision model to obtain the cascade power station scheduling model; obtaining a current scheduling task, wherein the current scheduling task includes the planned load and the cascade power station inflow at multiple continuous future scheduling time points included in a target future scheduling cycle; and scheduling a cascade hydropower station group based on the current scheduling task through the cascade power station scheduling model.

[0006] Furthermore, the objective function of the initial cascade power station dispatching model is: , ,in, is the first objective function of the initial cascade power station scheduling model, I is the total number of hydropower stations, T is the total number of scheduling time points included in a scheduling cycle, is an indicative function. When the water level of the i-th hydropower station at the t-th dispatching time point is within the allowable water level difference range, is 1, otherwise is 0, is the water level of the hydropower station at the tth dispatching time point, is the allowable water level difference range, It means that the water level of the hydropower station at the tth dispatching time point is within the allowable water level difference range. is the target water level, is the preset difference, is the second objective function of the initial cascade power station dispatching model, is the output of the i-th hydropower station at the t-th dispatching time point, is the planned load at the tth scheduling time point, is the third objective function of the initial cascade power station dispatching model, Used to indicate whether the water level control target is met. <T hour, When T is equal to 96 and the final water level of the cascade hydropower station group is equal to the target water level, then is 1 if the value is true, otherwise it is 0.

[0007] Furthermore, the constraint set of the initial cascade power station scheduling model includes at least the total water abandonment constraint of each hydropower station at each time point, the total discharge constraint of each hydropower station at each time point, the water level constraint in front of the dam of each hydropower station at each time point, the linear relationship constraint between the discharge flow and the water level behind the dam of each hydropower station at each time point, the relationship constraint between the output of each hydropower station at each time point and the power generation flow and head, the output constraint of each hydropower station at each time point, and the total climbing constraint of each unit of each hydropower station at each time point.

[0008] Furthermore, the comprehensive reward function of the Markov decision model is: , ,in, is the comprehensive reward function of the Markov decision model, and are weights, and are greater than 0, and , is the preset parameter, is the final water level of the cascade hydropower station group, where is the first return value, is the second return value, is the third return value.

[0009] Furthermore, the Bellman equation of the action value function of the Markov decision model is: ,in, For the status Take action Then follow the strategy The expected value of future cumulative rewards that can be obtained, For the status Take action After that, the immediate reward given by the environment, is the expectation operator, indicating the slave state Transition to all possible states Taking the average over the probability distribution p, is the discount factor, is the state value function, which means that in the state Next, follow the strategy The expected value of future cumulative rewards that can be obtained;

[0010] The Bellman equation of the state value function of the Markov decision model is:

[0011] in, For the status Next, follow the strategy The expected value of the future cumulative rewards that can be obtained when For the status Take action Then follow the strategy The expected value of the future cumulative rewards that can be obtained when is the entropy regularization coefficient, For a given state Next select action The probability distribution of .

[0012] Furthermore, the loss function of the state value function of the Markov decision model is: in, is the loss function of the state value function, are the parameters of the state value network, is the expectation operator, indicating the replay buffer from the experience D The state of sampling Take average, For the action network with given parameters θ ,state and actions The output when is the state value of the network in a given state and parameters The output of the following, is the expectation operator, indicating the slave strategy The sampling action is performed average, For the policy network in a given state and parameters The output is in state Take action probability.

[0013] Furthermore, the loss function of the Q function of the Markov decision model is: in, is the loss function of the Q function, θ are the parameters of the action network, is the target Q value, is the expectation operator, which indicates the distribution d The state-action pairs sampled in Take the average.

[0014] Furthermore, the strategy network loss function of the Markov decision model is: in, is the policy network loss function, is the expected operator, indicating that d The state of sampling and from the noise distribution N The noise sampled To take the average, Indicates that the policy network has given parameters ,state ,noise The probability distribution of the action output when To reduce noise and status Mapped to the action space, For the action network with given parameters θ ,state and actions Output when .

[0015] Furthermore, the Markov decision model is trained, including: training the Markov decision model through a hindsight experience replay algorithm and a gradient algorithm.

[0016] The present invention provides an interference-resistant cascade hydropower station multi-level linkage rapid real-time optimization system, comprising: a model building module, used to build an initial cascade power station scheduling model, wherein the input of the initial cascade power station scheduling model includes the planned loads and the cascade power station inflow at multiple consecutive time points included in a time period; a model conversion module, used to convert the initial cascade power station scheduling model into a model-free Markov decision model with a continuous state space and a continuous action space; a model training module, used to train the Markov decision model to obtain a cascade hydropower station group scheduling model; a task acquisition module, used to acquire a current scheduling task, wherein the current scheduling task includes the planned loads and the cascade power station inflow at multiple consecutive future time points included in a target future time period; a joint scheduling module, used to schedule the cascade hydropower station group based on the current scheduling task through the cascade power station scheduling model.

[0017] Compared with the prior art, the anti-interference cascade hydropower station multi-stage linkage rapid real-time optimization method and system provided by the present invention have at least the following beneficial effects:

[0018] The cascade power station dispatching model can handle continuous state space and action space, so that the dispatching strategy can be dynamically adjusted according to the actual reservoir state (such as water level, water storage) and external environment (such as inflow flow, planned load) changes, thereby improving the accuracy and efficiency of dispatching. By training the cascade power station dispatching model, the optimal dispatching strategy under given conditions can be learned, that is, how to reasonably allocate hydropower resources at different time points to meet the planned load demand and maximize the power generation efficiency. By optimizing the dispatching strategy, it can be ensured that the cascade hydropower station group can use water resources as much as possible for power generation while meeting the planned load demand, thereby improving the power generation efficiency. Compared with the existing dispatching method, the cascade power station dispatching model based on Markov decision can more accurately control the operation of hydropower stations, reduce unnecessary waste of water resources, and help achieve the goal of energy conservation and emission reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0020] Figure 1 It is a flow chart of a method for rapid real-time optimization of multi-stage linkage of cascade hydropower stations with anti-interference according to some embodiments of this specification;

[0021] Figure 2 is a schematic diagram of the training process of different reinforcement learning algorithms according to some embodiments of this specification;

[0022] Figure 3 is a schematic diagram of a model training process under different learning rates according to some embodiments of this specification;

[0023] Figure 4 is a schematic diagram of the total load deviation rate distribution according to some embodiments of this specification;

[0024] Figure 5 is a schematic diagram of an average water abandonment Monte Carlo simulation according to some embodiments of this specification;

[0025] Figure 6 is a schematic diagram of a Monte Carlo simulation of gate opening times according to some embodiments of this specification;

[0026] Figure 7 It is a module schematic diagram of an interference-resistant cascade hydropower station multi-stage linkage rapid real-time optimization system according to some embodiments of this specification. DETAILED DESCRIPTION

[0027] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0028] Figure 1 is a flow chart of a method for rapid real-time optimization of multi-stage linkage of cascade hydropower stations with anti-interference according to some embodiments of this specification, such as Figure 1 As shown, the interference-resistant multi-stage linkage rapid real-time optimization method for cascade hydropower stations may include the following steps.

[0029] Step 110: Establish an initial cascade power station scheduling model.

[0030] The input of the cascade power station dispatching model includes the planned load at multiple consecutive dispatching time points included in a dispatching cycle and the inflow flow of the cascade power station.

[0031] Specifically, the cascade power station scheduling model mainly solves the contradiction between water level stability and daily load power generation tasks, while taking into account the end-of-period water level control target. Therefore, three scheduling optimization targets are defined: water level stability target, power generation task target and water level control target.

[0032] In terms of the water level stability target, unlike the existing definition of volatility to evaluate water level stability, this method proposes to use the cumulative time of operation near the dangerous water level as a measure of water level stability. The reason is that in the actual process, within a given safe water level range, the water level is allowed to be adjusted to adapt to the complex and changeable scheduling environment. If the water level is close to a given dangerous boundary, it means that the risk of overflow or dam emptying increases. However, due to the existence of extreme working conditions, the water level is actually allowed to operate at the high water level boundary or the low water level boundary. Therefore, this method proposes to control the operation time close to the upper and lower water level safety boundaries. The longer the operation time in the safe range, the better.

[0033] As a preferred embodiment, the objective function of the initial cascade power station dispatching model is:

[0034] in, is the first objective function of the initial cascade power station scheduling model, I is the total number of hydropower stations, T is the total number of scheduling time points included in a scheduling cycle, is an indicative function. When the water level of the i-th hydropower station at the t-th dispatching time point is within the allowable water level difference range, is 1, otherwise is 0, is the water level of the hydropower station at the tth dispatching time point, is the allowable water level difference range, It means that the water level of the hydropower station at the tth dispatching time point is within the allowable water level difference range. is the target water level, is the preset difference, is the second objective function of the initial cascade power station dispatching model, is the output of the i-th hydropower station at the t-th dispatching time point, is the planned load at the tth scheduling time point, is the third objective function of the initial cascade power station dispatching model, Used to indicate whether the water level control target is met. <T hour, When T is equal to 96 and the final water level of the cascade hydropower station group is equal to the target water level, then is 1 if the value is true, otherwise it is 0.

[0035] Preferably, the constraint set of the initial cascade power station scheduling model includes at least the total water abandonment constraint of each hydropower station at each time point, the total discharge constraint of each hydropower station at each time point, the water level constraint in front of the dam of each hydropower station at each time point, the linear relationship constraint between the discharge flow and the water level behind the dam of each hydropower station at each time point, the relationship constraint between the output of each hydropower station at each time point and the power generation flow and head, the output constraint of each hydropower station at each time point, and the total climbing constraint of each unit of each hydropower station at each time point.

[0036] The constraint condition set is explained below by taking hydropower station 1, hydropower station 2, and hydropower station 3 as examples.

[0037] Since the main source of the inflow of Hydropower Station 2 and Hydropower Station 3 is related to the outflow of Hydropower Station 1, their inflow can be regarded as the total outflow of Hydropower Station 1; and the inflow of Hydropower Station 1 has strong autocorrelation and periodicity. The constraint set of the initial cascade power station scheduling model for scheduling the three hydropower stations is:

[0038]

[0039] Where t is time, with a value of 1 to 96; i is the power station number, with a value of 1, 2, or 3; k is the downstream power station number, with a value of 2 or 3; C1 is the total gate outflow of the hydropower station at each time point, indicating the range of the power generation flow of the i-th hydropower station at the t-th time point. is the minimum power generation flow of the i-th hydropower station, is the power generation flow of the i-th hydropower station at the t-th time point, is the maximum power generation flow of the i-th hydropower station; C2 represents the total outflow constraint of the i-th hydropower station at the t-th time point, represents the gate outflow of the i-th power station at time t, represents the minimum outflow of the i-th hydropower station, represents the maximum outflow of the i-th hydropower station; C3 represents the water level constraint in front of the dam of the i-th hydropower station at the t-th time point, The water level in front of the dam of the i-th hydropower station at time point t, represents the minimum water level constraint of the i-th hydropower station, represents the maximum water level constraint of the i-th hydropower station; C4 represents the outflow range constraint of the i-th hydropower station. represents the power generation flow of the i-th power station at time t-1, represents the gate outflow of the i-th power station at time t-1, represents the maximum allowable fluctuation range of the total outflow of the i-th power station. C5 represents the water balance formula of hydropower station 1; C6 represents the water balance formula of hydropower stations 2 and 3. represents the storage capacity of the kth power station at time t, represents the storage capacity of the kth power station at the t-1th time, represents the outflow from the gate of k-1 power station, represents the power generation flow of the k-1th power station at time t; C7 represents the relationship between the water level behind the power station dam and the outflow flow; represents the water level behind the dam at the tth time of the i-th hydropower station, The relationship between the water level in front of the dam and the outflow of the i-th hydropower station. C8 represents the relationship between the output and power generation flow of the power station i at time t and the water level in front of and behind the dam. represents the water level in front of the dam at the tth moment of the i-th power station, NHQ represents the functional relationship between the output and the front and back of the dam, The above represents the output of the i-th power station at the t-th time; C9 and C10 represent the power station numbers; C11 represents the output constraint of power station i at time t, represents the minimum output of the i-th power station, represents the maximum output of the i-th hydropower station; C12 represents the total climbing constraint of the units of the i-th hydropower station at time t, represents the output of the i-th hydropower station at time t-1, represents the maximum output variation at adjacent moments; C13 represents the daily average gate outflow of power station i that meets the constraint, represents the minimum daily average gate outflow flow of the i-th hydropower station, represents the minimum daily average gate outflow of the i-th hydropower station; C14 represents the value range of time t; C15 represents the maximum value constraint of the gate opening times of the i-th hydropower station on that day; represents the minimum outflow change value when the gate opening of the i-th hydropower station is adjusted, represents the maximum number of gate opening times allowed for the i-th power station on that day, In the actual experiment here, the value is 5.

[0040] Step 120: transform the initial cascade power station dispatching model into a model-free Markov decision model with a continuous state space and a continuous action space.

[0041] The environmental states in the initial cascade power station scheduling model are the water level, reservoir capacity, load, and water level behind the dam of each hydropower station. The state space has Markov properties and is continuous; the decision variables in the initial cascade power station scheduling model are the power generation flow and abandoned water flow of each hydropower station. The decision variables are all continuous. The Markov Decision Process (MDP) is a mathematical model of sequential decision-making. It is used in environments where the system state has Markov properties and obtains greater returns through continuous interaction with the environment. Therefore, the above optimization problem can be transformed into a discrete-time Markov decision model with continuous state space and continuous action space.

[0042] An MDP has four elements:<S,A,Pr,r > , where S represents the state space, A represents the action space, Pr represents the state transition probability, and r represents the reward function.

[0043] Still taking Hydropower Station 1, Hydropower Station 2, and Hydropower Station 3 as examples, the corresponding relationship between the variables of the Markov decision model and the reinforcement learning elements is as follows;

[0044] State Space:

[0045] Action Space:

[0046] Where t is time, is the set of hydropower station state variables at time t, is the water level above the dam of the hydropower station numbered 1 at time t, is the water level above the dam at the hydropower station numbered 2 at time t, is the water level above the dam of the hydropower station numbered 3 at time t, is the power generation load of the hydropower station numbered 1 at time t, is the power generation load of the hydropower station numbered 2 at time t, is the power generation load of the hydropower station numbered 3 at time t, are the total storage capacities of hydropower stations numbered 1, 2, and 3 at time t, is the action set at time t; are the total power generation flow at time t of hydropower stations numbered 1, 2, and 3, respectively. They are the total outflows from the gates of hydropower stations numbered 1, 2, and 3 at time t.

[0047] As a preferred embodiment, the comprehensive reward function of the Markov decision model is:

[0048]

[0049] in, is the comprehensive reward function of the Markov decision model, and are weights, and are greater than 0, and , is the preset parameter, is the final water level of the cascade hydropower station group, where is the first return value, is the second return value, is the third return value.

[0050] The state space transition is affected by the probability distribution of the load plan and the inflow flow forecast, and the state space is a continuous value. The probability of a continuous random variable transferring to a single state is 0. Therefore, it is not suitable for solving the Markov problem under a model. It can be regarded as unknown and solved in the framework of a model-free Markov problem through sampling. , execute action , you will get the next state of the system and instant rewards , reinforcement learning uses neural networks to construct a mapping π from state space to action space:

[0051]

[0052] When the strategy π is adopted, due to the uncertainty of the next action, the cumulative return is a distribution. Given the state s and action a, the action value function represents the expected value of the cumulative reward, that is, the cumulative return function within a period of time k. Therefore, given the current state s, the action value function of the scheduling strategy a generated by strategy π can be expressed as:

[0053]

[0054] in, is the immediate reward, with a discount factor λ ∈(0,1).

[0055] The iterative form of the Q function is called the Bellman equation:

[0056]

[0057] Where d is the distribution function of the state space, For the next action, the optimization goal of the Markov decision model is to find a decision strategy π for the load distribution ratio and water abandonment size of each station so that the expected index of the Q function is maximized:

[0058]

[0059] Step 130: training the Markov decision model to obtain a cascade power station dispatching model.

[0060] Preferably, the Bellman equation of the action value function of the Markov decision model is:

[0061]

[0062] in, For the status Take action Then follow the strategy The expected value of future cumulative rewards that can be obtained, For the status Take action After that, the immediate reward given by the environment, is the expectation operator, indicating the slave state Transition to all possible states The probability distribution p is averaged, is the discount factor, is the state value function, which means that in the state Next, follow the strategy The expected value of future cumulative rewards that can be obtained.

[0063] Preferably, the Bellman equation of the state value function of the Markov decision model is:

[0064]

[0065] in, For the status Next, follow the strategy The expected value of the future cumulative rewards that can be obtained when For the status Take action Then follow the strategy The expected value of the future cumulative rewards that can be obtained when is the entropy regularization coefficient, For a given state Next select action The probability distribution of .

[0066] Preferably, the loss function of the state value function of the Markov decision model is:

[0067]

[0068] in, is the loss function of the state value function, are the parameters of the state value network, is the expectation operator, indicating the replay buffer from the experience D The state of sampling Take average, For the action network with given parameters θ ,state and actions The output when is the state value of the network in a given state and parameters The output of the following, is the expectation operator, indicating the slave strategy Action sampled in Take average, For the policy network in a given state and parameters The output is in state Take action probability.

[0069] Preferably, the loss function of the Q function of the Markov decision model is:

[0070]

[0071] in, is the loss function of the Q function, θ are the parameters of the action network, is the target Q value, is the expectation operator, which indicates the distribution d The state-action pairs sampled in Take the average.

[0072] As a preferred embodiment, the strategy network loss function of the Markov decision model is:

[0073]

[0074] in, is the policy network loss function, is the expected operator, indicating that d The state of sampling and from the noise distribution N The noise sampled To take the average, Indicates that the policy network has given parameters ,state ,noise The probability distribution of the action output when To reduce noise and status Mapped to the action space, For the action network with given parameters θ ,state and actions Output when .

[0075] Preferably, the Markov decision model is trained by a hindsight experience replay algorithm and a gradient algorithm.

[0076] By way of example only, training a Markov decision model may include the following steps:

[0077] S11. Initialize model parameters:

[0078] : Parameters of the state value function V

[0079] : Target network parameters of the state value function V

[0080] : Parameters of the Q function

[0081] : Parameters of the policy network

[0082] S12. Data Collection

[0083] Execute the current policy in the environment (by Definition), collect a series of experience tuples .

[0084] S13, use the experience replay algorithm to enhance the data:

[0085] For each failed experience tuple (i.e., a state that did not reach the goal), the hindsight experience replay algorithm modifies the goal state to be one of the states that were actually reached in that experience, thereby "faking" a successful experience. This generates a series of new experience tuples that modify the goal state based on the original experience.

[0086] S14. Construct loss function:

[0087] S15, gradient descent update parameters:

[0088] For each loss function, calculate its gradient with respect to the corresponding parameters.

[0089] Use gradient descent (or more advanced optimizers like Adam) to update the parameters .

[0090] S16, repeat steps S12-S15

[0091] Continue to collect data in the environment, augment the data using the hindsight replay algorithm, and update the model parameters until a stopping condition is reached (such as reaching a maximum number of iterations, stable performance, or a time limit).

[0092] Step 140, obtaining the current scheduled task.

[0093] The current dispatching task includes the planned load and the inflow flow of the cascade power stations at multiple consecutive future dispatching time points included in the target future dispatching cycle.

[0094] Step 150, dispatching the cascade hydropower station group based on the current dispatching task through the cascade power station dispatching model.

[0095] Still taking the above-mentioned flood season daily dispatching of hydropower station 3, hydropower station 1 and hydropower station 2 as an example, the dispatching tasks of the day are: the initial water level of hydropower station 3 is 840 meters, the water level control target at the end of the period is 840.5, and the daily average water abandonment flow is not higher than 500 cubic meters / second; the initial water level of hydropower station 1 is 657 meters, the water level control target at the end of the period is 658.2, and the daily average water abandonment flow is not higher than 300 cubic meters / second; the initial water level of hydropower station 2 is 621.1 meters, the control target at the end of the period is 620.5, and the daily average water abandonment flow is not higher than 300 cubic meters / second; at the same time, the anti-interference cascade hydropower station multi-level linkage fast real-time optimization method proposed in this specification is used to predict Measure the load and inflow of the day; implement the code based on python3.9, first build an intraday scheduling simulation system, then simulate the real-time intraday scheduling scenario of a certain day, and use the hindsight experience replay algorithm and the gradient algorithm to solve the intraday scheduling problem; in order to evaluate the effectiveness of the hindsight experience replay algorithm combined with the gradient algorithm, other offline reinforcement learning algorithms are introduced for comparison, as shown in Table 1; in the experimental comparison stage, the learning rate is set to 0.0002, the maximum number of rounds is set to 3030 times, and the HER algorithm is set to randomly take 10 states in the future as the target; the experimental results are shown in the figure below:

[0096] Table 1

[0097]

[0098] From the perspective of algorithm convergence speed, the algorithm proposed in this paper converges faster and has the best effect. Figure 2 As shown in the figure, for the dynamic programming problem with continuous decision variables, uncertain external environment and high-dimensional decision variables given in this paper, the algorithm proposed in this paper and HER+TD3 can achieve convergence through training, but HER+DDPG has not trained an effective strategy; the algorithm proposed in this paper introduces the SAC method. Due to the random strategy it adopts, it increases the exploratory nature of the reinforcement learning agent, and has better and faster convergence than deterministic reinforcement learning algorithms. At the same time, from the perspective of the mean cumulative return of the path, this method is relatively stable compared to other algorithms.

[0099] like Figure 3 As shown in the figure, from the perspective of algorithm sensitivity, four groups of learning rate parameters were tested, namely 0.0002, 0.0008, 0.0014, and 0.003. Different learning rates can converge. When the parameter increases, the model error fluctuates greatly. However, although the smaller parameters converge more slowly, the results are relatively stable. Overall, the parameter sensitivity is not large. In practice, the robustness of the model can be further improved by combining multiple groups of parameters.

[0100] Through 300 Monte Carlo simulations, the model set the water level target of Hydropower Station 1 to 840.5, the water level target of Hydropower Station 2 to 658.2, and the water level target of Hydropower Station 3 to 620.5. First, the water level process of 24 hours a day was within the water level safety constraint range of the day, and there was no extreme water level within the day; secondly, from the perspective of the end-of-period dam water level target, due to the uncertainty of load plan and water inflow during the scheduling process, the end-of-period water level basically fluctuated around the set target, among which Hydropower Station 1 exceeded the planned water level by a maximum of 0.15 meters; the maximum error between the end-of-period dam water level of Hydropower Station 2 and the target water level was 0.4 meters, and the maximum error between the end-of-period water level of Hydropower Station 3 and the target water level was 1 meter; the error was mainly caused by the accumulation of errors caused by uncertain information in the decision-making process, but from the absolute value of the error, the end-of-period water level control has reached a relatively ideal state.

[0101] From the perspective of power generation load, the model not only realizes the load distribution among the three power stations, but also avoids the problem that often occurs in the actual process, that is, the phenomenon that a power station is pulled empty and the output is zero. All three power stations achieve normal output.

[0102] like Figure 4 As shown, the total load deviation rate distribution meets the requirements.

[0103] like Figure 5 As shown, from the perspective of the average daily water discharge, the average water discharge of hydropower station 1 is 400 cubic meters per second, the average water discharge of hydropower station 2 is 280 cubic meters per second, and the average water discharge of hydropower station 3 is 245 cubic meters per second; the model constraints are met.

[0104] This experiment requires that the number of gate opening and closing times per day should not exceed 5 times. Figure 6 It can be seen that after 300 simulations, all hydropower stations meet the requirements.

[0105] Figure 7 It is a module diagram of an interference-resistant cascade hydropower station multi-stage linkage rapid real-time optimization system according to some embodiments of the present specification. The interference-resistant cascade hydropower station multi-stage linkage rapid real-time optimization system may include a model building module, a model conversion module, a model training module, a task acquisition module and a joint scheduling module.

[0106] The model building module can be used to build an initial cascade power station scheduling model, wherein the input of the initial cascade power station scheduling model includes the planned loads at multiple consecutive time points included in a time period and the inflow flow of the cascade power station.

[0107] The model conversion module can be used to convert the initial cascade power station dispatching model into a model-free Markov decision model with a continuous state space and a continuous action space.

[0108] The model training module can be used to train the Markov decision model to obtain a cascade hydropower station group dispatching model;

[0109] The task acquisition module can be used to acquire the current scheduling task, wherein the current scheduling task includes the planned load and the inflow flow of the cascade power station at multiple consecutive future time points included in the target future time period.

[0110] The joint dispatch module can be used to dispatch a cascade hydropower station group based on the current dispatch task through the cascade power station dispatch model.

[0111] The anti-interference cascade hydropower station multi-stage linkage rapid real-time optimization system can be used to execute the anti-interference cascade hydropower station multi-stage linkage rapid real-time optimization method. For more descriptions of the anti-interference cascade hydropower station multi-stage linkage rapid real-time optimization system, please refer to the relevant descriptions of the anti-interference cascade hydropower station multi-stage linkage rapid real-time optimization method, which will not be repeated here.

[0112] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An anti-interference fast real-time optimization method for multi-level linkage of cascade hydropower stations, characterized in that: include: Establish an initial cascade power station scheduling model, wherein the input of the initial cascade power station scheduling model includes the planned load and the inflow flow of the cascade power station at multiple consecutive scheduling time points included in a scheduling cycle, and the cascade power station scheduling model has three scheduling optimization objectives: a water level stabilization objective, a power generation task objective and a water level control objective, and the constraint set of the initial cascade power station scheduling model includes at least the total water abandonment constraint of each hydropower station at each time point, the total outflow constraint of each hydropower station at each time point, the water level constraint in front of the dam of each hydropower station at each time point, the linear relationship constraint between the outflow flow of each hydropower station and the water level behind the dam at each time point, the relationship constraint between the output of each hydropower station at each time point and the power generation flow and head, the output constraint of each hydropower station at each time point, and the total climbing constraint of each unit of each hydropower station at each time point; The initial cascade power station scheduling model is converted into a model-free Markov decision model with a continuous state space and a continuous action space, wherein the environmental state in the initial cascade power station scheduling model is the water level, reservoir capacity, load, and water level behind the dam of each hydropower station, and the decision variables in the initial cascade power station scheduling model are the power generation flow and abandoned water flow of each hydropower station; The Markov decision model is trained to obtain a cascade power station dispatching model. Specifically, the Markov decision model is trained by a hindsight experience replay algorithm and a gradient algorithm; Acquire a current scheduling task, wherein the current scheduling task includes a planned load and an inflow flow of a cascade power station at a plurality of consecutive future scheduling time points included in a target future scheduling period; The cascade hydropower station group is dispatched through the cascade power station dispatching model based on the current dispatching task.

2. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 1 is characterized in that: The objective function of the initial cascade power station dispatching model is: in, is the first objective function of the initial cascade power station dispatching model, is the total number of hydropower stations, is the total number of scheduling time points included in a scheduling cycle, is an indicative function. When the water level of the i-th hydropower station at the t-th dispatching time point is within the allowable water level difference range, is 1, otherwise is 0, is the water level of the hydropower station at the tth dispatching time point, is the allowable water level difference range, It means that the water level of the hydropower station at the tth dispatching time point is within the allowable water level difference range. is the target water level, is the preset difference, is the second objective function of the initial cascade power station dispatching model, is the output of the i-th hydropower station at the t-th dispatching time point, is the planned load at the tth scheduling time point, is the third objective function of the initial cascade power station dispatching model, Used to indicate whether the water level control target is met. When t< hour, is 0, when is equal to 96, and the final water level of the cascade hydropower station group is equal to the target water level, then is 1 if the value is true, otherwise it is 0.

3. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 2 is characterized in that: The optimization goal of the Markov decision model is to find a decision strategy π for the load distribution ratio and water abandonment size of each station, so that the expected index of the Q function is maximized: in, is the optimization objective of the Markov decision model, For the strategy π Next, from the state Take Action After that, the expected cumulative reward.

4. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 3 is characterized in that: The comprehensive reward function of the Markov decision model is: in, is the comprehensive reward function of the Markov decision model, and are weights, and are greater than 0, and , is the preset parameter, is the final water level of the cascade hydropower station group, where is the first return value, is the second return value, is the third return value.

5. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 4 is characterized in that: The Bellman equation of the action value function of the Markov decision model is: in, For the status Take action Then follow the strategy The expected value of future cumulative rewards that can be obtained, For the status Take action After that, the immediate reward given by the environment, is the expectation operator, indicating the slave state Transition to all possible states The probability distribution of Take the average of the above, is the discount factor, is the state value function, which means that in the state Next, follow the strategy The expected value of future cumulative rewards that can be obtained; The Bellman equation of the state value function of the Markov decision model is: in, For the status Next, follow the strategy The expected value of the future cumulative rewards that can be obtained when For the status Take action Then follow the strategy The expected value of the future cumulative rewards that can be obtained when is the entropy regularization coefficient, For a given state Next select action The probability distribution of For action According to the strategy Calculate the expected distribution of .

6. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 5 is characterized in that: The loss function of the state value function of the Markov decision model is: in, is the loss function of the state value function, are the parameters of the state value network, is the expectation operator, indicating the replay buffer from the experience D The state of sampling Take average, For the action network with given parameters θ ,state and actions The output when is the state value of the network in a given state and parameters The output of the following, is the expectation operator, indicating the slave strategy Action sampled in Take average, For the policy network in a given state and parameters The output is in state Take action probability.

7. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 6 is characterized in that: The loss function of the Q function of the Markov decision model is: in, is the loss function of the Q function, θ are the parameters of the action network, is the target Q value, is the expectation operator, which indicates the distribution d The state-action pairs sampled in Take the average.

8. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 7 is characterized in that: The policy network loss function of the Markov decision model is: in, is the policy network loss function, is the expected operator, indicating that d The state of sampling and from the noise distribution N The noise sampled To take the average, Indicates that the policy network has given parameters ,state ,noise The probability distribution of the action output when To reduce noise and status Mapped to the action space, For the action network with given parameters θ ,state and actions Output when .

9. The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to claim 8 is characterized in that: The Markov decision model is trained through the hindsight experience replay algorithm and the gradient algorithm, including: S11, initializing model parameters, including parameters of the state value function V, parameters of the target network of the state value function V, parameters of the Q function, and parameters of the policy network; S12, data collection, execute the current strategy in the environment, collect a series of experience tuples ; S13, using the hindsight experience replay algorithm to enhance the data, for each experience tuple of the state that did not reach the target, the hindsight experience replay algorithm will modify the target state to one of the states actually reached in the experience, generate a series of new experience tuples, and modify the target state based on the original experience; S14. Construct loss function: S15, gradient descent updates parameters. For each loss function, calculate its gradient with respect to the corresponding parameter, and use gradient descent to update the model parameters. S16. Repeat steps S12-S15, continue to collect data in the environment, use the hindsight experience replay algorithm to enhance the data, and update the model parameters until the stopping condition is reached. 10.Anti-interference cascade hydropower station multi-level linkage rapid real-time optimization system, characterized by: The anti-interference multi-stage linkage rapid real-time optimization method for cascade hydropower stations according to any one of claims 1 to 9 comprises: A model building module, used to build an initial cascade power station dispatching model, wherein the input of the initial cascade power station dispatching model includes a planned load at a plurality of consecutive time points included in a time period and an inflow flow of the cascade power station; A model conversion module, used for converting the initial cascade power station dispatching model into a model-free Markov decision model with a continuous state space and a continuous action space; A model training module, used to train the Markov decision model to obtain a cascade hydropower station group scheduling model; A task acquisition module, used to acquire a current scheduling task, wherein the current scheduling task includes a planned load and an inflow flow of a cascade power station at a plurality of consecutive future time points included in a target future time period; The joint dispatching module is used to dispatch the cascade hydropower station group based on the current dispatching task through the cascade hydropower station dispatching model.

Images