Random optimization method for cascade reservoirs of wind-solar hybrid pumped storage power station

Abstract

The application provides a wind-light-hybrid pumped storage power station cascade reservoir random optimization scheduling method, including: based on historical data, a water, wind, light random scene is constructed by solving Markov state transition probability; an ISODATA algorithm based on correlation distance is used for scene reduction, and a long-term scheduling environment of a water-wind-light complementary system is constructed by combination; a long-term random optimization scheduling model of the water-wind-light complementary system, a long-term and short-term coupled optimization scheduling model are constructed respectively; a PER-DQN algorithm is used for solving a long-term random optimization scheduling strategy, and a Q-learning algorithm is used for solving a long-term and short-term coupled optimization scheduling strategy; a real-time scheduling simulation model is constructed to compensate for the prediction deviation of water and electricity, and a hybrid pumped storage power station cascade reservoir scheduling process is updated in a rolling manner. Through the method, the calculation efficiency can be improved, and the economy, reliability and stability of the water-wind-light complementary system can be improved.

Description

Technical Field

[0001] This invention relates to the research fields of new energy and reservoir optimization scheduling, specifically to a stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station. Background Technology

[0002] In recent years, to address global climate change, countries have actively conducted research on renewable energy consumption. Currently, my country is making every effort to promote the high-quality development of renewable energy, and its installed capacity for hydropower, wind power, photovoltaic power, and biomass power has ranked first globally for many consecutive years. However, with the rapid development of renewable and clean energy, the uncertainties of wind and solar power have placed significant pressure on renewable energy consumption and the safe and stable operation of the power system. Pumped storage power stations, as the most technologically mature, economically optimal, and readily available green, low-carbon, and flexible power source for large-scale development, perform well in conjunction with wind power, photovoltaic power, hydropower, nuclear power, and thermal power. Therefore, it is necessary to optimize the scheduling of complementary systems including pumped storage power stations to reduce wind, solar, and hydropower curtailment, while ensuring the reliability and stability of the power system while maximizing the economic efficiency of complementary systems. Summary of the Invention

[0003] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a stochastic optimization scheduling method for cascade reservoirs in wind-solar hybrid pumped storage power stations, which solves the problems of large prediction errors in medium- and long-term runoff, wind power output, and photovoltaic power output, as well as the reduced stability and reliability of the power system caused by the uncertainty and volatility of wind and solar power output.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by this invention is: a stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station, which includes the following steps: Step 1: Based on historical data, conduct randomness analysis on wind power output, photovoltaic power output, and water inflow of cascade reservoirs, and construct long-term scheduling random scenarios for water, wind, and solar power in the water-wind-solar complementary system. Step 2: Propose the ISODATA algorithm based on relevance distance to reduce random scenes of water, wind, and light respectively, and construct the long-term scheduling environment of the water-wind-light complementary system; Step 3: Construct a long-term stochastic optimization scheduling model for the hydro-wind-solar hybrid system with the objective of maximizing system power generation; Step 4: Using the output of each scheduling period obtained from the medium- and long-term optimization scheduling as the boundary constraints of the short-term scheduling model, from the perspectives of the reliability, stability and economy of the complementary system, a medium- and long-term coupled optimization scheduling model for the hydro-wind-solar complementary system is constructed with the objectives of minimizing the mean square error of the remaining load, minimizing the fluctuation of the system output, and minimizing the square value of the total system output deviation. Step 5: Solve the medium- to long-term stochastic optimization scheduling strategy based on the reinforcement learning PER-DQN algorithm; Step 6: Based on the reinforcement learning Q-learning algorithm, solve the short-term optimal scheduling strategy of the medium- and long-term coupled optimal scheduling model, update the medium- and long-term stochastic optimal scheduling strategy, and determine the day-ahead power generation plan; Step 7: Construct a real-time scheduling simulation model to compensate for the prediction deviations of hydropower, wind and solar power, and update the scheduling process of cascade reservoirs including hybrid pumped storage power stations on a rolling basis according to the day-ahead power generation plan, actual wind and solar power output, and runoff data.

[0005] Preferably, in step 1, the method for constructing random scenes of water, wind, and light is as follows: S1.1: Discretize the historical data of random variables at each stage into the same number of discrete values ​​with the maximum and minimum values ​​as the states of each stage of the Markov decision process. For the correlation of random values ​​corresponding to states in different time periods, the Pearson correlation test method is used, as shown in Equation (1): (1) In the formula: This represents the correlation coefficient between the random values ​​corresponding to the states of random variables in adjacent time periods; N The total number of discrete random values ​​of the random variable at each stage. , Let i be the i-th discrete random value in time period t and time period t+1, respectively. , Let be the mean of the i-th discrete random value in time interval t and time interval t+1, respectively. , Let be the mean square error of the i-th discrete random value in time period t and time period t+1, respectively; If two adjacent time periods are strongly correlated, they conform to the Markov property. The discrete random values ​​of adjacent stages can be represented by Markov transition probabilities. Otherwise, the discrete random values ​​of adjacent stages are independent of each other. That is, there is a possibility of transition between each discrete value in the adjacent stage. Each element in the state transition probability matrix is ​​replaced by the reciprocal of the total number of discrete values ​​in each stage. S1.2: Based on historical data, by statistically analyzing the observed values ​​of wind power output, photovoltaic power output, and water inflow of cascade reservoirs during each scheduling period in previous years, the frequency of discrete value transition processes between adjacent stages that conform to Markov characteristics is obtained, and the Markov state transition probability matrix of each stage is solved. The state transition probabilities between each state are combined to form the state transition probability matrix of adjacent stages. S1.3: Based on the state probability matrix of each stage, the Monte Carlo sampling method is used to directly sample each stage to obtain the long series time numbers of water, wind and light and their corresponding state transition probabilities as random scenarios of water, wind and light.

[0006] Preferably, in step 2, the method of scene reduction and construction of long-term scheduling environment in water-wind-solar hybrid system using ISODATA algorithm based on correlation distance is proposed. Since random values ​​in random scenes of water-wind-solar hybrid system are represented by long series of time-series data, the distance function commonly used in clustering algorithms to represent the distance between two points is insufficient to describe the distance between two time-series data. Therefore, when reducing random scenes in water-wind-solar hybrid system, correlation distance is proposed for clustering. The specific process is as follows: S2.1: Input random variable time series data; pre-select Initial cluster centers Expected number of cluster centers Minimum number of samples in each cluster region (If the number is less than this, it is not considered an independent cluster) Standard deviation of the distance distribution of samples within a cluster region The minimum distance between two cluster centers (If the number is less than this, the two clusters need to be merged.) The maximum number of cluster centers that can be merged in one iteration. L and the number of iterations ; S2.2: Calculate the relevance distance from each sample data point to the cluster center, and... N Each sample is assigned to the nearest cluster. ; S2.3: If clustering The number of samples in the sample is less than If the clustering is invalid, then the number of cluster centers is not valid. Decrease by 1 and restart clustering; S2.4: Recalculate all parameters after clustering, including cluster centers. The correlation distance between sample data within a cluster and the cluster center Total correlation distance of all sample data to the corresponding center ; The cluster centers are shown in equation (2): (2) In the formula, For the first j Cluster centers; For the first j The number of samples in each cluster; Represents samples in a cluster; For the first j One cluster; The relevant distances from the sample data within a cluster to the cluster center are shown in Equation (3): (3) In the formula: The relative distance from the sample data to the cluster center; For sequence and covariance; , Sequences and covariance; The total correlation distance of all sample data to the corresponding center is shown in Equation (4): (4) In the formula, The relevant distances from samples in all clusters to their respective cluster centers; The number of cluster centers; For the first j The number of samples in each cluster; For the first j The relevant distances from samples in each cluster to the cluster centers; S2.5: Stop splitting and merging calculations If the current iteration number equals the iteration threshold, then proceed to S2.6, and... Set to 0; like If the number of clusters is too small, proceed to S2.6; like If the number of clusters is too large, proceed to S2.8; like If the current iteration is an even number, proceed to S2.9; otherwise, proceed to S2.7. S2.6: Calculate the standard deviation vector of sample data in the cluster based on the distance values: The vector components are shown in equation (5): (5) In the formula, i Represents the dimension of the sample feature vector; j Represents the cluster number; Representing the j The total number of samples in each cluster; Representing the i The th sample feature vector dimension k One sample; Representing the i The th sample feature vector dimension j There are cluster centers. And we obtain... The maximum value in ; S2.7: If Greater than and satisfy or If any condition is met, then It splits into two clusters, and Increment by 1. Increment the iteration count by 1 and proceed to Step 2.1; S2.8: Calculate the correlation distance between cluster centers ; S2.9: Comparison and The value of will satisfy Sort the values ​​in descending order; S2.10: Merge two adjacent cluster centers after the arrangement, and calculate the new cluster centers according to equation (6). Subtract 1; (6) In the formula, As the new cluster center; For the rearranged number The number of samples in each cluster center; For the rearranged number The number of samples in each cluster center; For the first Cluster centers; For the first Cluster centers; This is the number of cluster centers remaining at this point; S2.11: If the iteration threshold is reached I If the algorithm stops splitting or merging (cluster centers no longer change), the algorithm terminates. Otherwise, the algorithm iterates again. S2.12: Output cluster centers; Cluster centers of time-series data are used as representative random scenarios for water, wind, and solar power. Random scenarios of the water-wind-solar system are obtained by combining these cluster centers. By comparing the Markov state transition probability matrices of each stage, the Markov state transition probabilities of each stage corresponding to the time-series data of the cluster centers are obtained, and the long-term scheduling environment of the water-wind-solar complementary system is constructed.

[0007] Preferably, in step 3, the long-term stochastic optimization scheduling model of the hydro-wind-solar hybrid system aims to maximize the system's power generation. By scheduling on a daily basis, the total monthly power generation of the hydro-wind-solar hybrid system is maximized. The total power generation of the system in each scheduling period is shown in equation (7): (7) In the formula: express t Total power generation of the hydro-wind-solar cascade complementary system during the time period; For reservoirt The duration of the time slot scheduling; It includes a cascade reservoir with a mixed pumped storage power station. t Efforts during a specific time period; express t Total wind power output during the period; express t Total photovoltaic power generation output during the period; N This represents the total number of cascade reservoirs. for t Time period n The power generation flow of the reservoir; For the first n The overall output coefficient of a reservoir at level 1; For the first n Level 1 reservoir t The average hydropower head for the period was the first n Level 1 reservoir and the first n+ Level 1 reservoir t Water level difference over time; for t Operating conditions of time-reversible pump-turbine units For power generation operation, For pumping operation; , They are respectively t The power generation and pumping power consumption of the time-reversible pump-turbine unit are shown in equation (8): (8) In the formula: , They are respectively t Power generation and pumping power consumption of a time-reversible pump-turbine unit; , They are respectively t The power generation and pumping power of a time-reversible pump-turbine generator set; , These are the power generation efficiency and pumping efficiency of the reversible pump-turbine generator set, respectively. for t Average hydropower head over the period; , They are respectively t The flow rate of the time-reversible pump-turbine generator set under both power generation and pumping conditions; For pipeline efficiency; for t Time period scheduling duration; The density of water; It is the acceleration due to gravity; The water balance constraint of the cascade reservoirs containing the hybrid pumped storage power station is shown in equation (9): (9) In the formula: , The first n Level 1 reservoir t Initial and final storage capacity of the time period; , The first n Level 1 reservoir t Inflow and outflow rates of each reservoir during the specified time period; For the first n The water inflow from the reservoir is between the two levels, and That is, the first-level reservoir has no inter-regional water inflow; , They are respectively t The flow rate of the time-reversible pump-turbine generator set under both power generation and pumping conditions; For reservoir t The duration of the time slot scheduling; The cascade water level constraint is shown in equation (10): (10) In the formula: , The first n Level 1 reservoir t Minimum and maximum water level limits at the beginning of the time period; For the first n Level 1 reservoir t Water level at the beginning of the period; The power generation flow constraint is shown in equation (11): (11) In the formula: For the first n Maximum flow rate of the reservoir; For the first n The power generation flow of the reservoir.

[0008] The power generation and pumping power limits of the reversible pump-turbine generator set are shown in equation (12): (12) In the formula: , They are respectively t The power generation and pumping power of a time-reversible pump-turbine generator set; , These are the maximum power generation and maximum pumping power of the reversible pump-turbine generator set, respectively. The limitations on wind and solar power output are shown in equation (13): (13) In the formula: express t Wind power output during certain periods; Indicates photovoltaic power output; This is the maximum output limit for wind power. This is the maximum output limit for photovoltaic power.

[0009] Preferably, in step 4, the long-term and short-term coupled optimization scheduling model of the hydro-wind-solar hybrid system uses the daily power output of the system in the medium- and long-term scheduling strategy as the boundary constraint of the total power output deviation of the short-term scheduling system to solve the short-term scheduling strategy, and feeds the short-term scheduling results back to the medium- and long-term scheduling to update the scheduling process of solving the scheduling strategy in the medium- and long-term scheduling; and the short-term scheduling is carried out at the hourly level to minimize the daily residual load mean square error, the system power output fluctuation, and the total power output deviation square value of the hydro-wind-solar hybrid system. The mean square error of the remaining load is expressed as shown in equation (14): (14) In the formula: The mean squared error of the remaining load; T Total number of time periods; , , These are cascade reservoirs containing mixed pumped storage power stations. t Efforts during the period t Total wind power output during the period t Total photovoltaic power generation output during the period; for t Time-of-day load demand; The power output fluctuation of the system is expressed by the coefficient of variation, as shown in equation (15): (15) In the formula: The coefficient of variation of system output fluctuation; T Total number of time periods; , , These are cascade reservoirs containing mixed pumped storage power stations. t Efforts during the period t Total wind power output during the period t Total photovoltaic power generation output during the period; This represents the average total output of the complementary system. The minimum value of the square of the total output deviation of the system is shown in equation (16): (16) In the formula: Punishment for exertion; The total output on day d is obtained from the medium- and long-term scheduling model; The total output on day d is obtained from the short scheduling model; The three objective functions above are transformed into a single-objective integrated objective function by weighting, as shown in equation (17): (17) In the formula: F It is the short-term scheduling comprehensive objective function; , , The objective functions are respectively , , The weighting coefficients.

[0010] Preferably, in step 5, the method for solving the medium- and long-term stochastic optimization scheduling strategy based on the reinforcement learning PER-DQN algorithm considers the medium- and long-term stochastic optimization scheduling of cascade reservoirs with mixed pumped storage power stations connected to wind and solar power as a multi-stage decision problem, with each scheduling period as a stage, and the optimal decision sequence, i.e. the optimal scheduling strategy, is solved by the reinforcement learning PER-DQN algorithm. Priority Experience Replay uses temporal difference (TD) error as a priority indicator and combines it with the Sumtree algorithm to sample sample priorities. A larger TD error indicates a greater improvement in prediction accuracy, higher learning value, and a higher priority for the sample. The Sumtree algorithm uses a binary tree structure for sample sampling. Each leaf node in the binary tree corresponds to the priority of a sample, while other nodes have no practical significance. The priority of a parent node is the sum of the priorities of its two child nodes, and the priority of the root node is the sum of the priorities of all samples. During data training, the samples are first divided into multiple intervals, the size of which is the priority of the root node divided by the total number of samples in each batch. A number is randomly selected from each interval, and the leaf nodes are searched from top to bottom starting from the root node to determine the final sampled data.

[0011] Therefore, the priority definition based on TD error is as shown in equation (18): (18) In the formula: Indicates the first i Temporal difference error of each sample; No. i Priority of each sample; It is a very small constant to prevent priority from being 0; for t The reward for the +1 period is the action reward value; In the state action Below, the parameters of the main neural network are: The value of the action obtained at that time; In the state action Below, the target network parameters are fixed as follows: The resulting action value is the Q target value. The discount rate describes the impact of future earnings on the present. These are the network parameters of the main neural network; To fix the network parameters of the target network; , , respectively t Time period t The state of the random variable in time interval +1; , They are respectively t Time period t Actions to be taken during the +1 time period; Furthermore, the Q-value and neural network parameter updates of the PER-DQN algorithm are the same as those of the DQN algorithm. By continuously iterating, i.e., the agent trains and updates the neural network parameters and Q-value, the agent's decision-making ability is improved and the optimal scheduling strategy is obtained. As shown in expression (19): (19) In the formula: This refers to timing difference error; , Representing the first k, k+ 1 training session action Below, the parameters of the main neural network are: At that time, the action value obtained is the Q estimate; In the state action Below, the target network parameters are fixed as follows: The resulting action value is the Q target value. The discount rate describes the impact of future earnings on the present. These are the network parameters of the main neural network; To fix the network parameters of the target network; for t +1 period reward is the action reward value; This represents the gradient of the loss function; It is expressed as the learning rate, which determines the extent to which the error is learned; The specific solution steps are as follows: S5.1: Initialize neural network parameters, training iteration limit, and neural network parameter update interval (training iterations); S5.2: Input relevant data on the long-term scheduling environment of the water-wind-solar hybrid system; S5.3: Place time-series data from random scenarios as samples into the experience pool for priority experience replay; S5.4: Take a batch of samples and input them into the neural network to map the Q value; S5.5: The agent interacts with the environment to acquire knowledge and obtain the Q value; S5.6: Update the parameters of the main neural network based on the time difference and gradient descent ideas, reduce the prediction bias of the neural network, update the Q value, and improve the decision-making ability of the agent, see equation (19). S5.7: The interval for updating neural network parameters involves copying the main neural network parameters to the fixed target network parameters after each training iteration. S5.8: Repeat steps S3 to S4 until the agent reaches the final state to complete one training cycle; S5.9: Repeat S5 until the maximum number of training iterations is reached; S5.10: Output the long-term scheduling strategy of the hydro-wind-solar hybrid system.

[0012] Preferably, in step 6, the method for solving the short-term optimal scheduling strategy in a long-term and short-term coupled optimal scheduling model based on the reinforcement learning Q-learning algorithm involves the Q-learning algorithm constructing a temporal difference value to update the current value estimate by calculating the difference between the value at the next time step and the value at the current time step. The Q-value table stores the Q-values ​​of all states at each stage of the multi-stage decision problem during the update process, and always overwrites the old values ​​with better Q-values ​​during the training process, so that the Q-values ​​are close to the optimal action value, thereby improving the agent's decision-making ability and obtaining the optimal scheduling strategy. For deterministic models with small discrete state spaces, the relatively simple Q-learning algorithm can ensure computational accuracy with a small memory footprint, thereby saving computational costs. The formula for updating the Q-value of the Q-learning algorithm is shown in equation (20): (20) In the formula: , Representing the first k, k+ 1 training session Take action below The value of the action obtained afterward is the Q value; No. k Secondary training state action The value of the action obtained is the Q value; The Markov transition probability of the runoff state of the cascade reservoirs in adjacent time periods; for t+1 period reward is the action reward value; , They are respectively t Time period t The state of the random variable in time interval +1; , They are respectively t Time period t Actions to be taken during the +1 time period; Set of random variable states; Indicates timing difference error; The learning rate is used to determine the extent to which the error is learned. The specific solution steps are as follows: S6.1: Initialize the Q-value table, setting the maximum number of training iterations to 1. N The state transition probability between adjacent stages is 1, and the number of days is... i =0; S6.2: Input the first... i Short-term forecasts of random variables such as water, wind, and solar power, and the first in medium- and long-term scheduling strategies. i The sun produces strength; S6.3: Construct a deterministic scheduling environment based on the short-term predicted values ​​of water, wind, and solar random variables, i.e., the transition probability at each stage is 1; S6.4: The agent interacts with the environment to make decisions and acquire knowledge; S6.5: Calculate the Q value based on the Q-learning algorithm Q value update formula (16) according to the knowledge sample, and compare it with the Q value in the Q value table. Then, overwrite the corresponding position of the Q value table with the larger Q value to update it, so as to improve the decision-making ability of the agent. S6.6: Repeat steps S3 to S4 until the agent reaches the final state to complete one training cycle; S6.7: Repeat S5 until the maximum number of training iterations is reached; S6.8: Output the first... i Short-term scheduling strategy for the solar-water-wind-solar hybrid system; S6.9: The first i The final state of the short-term scheduling strategy of the solar-water-wind-solar hybrid system is used as the initial state of the medium- and long-term scheduling. The medium- and long-term stochastic optimization scheduling model of the solar-water-wind-solar hybrid system based on the PER-DQN algorithm is called to calculate and update the medium- and long-term stochastic optimization scheduling strategy. S6.10: Output in the order of the first... i+ Long-term stochastic optimization scheduling strategy for a hydro-wind-solar hybrid system with an initial state on day 1; Repeat the above steps to update the short-term dispatch strategy of the hydro-wind-solar hybrid system, so as to compile and update the day-ahead power generation plan for use in the real-time dispatch simulation model.

[0013] Preferably, in step 7, the real-time scheduling simulation model of the hydro-wind-solar complementary system follows the principle of "water determined by electricity", strictly monitors the operation status of the hydropower station reservoir in real time according to the day-ahead power generation plan, and uses a model that includes cascade reservoirs of hybrid pumped storage power stations to adjust the prediction deviation of hydro-wind-solar power. Furthermore, the cascade reservoirs of hybrid pumped storage power stations are scheduled by updating the hydro-wind-solar power prediction values ​​with the observed values ​​of hydro-wind-solar power in each time period, and the day-ahead scheduling plan is updated with the water level after scheduling as the initial water level at the beginning of the next time period. This process is repeated continuously for rolling updates. The specific steps are as follows: S7.1: Call the long-term and short-term coupled optimization scheduling model of the hydro-wind-solar hybrid system, solve the short-term scheduling strategy through the Q-learning algorithm, and compile the day-ahead power generation plan; S7.2: Based on the day-ahead power generation plan and the measured data of the first dispatch period of cascade reservoir inflow, wind power output, and photovoltaic power output, the deviation between the predicted values ​​and the measured data of cascade reservoir inflow, wind power output, and photovoltaic power output is compensated by cascade reservoirs containing hybrid pumped storage power stations. S7.3: Use the compensated final water level as the initial state of the deterministic medium- and long-term coupled optimization scheduling model to update the day-ahead power generation plan; S7.4: Repeat the above steps to continuously update the cascade reservoir scheduling process, which includes hybrid pumped storage power stations.

[0014] This invention proposes a stochastic optimization scheduling method for cascade reservoirs in wind-solar hybrid pumped storage power stations, taking into account the stochasticity of medium- and long-term scheduling runoff, wind power output, and photovoltaic power output. This method combines medium- and long-term scheduling, short-term scheduling, and real-time scheduling simulation models, and the technical effects are as follows: 1) Based on historical data, this invention solves the Markov transition probability to construct a Markov chain, generating random scenarios corresponding to runoff, wind power output, and photovoltaic output, effectively analyzing and describing the randomness of runoff, wind power output, and photovoltaic output in the long-term scheduling of the water-wind-solar complementary system.

[0015] 2) To address the issue of redundant random scenarios caused by complex randomness, this invention proposes the ISODATA algorithm based on relevance distance to reduce the number of random scenarios for runoff, wind power output, and photovoltaic output, and then combine them to construct a long-term scheduling environment for the hydro-wind-solar hybrid system. This method enables the construction of long-term scheduling random scenarios in the hydro-wind-solar hybrid system to reduce the number of scenarios while effectively describing the random characteristics of the scheduling environment, thereby improving the training speed of reinforcement learning algorithms for long-term random scheduling in the hydro-wind-solar hybrid system.

[0016] 3) To address the issue of inaccurate medium- and long-term forecasts, this invention constructs a medium- and long-term stochastic optimization scheduling model for a hydro-wind-solar hybrid system, considering cascade reservoirs with hybrid pumped storage power stations and taking the maximization of total power generation as the objective. This model can improve the economic efficiency of the hydro-wind-solar hybrid system from a medium- and long-term perspective and obtain monthly scheduling strategies for the system to guide short-term scheduling.

[0017] 4) This invention proposes using the PER-DQN algorithm to solve the long-term stochastic optimization scheduling model of the constructed water-wind-solar complementary system. As an improved reinforcement learning algorithm, the PER-DQN algorithm, within the reinforcement learning framework, is based on Markov decision processes, utilizes deep learning neural networks, and employs priority experience replay techniques. This improves upon the computational limitations of traditional reinforcement learning in dealing with the curse of dimensionality in large discrete spaces, while also accelerating the learning speed. It can more effectively solve scheduling strategies for complex optimization scheduling models with various characteristics such as high dimensionality, stochasticity, non-convexity, multi-stage nature, and discretizability.

[0018] 5) To improve the economy, reliability and stability of the hydro-wind-solar hybrid system, this invention constructs a long-term and short-term coupled optimization scheduling model for the hydro-wind-solar hybrid system, taking the cascade reservoirs of the hybrid pumped storage power station that consider wind and solar access as the object, and aiming at minimizing the standard deviation of the remaining load, minimizing the fluctuation of the system output, and minimizing the square value of the total output deviation of the system.

[0019] This model can effectively smooth out power output and surplus load fluctuations of hydro-wind-solar hybrid systems in the short term, while maximizing the economic benefits of hydro-wind-solar hybrid systems through feedback between the medium and long term and the short term.

[0020] 6) This invention proposes using the Q-learning algorithm to solve the long-term and short-term coupled optimization scheduling model of the constructed water-wind-solar complementary system. Since the short-term predictions of water, wind, and solar power are relatively accurate and the Q-learning algorithm consumes less memory during runtime, the traditional reinforcement learning Q-learning algorithm is used to directly perform deterministic calculations. This saves computational costs while ensuring computational accuracy and speed, and demonstrates good effectiveness in solving the scheduling strategy for the short-term optimization scheduling model of the water-wind-solar complementary system.

[0021] 7) This invention constructs a real-time scheduling simulation model using cascade reservoirs containing hybrid pumped storage power stations to compensate for wind and solar power forecast deviations. This model addresses the discrepancy between actual and predicted power output caused by inaccurate wind and solar power forecasts. Because cascade reservoirs containing hybrid pumped storage power stations have strong regulation capabilities and high scheduling flexibility, compensating for wind and solar power forecast deviations through these reservoirs ensures the smooth implementation of day-ahead plans, makes real-time operation safer and more reliable, and more fully utilizes the benefits of the reservoirs. Attached Figure Description

[0022] Figure 1 Flowchart of the stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station; Figure 2 A flowchart of scene reduction using the ISODATA algorithm based on relevance distance; Figure 3 The flowchart shows the medium- to long-term stochastic optimization scheduling process based on the PER-DQN algorithm. Figure 4 The flowcharts for medium- to long-term and short-term coupled scheduling based on the Q-learning algorithm are shown. Figure 5 This is a flowchart for real-time scheduling simulation. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown, a stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station includes the following steps: Step 1: Based on historical data, conduct randomness analysis on wind power output, photovoltaic power output, and water inflow of cascade reservoirs, and construct long-term scheduling random scenarios for water, wind, and solar power in the water-wind-solar complementary system. In step 1, the method for constructing random scenes of water, wind, and light is as follows: S1.1: Discretize the historical data of random variables at each stage into the same number of discrete values ​​with the maximum and minimum values ​​as the states of each stage of the Markov decision process. For the correlation of random values ​​corresponding to states in different time periods, the Pearson correlation test method is used, as shown in Equation (1): (1) In the formula: This represents the correlation coefficient between the random values ​​corresponding to the states of random variables in adjacent time periods; N The total number of discrete random values ​​of the random variable at each stage. , Let i be the i-th discrete random value in time period t and time period t+1, respectively. , Let be the mean of the i-th discrete random value in time interval t and time interval t+1, respectively. , Let be the mean square error of the i-th discrete random value in time period t and time period t+1, respectively; If two adjacent time periods are strongly correlated, they conform to the Markov property. The discrete random values ​​of adjacent stages can be represented by Markov transition probabilities. Otherwise, the discrete random values ​​of adjacent stages are independent of each other. That is, there is a possibility of transition between each discrete value in the adjacent stage. Each element in the state transition probability matrix is ​​replaced by the reciprocal of the total number of discrete values ​​in each stage. S1.2: Based on historical data, by statistically analyzing the observed values ​​of wind power output, photovoltaic power output, and water inflow of cascade reservoirs during each scheduling period in previous years, the frequency of discrete value transition processes between adjacent stages that conform to Markov characteristics is obtained, and the Markov state transition probability matrix of each stage is solved. The state transition probabilities between each state are combined to form the state transition probability matrix of adjacent stages. S1.3: Based on the state probability matrix of each stage, the Monte Carlo sampling method is used to directly sample each stage to obtain the long series time numbers of water, wind and light and their corresponding state transition probabilities as random scenarios of water, wind and light.

[0025] Step 2: Propose the ISODATA algorithm based on relevance distance to reduce random scenes of water, wind, and light respectively, and construct the long-term scheduling environment of the water-wind-light complementary system; In step 2, a method for scene reduction and construction of a long-term scheduling environment in a hydro-wind-solar hybrid system is proposed using the ISODATA algorithm based on correlation distance. Since random values ​​in the random scenes of a hydro-wind-solar hybrid system are represented by long-series time-series data, the distance function commonly used in clustering algorithms to represent the distance between two points is insufficient to describe the distance between two time-series data points. Therefore, when reducing random scenes in a hydro-wind-solar hybrid system, a method using correlation distance for clustering is proposed. The specific process is as follows: S2.1: Input random variable time series data; pre-select Initial cluster centers Expected number of cluster centers Minimum number of samples in each cluster region (If the number is less than this, it is not considered an independent cluster) Standard deviation of the distance distribution of samples within a cluster region The minimum distance between two cluster centers (If the number is less than this, the two clusters need to be merged.) The maximum number of cluster centers that can be merged in one iteration. L and the number of iterations ; S2.2: Calculate the relevance distance from each sample data point to the cluster center, and... N Each sample is assigned to the nearest cluster. ; S2.3: If clustering The number of samples in the sample is less than If the clustering is invalid, then the number of cluster centers is not valid. Decrease by 1 and restart clustering; S2.4: Recalculate all parameters after clustering, including cluster centers. The correlation distance between sample data within a cluster and the cluster center Total correlation distance of all sample data to the corresponding center ; The cluster centers are shown in equation (2): (2) In the formula, For the first j Cluster centers; For the first j The number of samples in each cluster; Represents samples in a cluster; For the first j One cluster; The relevant distances from the sample data within a cluster to the cluster center are shown in Equation (3): (3) In the formula: The relative distance from the sample data to the cluster center; For sequence and covariance; , Sequences and covariance; The total correlation distance of all sample data to the corresponding center is shown in Equation (4): (4) In the formula, The relevant distances from samples in all clusters to their respective cluster centers; The number of cluster centers; For the first j The number of samples in each cluster; For the first j The relevant distances from samples in each cluster to the cluster centers; S2.5: Stop splitting and merging calculations If the current iteration number equals the iteration threshold, then proceed to S2.6, and... Set to 0; like If the number of clusters is too small, proceed to S2.6; like If the number of clusters is too large, proceed to S2.8; like If the current iteration is an even number, proceed to S2.9; otherwise, proceed to S2.7. S2.6: Calculate the standard deviation vector of sample data in the cluster based on the distance values: The vector components are shown in equation (5): (5) In the formula, i Represents the dimension of the sample feature vector; j Represents the cluster number; Representing the j The total number of samples in each cluster; Representing the i The th sample feature vector dimension k One sample; Representing the i The th sample feature vector dimension j There are cluster centers. And we obtain... The maximum value in ; S2.7: If Greater than and satisfy or If any condition is met, then It splits into two clusters, and Increment by 1. Increment the iteration count by 1 and go to S2.1; S2.8: Calculate the correlation distance between cluster centers ; S2.9: Comparison and The value of will satisfy Sort the values ​​in descending order; S2.10: Merge two adjacent cluster centers after the arrangement, and calculate the new cluster centers according to equation (6). Subtract 1; (6) In the formula, As the new cluster center; For the rearranged number The number of samples in each cluster center; For the rearranged number The number of samples in each cluster center; For the first Cluster centers; For the first Cluster centers; This is the number of cluster centers remaining at this point; S2.11: If the iteration threshold is reached IIf the algorithm stops splitting or merging (cluster centers no longer change), the algorithm terminates. Otherwise, the algorithm iterates again. S2.12: Output cluster centers; Cluster centers of time-series data are used as representative random scenarios for water, wind, and solar power. Random scenarios of the water-wind-solar system are obtained by combining these cluster centers. By comparing the Markov state transition probability matrices of each stage, the Markov state transition probabilities of each stage corresponding to the time-series data of the cluster centers are obtained, and the long-term scheduling environment of the water-wind-solar complementary system is constructed.

[0026] Step 3: Construct a long-term stochastic optimization scheduling model for the hydro-wind-solar hybrid system with the objective of maximizing system power generation; In step 3, the long-term stochastic optimization scheduling model of the hydro-wind-solar hybrid system aims to maximize the system's power generation. By scheduling on a daily basis, the system maximizes its monthly total power generation. The total power generation of the system in each scheduling period is shown in equation (7): (7) In the formula: express t Total power generation of the hydro-wind-solar cascade complementary system during the time period; For reservoir t The duration of the time slot scheduling; It includes a cascade reservoir with a mixed pumped storage power station. t Efforts during a specific time period; express t Total wind power output during the period; express t Total photovoltaic power generation output during the period; N This represents the total number of cascade reservoirs. for t Time period n The power generation flow of the reservoir; For the first n The overall output coefficient of a reservoir at level 1; For the first n Level 1 reservoir t The average hydropower head for the period was the first n Level 1 reservoir and the first n+ Level 1 reservoir t Water level difference over time; for t Operating conditions of time-reversible pump-turbine units For power generation operation, For pumping operation; , They are respectively t The power generation and pumping power consumption of the time-reversible pump-turbine unit are shown in equation (8): (8) In the formula: , They are respectively t Power generation and pumping power consumption of a time-reversible pump-turbine unit; , They are respectively t The power generation and pumping power of a time-reversible pump-turbine generator set; , These are the power generation efficiency and pumping efficiency of the reversible pump-turbine generator set, respectively. for t Average hydropower head over the period; , They are respectively t The flow rate of the time-reversible pump-turbine generator set under both power generation and pumping conditions; For pipeline efficiency; for t Time period scheduling duration; The density of water; It is the acceleration due to gravity; The water balance constraint of the cascade reservoirs containing the hybrid pumped storage power station is shown in equation (9): (9) In the formula: , The first n Level 1 reservoir t Initial and final storage capacity of the time period; , The first n Level 1 reservoir t Inflow and outflow rates of each reservoir during the specified time period; For the first n The water inflow from the reservoir is between the two levels, and That is, the first-level reservoir has no inter-regional water inflow; , They are respectively t The flow rate of the time-reversible pump-turbine generator set under both power generation and pumping conditions; For reservoir t The duration of the time slot scheduling; The cascade water level constraint is shown in equation (10): (10) In the formula: , The first n Level 1 reservoir t Minimum and maximum water level limits at the beginning of the time period; For the firstn Level 1 reservoir t Water level at the beginning of the period; The power generation flow constraint is shown in equation (11): (11) In the formula: For the first n Maximum flow rate of the reservoir; For the first n The power generation flow of the reservoir.

[0027] The power generation and pumping power limits of the reversible pump-turbine generator set are shown in equation (12): (12) In the formula: , They are respectively t The power generation and pumping power of a time-reversible pump-turbine generator set; , These are the maximum power generation and maximum pumping power of the reversible pump-turbine generator set, respectively. The limitations on wind and solar power output are shown in equation (13): (13) In the formula: express t Wind power output during certain periods; Indicates photovoltaic power output; This is the maximum output limit for wind power. This is the maximum output limit for photovoltaic power.

[0028] Step 4: Using the output of each scheduling period obtained from the medium- and long-term optimization scheduling as the boundary constraints of the short-term scheduling model, from the perspectives of the reliability, stability and economy of the complementary system, a medium- and long-term coupled optimization scheduling model for the hydro-wind-solar complementary system is constructed with the objectives of minimizing the mean square error of the remaining load, minimizing the fluctuation of the system output, and minimizing the square value of the total system output deviation. In step 4, the long-term and short-term coupled optimization scheduling model of the hydro-wind-solar hybrid system uses the daily power output of the system in the medium- and long-term scheduling strategy as the boundary constraint of the total power output deviation of the short-term scheduling system to solve the short-term scheduling strategy, and feeds the short-term scheduling results back to the medium- and long-term scheduling to update the scheduling process of solving the scheduling strategy in the medium- and long-term scheduling. Furthermore, the short-term scheduling is carried out at the hourly level to minimize the daily residual load variance, the system power output fluctuation, and the square value of the total power output deviation of the hydro-wind-solar hybrid system. The mean square error of the remaining load is expressed as shown in equation (14): (14) In the formula: The mean squared error of the remaining load; T Total number of time periods; , , These are cascade reservoirs containing mixed pumped storage power stations. t Efforts during the period t Total wind power output during the period t Total photovoltaic power generation output during the period; for t Time-of-day load demand; The power output fluctuation of the system is expressed by the coefficient of variation, as shown in equation (15): (15) In the formula: The coefficient of variation of system output fluctuation; T Total number of time periods; , , These are cascade reservoirs containing mixed pumped storage power stations. t Efforts during the period t Total wind power output during the period t Total photovoltaic power generation output during the period; This represents the average total output of the complementary system. The minimum value of the square of the total output deviation of the system is shown in equation (16): (16) In the formula: Punishment for exertion; The total output on day d is obtained from the medium- and long-term scheduling model; The total output on day d is obtained from the short scheduling model; The three objective functions above are transformed into a single-objective integrated objective function by weighting, as shown in equation (17): (17) In the formula: F It is the short-term scheduling comprehensive objective function; , , The objective functions are respectively , , The weighting coefficients.

[0029] Step 5: Solve the medium- to long-term stochastic optimization scheduling strategy based on the reinforcement learning PER-DQN algorithm; In step 5, the method for solving the medium- and long-term stochastic optimization scheduling strategy based on the reinforcement learning PER-DQN algorithm considers the medium- and long-term stochastic optimization scheduling of cascade reservoirs with mixed pumped storage power stations connected to wind and solar power as a multi-stage decision problem, with each scheduling period as a stage, and the optimal decision sequence, i.e. the optimal scheduling strategy, is solved by the reinforcement learning PER-DQN algorithm. Priority Experience Replay uses temporal difference (TD) error as a priority indicator and combines it with the Sumtree algorithm to sample sample priorities. A larger TD error indicates a greater improvement in prediction accuracy, higher learning value, and a higher priority for the sample. The Sumtree algorithm uses a binary tree structure for sample sampling. Each leaf node in the binary tree corresponds to the priority of a sample, while other nodes have no practical significance. The priority of a parent node is the sum of the priorities of its two child nodes, and the priority of the root node is the sum of the priorities of all samples. During data training, the samples are first divided into multiple intervals, the size of which is the priority of the root node divided by the total number of samples in each batch. A number is randomly selected from each interval, and the leaf nodes are searched from top to bottom starting from the root node to determine the final sampled data.

[0030] Therefore, the priority definition based on TD error is as shown in equation (18): (18) In the formula: Indicates the first i Temporal difference error of each sample; No. i Priority of each sample; It is a very small constant to prevent priority from being 0; for t The reward for the +1 period is the action reward value; In the state action Below, the parameters of the main neural network are: The value of the action obtained at that time; In the state action Below, the target network parameters are fixed as follows: The resulting action value is the Q target value. The discount rate describes the impact of future earnings on the present. These are the network parameters of the main neural network; To fix the network parameters of the target network; , , respectively t Time period t The state of the random variable in time interval +1; , They are respectively t Time period t Actions to be taken during the +1 time period; Furthermore, the Q-value and neural network parameter updates of the PER-DQN algorithm are the same as those of the DQN algorithm. By continuously iterating, i.e., the agent trains and updates the neural network parameters and Q-value, the agent's decision-making ability is improved and the optimal scheduling strategy is obtained. As shown in expression (19): (19) In the formula: This refers to timing difference error; , Representing the first k, k+ 1 training session action Below, the parameters of the main neural network are: At that time, the action value obtained is the Q estimate; In the state action Below, the target network parameters are fixed as follows: The resulting action value is the Q target value. The discount rate describes the impact of future earnings on the present. These are the network parameters of the main neural network; To fix the network parameters of the target network; for t +1 period reward is the action reward value; This represents the gradient of the loss function; It is expressed as the learning rate, which determines the extent to which the error is learned; The specific solution steps are as follows: S5.1: Initialize neural network parameters, training iteration limit, and neural network parameter update interval (training iterations); S5.2: Input relevant data on the long-term scheduling environment of the water-wind-solar hybrid system; S5.3: Place time-series data from random scenarios as samples into the experience pool for priority experience replay; S5.4: Take a batch of samples and input them into the neural network to map the Q value; S5.5: The agent interacts with the environment to acquire knowledge and obtain the Q value; S5.6: Update the parameters of the main neural network based on the time difference and gradient descent ideas, reduce the prediction bias of the neural network, update the Q value, and improve the decision-making ability of the agent, see equation (19). S5.7: The interval for updating neural network parameters involves copying the main neural network parameters to the fixed target network parameters after each training iteration. S5.8: Repeat steps S3 to S4 until the agent reaches the final state to complete one training cycle; S5.9: Repeat S5 until the maximum number of training iterations is reached; S5.10: Output the long-term scheduling strategy of the hydro-wind-solar hybrid system.

[0031] Step 6: Based on the reinforcement learning Q-learning algorithm, solve the short-term optimal scheduling strategy of the medium- and long-term coupled optimal scheduling model, update the medium- and long-term stochastic optimal scheduling strategy, and determine the day-ahead power generation plan; In step 6, a method for solving short-term optimal scheduling strategies in a long-term and short-term coupled optimal scheduling model based on the reinforcement learning Q-learning algorithm is described. The Q-learning algorithm calculates the difference between the value at the next time step and the value at the current time step, constructs a temporal difference value to update the current time step's value estimate, and stores the Q-values ​​in all states at each stage of the multi-stage decision problem during the update process. During training, the old value is always replaced with a better Q-value, ensuring that the Q-value is close to the optimal action value, thereby improving the agent's decision-making ability and obtaining the optimal scheduling strategy. For deterministic models with small discrete state spaces, the relatively simple Q-learning algorithm can guarantee computational accuracy while occupying less memory, thus saving computational costs. The formula for updating the Q-value of the Q-learning algorithm is shown in equation (20): (20) In the formula: , Representing the first k, k+ 1 training session Take action below The value of the action obtained afterward is the Q value; No. k Secondary training state action The value of the action obtained is the Q value; The Markov transition probability of the runoff state of the cascade reservoirs in adjacent time periods; for t +1 period reward is the action reward value; , They are respectively t Time period t The state of the random variable in time interval +1; , They are respectively t Time period t Actions to be taken during the +1 time period; Set of random variable states; Indicates timing difference error; The learning rate is used to determine the extent to which the error is learned. The specific solution steps are as follows: S6.1: Initialize the Q-value table, setting the maximum number of training iterations to 1. N The state transition probability between adjacent stages is 1, and the number of days is... i =0; S6.2: Input the first... i Short-term forecasts of random variables such as water, wind, and solar power, and the first in medium- and long-term scheduling strategies. i The sun produces strength; S6.3: Construct a deterministic scheduling environment based on the short-term predicted values ​​of water, wind, and solar random variables, i.e., the transition probability at each stage is 1; S6.4: The agent interacts with the environment to make decisions and acquire knowledge; S6.5: Calculate the Q value based on the Q-learning algorithm Q value update formula (16) according to the knowledge sample, and compare it with the Q value in the Q value table. Then, overwrite the corresponding position of the Q value table with the larger Q value to update it, so as to improve the decision-making ability of the agent. S6.6: Repeat steps S3 to S4 until the agent reaches the final state to complete one training cycle; S6.7: Repeat S5 until the maximum number of training iterations is reached; S6.8: Output the first... i Short-term scheduling strategy for the solar-water-wind-solar hybrid system; S6.9: The first i The final state of the short-term scheduling strategy of the solar-water-wind-solar hybrid system is used as the initial state of the medium- and long-term scheduling. The medium- and long-term stochastic optimization scheduling model of the solar-water-wind-solar hybrid system based on the PER-DQN algorithm is called to calculate and update the medium- and long-term stochastic optimization scheduling strategy. S6.10: Output in the order of the first... i+ Long-term stochastic optimization scheduling strategy for a hydro-wind-solar hybrid system with an initial state on day 1; Repeat the above steps to update the short-term dispatch strategy of the hydro-wind-solar hybrid system, so as to compile and update the day-ahead power generation plan for use in the real-time dispatch simulation model.

[0032] Step 7: Construct a real-time scheduling simulation model to compensate for the prediction deviations of hydropower, wind and solar power, and update the scheduling process of cascade reservoirs including hybrid pumped storage power stations on a rolling basis according to the day-ahead power generation plan, actual wind and solar power output, and runoff data.

[0033] In step 7, the real-time scheduling simulation model of the hydro-wind-solar complementary system follows the principle of "water determined by electricity". It strictly monitors the operation status of the hydropower station reservoir in real time according to the day-ahead power generation plan. It uses a model that includes cascade reservoirs of hybrid pumped storage power stations to adjust the prediction deviation of hydro-wind-solar power. Furthermore, it updates the prediction values ​​of hydro-wind-solar power with the observed values ​​of hydro-wind-solar power in each time period to schedule the cascade reservoirs of hybrid pumped storage power stations. The water level after scheduling is used as the initial water level at the beginning of the next time period to update the day-ahead scheduling plan. This process is repeated continuously for rolling updates. The specific steps are as follows: S7.1: Call the long-term and short-term coupled optimization scheduling model of the hydro-wind-solar hybrid system, solve the short-term scheduling strategy through the Q-learning algorithm, and compile the day-ahead power generation plan; S7.2: Based on the day-ahead power generation plan and the measured data of the first dispatch period of cascade reservoir inflow, wind power output, and photovoltaic power output, the deviation between the predicted values ​​and the measured data of cascade reservoir inflow, wind power output, and photovoltaic power output is compensated by cascade reservoirs containing hybrid pumped storage power stations. S7.3: Use the compensated final water level as the initial state of the deterministic medium- and long-term coupled optimization scheduling model to update the day-ahead power generation plan; S7.4: Repeat the above steps to continuously update the cascade reservoir scheduling process, which includes hybrid pumped storage power stations.

[0034] In step 1, wind power output refers to the output power of wind turbine generators that convert the kinetic energy of wind into electrical energy. Photovoltaic output refers to the output power of solar cells that directly convert solar energy into electrical energy. Cascade reservoir inflow refers to the inflow runoff from the first-level reservoir of a series of stepped reservoirs constructed from the upstream to the downstream of a river basin, and the inflow between the downstream reservoirs. Inflow runoff and inflow between reservoirs refer to rainfall and snowmelt entering the reservoirs, or water flowing along the surface or underground under gravity during irrigation. Historical data specifically refers to the annual data of wind power output, photovoltaic output, and cascade reservoir inflow. In this invention, a hydro-wind-solar hybrid system specifically refers to a combined power generation system consisting of a cascade hydropower station with annual or multi-year regulation capabilities, including a hybrid pumped storage power station, and wind and solar power. Furthermore, the hybrid pumped storage power station utilizes reversible units in the cascade hydropower station, with adjacent reservoirs serving as upper and lower reservoirs for pumping and storage. Among them, a hybrid pumped storage power station is a hybrid pumped storage power station. Medium- and long-term dispatch refers to a reservoir dispatching method with a monthly dispatch cycle and daily dispatch periods. Water refers to the inflow of water into a cascade hydropower station containing a hybrid pumped storage power station; wind refers to the power output of a wind farm; and solar refers to the power output of a photovoltaic power station. Among them, a hybrid pumped storage power station is a hybrid pumped storage power station.

[0035] Randomness analysis refers to the process of describing the probability of random events occurring in terms of probability. In this invention, it is represented by the Markov transition probability of the scheduling period and is obtained based on historical data. The Markov state transition probability is the probability of a random variable undergoing a state transition during the Markov decision process. During the Markov state transition process, the next state is determined only by the previous state and has no aftereffect, as shown in equation (21): (twenty one) In the formula, t Represents discrete time; for t The set of states of a random variable at any given time; for t The state at any given moment.

[0036] The state transition probabilities are in matrix form, as shown in equation (22): (twenty two) In the formula: M is the total number of discrete values, where , j for t -1 moment j One state, k for t Time of the first k One state; for The state transition probability during a time period, i.e. Time period, state j under conditions The time period is the entry status. k conditional probability .

[0037] and The following two conditions must be met, as shown in equation (23): (twenty three) In the formula: M is the total number of discrete values, where , j for t -1 moment j One state, k for t Time of the first k One state; for The state transition probability during a time period, i.e. Time period, state j under conditions The time period is the entry status. k conditional probability .

[0038] Elements in the state transition probability matrix The solution formula is shown in equation (24): (twenty four) In the formula: M is the total number of discrete values, where , j for t -1 moment j One state, k for t Time of the first k One state; for The state transition probability during a time period, i.e. Time period, state j under conditions The time period is the entry status. k conditional probability ; The state transition frequency represents the runoff state from... Time period State transition Time period The frequency of a state is obtained from historical data statistics.

[0039] In step 1, the random scenario in this invention refers to a long series of time-series data containing the values ​​of random variables for each scheduling period in the scheduling cycle.

[0040] In step 2, the correlation distance is a value that reflects the degree of correlation between two sequences. The smaller the value, the greater the correlation between the two sequences, and the larger the value, the smaller the correlation between the two sequences.

[0041] Since random values ​​in the random scenarios of the hydro-wind-solar hybrid system are represented by long series of time-series data, the distance function commonly used in clustering algorithms to represent the distance between two points is difficult to describe the distance between two time-series data. Therefore, when reducing the random scenarios of the hydro-wind-solar hybrid system, it is proposed to use correlation distance for clustering.

[0042] In step 2, the ISODATA algorithm is a clustering method. Based on the traditional K-means clustering algorithm, ISODATA merges and splits the clustering results, solving the problem of the k-value depending on experience. Specifically, it merges clusters when the number of samples in a particular cluster is too small, the distance between two clusters is too close, or the number of clusters is much larger than the set number of clusters; and it splits clusters when the number of samples in a particular cluster is too large, the variance within a cluster is too high, or the number of clusters is much smaller than the set number of clusters.

[0043] In step 2, scene reduction refers to removing some scenes with low probability and high similarity from all scenes, thereby shortening the calculation time and improving the calculation efficiency.

[0044] In step 2, the scheduling environment refers to the set of each random scenario and its corresponding Markov state transition probability, which can describe the characteristics of random variables in the water-wind-solar hybrid system and is used to solve the scheduling strategy.

[0045] In step 4, the coupled scheduling of medium- and long-term and short-term operations refers to the process of using the daily output in the medium- and long-term scheduling strategy as the boundary constraint for the total output deviation of the short-term scheduling system to solve the short-term scheduling strategy, and then feeding the short-term scheduling results back to the medium- and long-term scheduling to update the scheduling strategy of the medium- and long-term scheduling. Here, short-term scheduling refers to a scheduling method with a daily scheduling cycle and hourly scheduling periods.

[0046] In step 4, the reliability of the complementary system is an indicator of its ability to meet load demand. The complementary system's output should meet load demand, and its output curve should be as consistent as possible with the load curve. Therefore, this invention improves the reliability of the complementary system by setting the objective function that minimizes the mean square error of the remaining load in the short-term scheduling model.

[0047] In step 4, the stability of the complementary system is an indicator for evaluating the output stability of the complementary system. Since wind and solar power output is volatile, to ensure the safe and stable operation of the power system, it is necessary to smooth out the volatility of wind and solar power output through hydropower, making the output of the complementary system more stable. Therefore, this invention improves the stability of the complementary system by setting an objective function that minimizes the volatility of system output in the short-term dispatch model.

[0048] In step 4, the economic efficiency of the complementary system is an indicator for measuring the power generation benefits of the complementary system. In order to maximize the power generation benefits of the complementary system, this invention solves the monthly dispatch strategy in the medium and long-term dispatch model with the goal of maximizing the total power generation, and considers the short-term dispatch output deviation penalty to minimize the deviation between the short-term dispatch output and the daily output that maximizes the total monthly power generation, thereby improving the economic efficiency of the complementary system.

[0049] In step 4, the short-term scheduling model with the objectives of minimizing the mean square error of the remaining load, minimizing the fluctuation of the system output, and minimizing the square value of the total output deviation refers to a model that minimizes the daily mean square error of the hydro-wind-solar hybrid system, minimizes the fluctuation of the system output, and minimizes the square value of the total output deviation through hourly-level scheduling.

[0050] In step 5, reinforcement learning is a method for solving multi-stage decision problems by continuously trying and failing, with the goal of maximizing long-term gains. A multi-stage decision problem refers to a problem where the decision-making process can be divided into several interconnected stages. By making decisions at each stage, a decision sequence that optimizes the outcome is obtained. In this invention, each stage refers to a scheduling period.

[0051] To address the issues of high sampling randomness and low learning rate caused by the uniform sampling of experience pools at the same frequency in traditional DQN algorithms, the PER-DQN algorithm introduces a priority experience replay mechanism with a summation tree structure to improve this deficiency, thereby increasing learning efficiency and accelerating convergence speed. Furthermore, its Q-value update method based on neural networks mitigates the curse of dimensionality in large discrete state spaces, demonstrating good effectiveness for long-term stochastic optimization scheduling in hydro-wind-solar hybrid systems.

[0052] The key elements of reinforcement learning include: agent, environment, action policy, reward signal, and value function. At each stage, the environment is in a certain state. The agent, acting as the decision-maker in the decision-making problem, interacts with the environment in discrete time steps based on a Markov decision process and action policy to acquire knowledge. Interaction with the environment refers to the agent's actions in making decisions. Repeating these steps trains the agent, updates the value estimate, maximizes the value function estimate, and ultimately obtains the optimal policy. The value function is the expected value obtained by the agent from taking a certain action in a given state; its value is also called the Q-value. The action policy defines the agent's behavior, i.e., its decision-making method, at a specific time.

[0053] In the Markov decision-making process of reinforcement learning, the knowledge acquired by the agent through interaction with the environment consists of a quinary set of variables. It represents the state, action, reward, policy, and state transition probability, and its state-action sequence is a Markov chain with a reward process.

[0054] In step 5, the PER-DQN algorithm is an improved version of the Deep Q-network (DQN) algorithm. To address the problem in traditional DQN algorithms where experience transfer is performed by uniformly sampling from the experience pool at the same frequency, resulting in high sampling randomness and a low learning rate, the PER-DQN algorithm introduces a priority experience replay mechanism using a summation tree structure to improve this deficiency.

[0055] The goal of Priority Experience Replay is to measure the importance of samples using priority, increasing the sampling weight of high-priority samples, and thus avoiding the drawback of uniform sampling in the DQN algorithm. Priority Experience Replay uses temporal difference (TD) error as a priority indicator, combined with a sumtree algorithm, to sample samples with priority. A larger TD error indicates a greater improvement in prediction accuracy, higher learning value, and a higher priority for the sample. The Sumtree algorithm uses a binary tree structure for sample sampling. Each leaf node in the binary tree corresponds to the priority of a sample; other nodes have no practical significance. The priority of a parent node is the sum of the priorities of its two child nodes, and the priority of the root node is the sum of the priorities of all samples. During data training, the samples are first divided into multiple intervals, the size of which is the priority of the root node divided by the total number of samples in each batch. A number is randomly selected from each interval, and the leaf nodes are searched from top to bottom starting from the root node to determine the final sampled data.

[0056] Furthermore, the PER-DQN algorithm is identical to the DQN algorithm, possessing a main neural network and a fixed target network with completely identical structures, and their parameter and Q-value update processes are also the same. The neural network refers to a computational structure that simulates the working process of a biological neural network, generally composed of three layers: an input layer, an output layer, and hidden layers. Data is input into the input layer, processed by the hidden layers, and output from the output layer, and the predictive ability is improved through training and parameter adjustments. The main neural network and the fixed target network are used to fit the Q-value corresponding to the current action state (Q-predicted value) and the Q-value corresponding to the next action state (Q-target value), respectively. The difference lies in that the parameters of the main neural network are updated with each Q-value update, while the parameters of the fixed target network are copied from the parameters of the main neural network at regular intervals of training.

[0057] By continuously iterating and training the agent, we can improve the agent's decision-making ability and obtain the optimal scheduling strategy.

[0058] In step 5, the medium-to-long-term stochastic optimization scheduling strategy refers to considering the randomness of random variables in the medium-to-long-term model, treating the scheduling process as a multi-stage decision problem, and obtaining the optimal decision sequence that satisfies the model's objective function by optimizing the scheduling process of the hydro-wind-solar hybrid system. Here, the scheduling process refers to the cascade reservoir water level regulation process, and the scheduling strategy is an array containing the water level values ​​of the cascade reservoirs for each scheduling period.

[0059] In step 6, the Q-learning algorithm is a reinforcement learning algorithm based on the core idea of ​​temporal difference (TD) for updating the value function. The Q-learning algorithm constructs a temporal difference value by calculating the difference between the value at the next time step and the value at the current time step to update the current value estimate. The Q-value table stores the Q-values ​​for all states at each stage of the multi-stage decision problem during the update process, and during training, it consistently replaces old values ​​with better Q-values, making the Q-values ​​closer to the optimal action value, thereby improving the agent's decision-making ability and obtaining the optimal scheduling strategy.

[0060] In step 6, the short-term optimization scheduling strategy refers to treating the short-term optimization scheduling process as a multi-stage decision problem. By optimizing the short-term scheduling process of the hydro-wind-solar hybrid system, the optimal decision sequence that satisfies the model's objective function is obtained, i.e., the optimal scheduling strategy. Here, the scheduling process refers to the cascade reservoir water level regulation process, and the scheduling strategy is an array containing the water level values ​​of the cascade reservoirs for each scheduling period.

[0061] In step 7, hydropower specifically refers to power generation from cascade reservoirs containing hybrid pumped-storage power stations. The hydro-wind-solar forecast deviation refers to the difference between the predicted and observed inflow values ​​of the cascade reservoirs, and the difference between the predicted and observed wind and solar power output values. The real-time scheduling simulation model of the hydro-wind-solar complementary system simulates the principle of "water supply determined by electricity demand," strictly monitors the real-time operation status of the hydropower station reservoirs according to the day-ahead power generation plan, and uses cascade reservoirs containing hybrid pumped-storage power stations to adjust for the hydro-wind-solar forecast deviation.

[0062] In step 7, the rolling update process of the cascade reservoir scheduling including the hybrid pumped storage power station refers to: updating the water, wind and solar forecast values ​​with the water, wind and solar observation values ​​for each time period, scheduling the cascade reservoir including the hybrid pumped storage power station, and using the water level after scheduling as the initial water level at the beginning of the next time period to update the day-ahead scheduling plan, and continuously repeating this update process.

[0063] To demonstrate the advantages of the PER-DQN algorithm compared to the DQN algorithm, both algorithms were applied to a long-term stochastic optimization scheduling model in a hydro-wind-solar complementary system for comparative analysis and verification. Taking a cascade reservoir system consisting of two tandem reservoirs A and B as an example, based on historical data from the first month of the hydrological year, 1000 random scene samples of the wind-solar-hydro-wind-solar system were calculated as the algorithm dataset. The ratio of the training set, validation set, and test set was set to 70%, 20%, and 10%, respectively. The model parameter settings are shown in Table 1, and the solution results are shown in Table 2.

[0064] As shown in Table 2, the PER-DQN algorithm improved the annual power generation of each reservoir compared to the DQN algorithm, with a total increase of 1.87 × 10⁻⁶. 6While the DQN algorithm achieves a lower kW·h convergence rate, the PER-DQN algorithm converges significantly faster, with the DQN algorithm converging around 7500 times, while the PER-DQN algorithm converges around 5000 times. Furthermore, the solution times are 246.35 minutes and 378.64 minutes respectively, representing a 34.93% improvement in speed compared to the DQN algorithm. Therefore, the PER-DQN algorithm offers improvements in both solution quality and speed compared to the DQN algorithm.

[0065] Table 1 DQN Model Parameters

[0066] Table 2 Comparison of Algorithm Optimization Results

Claims

1. A stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station, characterized in that: It includes the following steps: Step 1: Based on historical data, conduct randomness analysis on wind power output, photovoltaic power output, and water inflow of cascade reservoirs, and construct long-term scheduling random scenarios for water, wind, and solar power in the water-wind-solar complementary system. Step 2: Propose the ISODATA algorithm based on relevance distance to reduce random scenes of water, wind, and light respectively, and construct the long-term scheduling environment of the water-wind-light complementary system; Step 3: Construct a long-term stochastic optimization scheduling model for the hydro-wind-solar hybrid system with the objective of maximizing system power generation; Step 4: Using the output of each scheduling period obtained from the medium- and long-term optimization scheduling as the boundary constraints of the short-term scheduling model, from the perspectives of the reliability, stability and economy of the complementary system, a medium- and long-term coupled optimization scheduling model for the hydro-wind-solar complementary system is constructed with the objectives of minimizing the mean square error of the remaining load, minimizing the fluctuation of the system output, and minimizing the square value of the total system output deviation. Step 5: Solve the medium- to long-term stochastic optimization scheduling strategy based on the reinforcement learning PER-DQN algorithm; Step 6: Based on the reinforcement learning Q-learning algorithm, solve the short-term optimal scheduling strategy of the medium- and long-term coupled optimal scheduling model, update the medium- and long-term stochastic optimal scheduling strategy, and determine the day-ahead power generation plan; Step 7: Construct a real-time scheduling simulation model to compensate for the prediction deviations of hydropower, wind and solar power, and update the scheduling process of cascade reservoirs including hybrid pumped storage power stations on a rolling basis according to the day-ahead power generation plan, actual wind and solar power output, and runoff data.

2. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 1, the method for constructing random scenes of water, wind, and light is as follows: S1.1: Discretize the historical data of random variables at each stage into the same number of discrete values ​​with the maximum and minimum values ​​as the states of each stage of the Markov decision process. For the correlation of random values ​​corresponding to states in different time periods, the Pearson correlation test method is used, as shown in Equation (1): （1）； In the formula: This represents the correlation coefficient between the random values ​​corresponding to the states of random variables in adjacent time periods; N The total number of discrete random values ​​of the random variable at each stage. , Let i be the i-th discrete random value in time period t and time period t+1, respectively. , Let be the mean of the i-th discrete random value in time period t and time period t+1, respectively. , are the mean square errors of the i-th discrete random value in time period t and time period t+1, respectively; If two adjacent time periods conform to the Markov property, the discrete random values ​​of adjacent periods are represented by Markov transition probabilities; otherwise, the discrete random values ​​of adjacent periods are independent of each other, and each element in the state transition probability matrix is ​​replaced by the reciprocal of the total number of discrete values ​​in each period. S1.2: Based on historical data, by statistically analyzing the observed values ​​of wind power output, photovoltaic power output, and water inflow of cascade reservoirs during each scheduling period in previous years, the frequency of discrete value transition processes between adjacent stages that conform to Markov characteristics is obtained, and the Markov state transition probability matrix of each stage is solved. The state transition probabilities between each state are combined to form the state transition probability matrix of adjacent stages. S1.3: Based on the state probability matrix of each stage, the Monte Carlo sampling method is used to directly sample each stage to obtain the long series time numbers of water, wind and light and their corresponding state transition probabilities as random scenarios of water, wind and light.

3. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 2, a method for scene reduction and construction of a long-term scheduling environment in a hydro-wind-solar hybrid system is proposed using the ISODATA algorithm based on correlation distance. Since random values ​​in the random scenes of a hydro-wind-solar hybrid system are represented by long-series time-series data, the distance function commonly used in clustering algorithms to represent the distance between two points is insufficient to describe the distance between two time-series data points. Therefore, when reducing random scenes in a hydro-wind-solar hybrid system, correlation distance is proposed for clustering. The specific process is as follows: S2.1: Input random variable time series data; pre-select Initial cluster centers Expected number of cluster centers Minimum number of samples in each cluster region The standard deviation of the distance distribution of samples in a clustering region The minimum distance between two cluster centers The maximum number of cluster centers that can be merged in a single iteration. L and the number of iterations ; If less This number is not considered an independent cluster; if it is less than... At this point, the two clusters need to be merged; S2.2: Calculate the relevance distance from each sample data point to the cluster center, and... N Each sample is assigned to the nearest cluster. ; S2.3: If clustering The number of samples in the sample is less than If the clustering is invalid, then the number of cluster centers is not valid. Decrease by 1 and restart clustering; S2.4: Recalculate all parameters after clustering, including cluster centers. The correlation distance between sample data within a cluster and the cluster center Total correlation distance of all sample data to the corresponding center ; The cluster centers are shown in equation (2): （2）； In the formula, For the first j Cluster centers; For the first j The number of samples in each cluster; Represents samples in a cluster; For the first j One cluster; The relevant distances from the sample data within a cluster to the cluster center are shown in Equation (3): （3）； In the formula: The relative distance from the sample data to the cluster center; For sequence and covariance; , Sequences and covariance; The total correlation distance of all sample data to the corresponding center is shown in Equation (4): （4）； In the formula, The relevant distances from samples in all clusters to their respective cluster centers; The number of cluster centers; For the first j The number of samples in each cluster; For the first j The relevant distances from samples in each cluster to the cluster centers; S2.5: Stop splitting and merging calculations; If the current iteration number equals the iteration threshold, then proceed to S2.6, and... Set to 0; like If the number of clusters is too small, proceed to S2.6; like If the number of clusters is too large, proceed to S2.8; like If the current iteration is an even number, proceed to S2.9; otherwise, proceed to S2.

7. S2.6: Calculate the standard deviation vector of sample data in the cluster based on the distance values: The vector components are shown in equation (5): （5）； In the formula, i Represents the dimension of the sample feature vector; j Represents the cluster number; Representing the j The total number of samples in each cluster; Representing the i The th sample feature vector dimension k One sample; Representing the i The th sample feature vector dimension j Identify cluster centers and obtain... The maximum value in ; S2.7: If Greater than and satisfy or If any condition is met, then It splits into two clusters, and Increment by 1, increment the iteration count by 1, and go to S2.1; S2.8: Calculate the correlation distance between cluster centers ; S2.9: Comparison and The value of will satisfy Sort the values ​​in descending order; S2.10: Merge two adjacent cluster centers after the arrangement, and calculate the new cluster centers according to equation (6). Subtract 1; （6）； In the formula, As the new cluster center; For the rearranged number The number of samples in each cluster center; For the rearranged number The number of samples in each cluster center; For the first Cluster centers; For the first Cluster centers; This is the number of cluster centers remaining at this point; S2.11: If the iteration threshold is reached I If the algorithm stops splitting or merging, meaning the cluster centers no longer change, then the algorithm terminates; otherwise, the algorithm iterates again. S2.12: Output cluster centers; The cluster centers of time-series data are used as representative random scenarios of water, wind, and solar energy. Random scenarios of the water-wind-solar system are obtained by combining them. By comparing the Markov state transition probability matrix of each stage, the Markov state transition probabilities of each stage corresponding to the time-series data of the cluster centers are obtained, and the long-term scheduling environment of the water-wind-solar complementary system is constructed.

4. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 3, the long-term stochastic optimization scheduling model of the hydro-wind-solar hybrid system aims to maximize the system's power generation. By scheduling on a daily basis, the system maximizes its monthly total power generation. The total power generation of the system in each scheduling period is shown in equation (7): （7）； In the formula: express t Total power generation of the hydro-wind-solar cascade complementary system during the time period; For reservoir t The duration of the time slot scheduling; It includes a cascade reservoir with a mixed pumped storage power station. t Efforts during a specific time period; express t Total wind power output during the period; express t Total photovoltaic power generation output during the period; N This represents the total number of cascade reservoirs. for t Time period n The power generation flow of the reservoir; For the first n The overall output coefficient of a reservoir at level 1; For the first n Level 1 reservoir t The average hydropower head for the period was the first n Level 1 reservoir and the first n+ Level 1 reservoir t Water level difference over time; for t Operating conditions of time-reversible pump-turbine units For power generation operation, For pumping operation; , They are respectively t The power generation and pumping power consumption of the time-reversible pump-turbine unit are shown in equation (8): （8）； In the formula: , They are respectively t Power generation and pumping power consumption of a time-reversible pump-turbine unit; , They are respectively t The power generation and pumping power of a time-reversible pump-turbine generator set; , These are the power generation efficiency and pumping efficiency of the reversible pump-turbine generator set, respectively. for t Average hydropower head over the period; , They are respectively t The flow rate of the time-reversible pump-turbine generator set under both power generation and pumping conditions; For pipeline efficiency; for t Time period scheduling duration; The density of water; It is the acceleration due to gravity; The water balance constraint of the cascade reservoirs containing the hybrid pumped storage power station is shown in equation (9): （9）； In the formula: , The first n Level 1 reservoir t Initial and final storage capacity of the time period; , The first n Level 1 reservoir t Inflow and outflow rates of each reservoir during the specified time period; For the first n The water inflow from the reservoir is between the two levels, and That is, the first-level reservoir has no inter-regional water inflow; , They are respectively t The flow rate of the time-reversible pump-turbine generator set under both power generation and pumping conditions; For reservoir t The duration of the time slot scheduling; The cascade water level constraint is shown in equation (10): （10）； In the formula: , The first n Level 1 reservoir t Minimum and maximum water level limits at the beginning of the time period; For the first n Level 1 reservoir t Water level at the beginning of the period; The power generation flow constraint is shown in equation (11): （11）； In the formula: For the first n Maximum flow rate of the reservoir; For the first n Hydropower generation flow of the reservoir; The power generation and pumping power limits of the reversible pump-turbine generator set are shown in equation (12): （12）； In the formula: , They are respectively t The power generation and pumping power of a time-reversible pump-turbine generator set; , These are the maximum power generation and maximum pumping power of the reversible pump-turbine generator set, respectively. The limitations on wind and solar power output are shown in equation (13): （13）； In the formula: express t Wind power output during certain periods; Indicates photovoltaic power output; This is the maximum output limit for wind power. This is the maximum output limit for photovoltaic power.

5. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 4, the long-term and short-term coupled optimization scheduling model of the hydro-wind-solar hybrid system uses the daily power output of the system in the medium- and long-term scheduling strategy as the boundary constraint of the total power output deviation of the short-term scheduling system to solve the short-term scheduling strategy, and feeds the short-term scheduling results back to the medium- and long-term scheduling to update the scheduling process of solving the scheduling strategy in the medium- and long-term scheduling. Furthermore, the short-term scheduling is carried out at the hourly level to minimize the daily residual load variance, the system power output fluctuation, and the square value of the total power output deviation of the hydro-wind-solar hybrid system. The mean square error of the remaining load is expressed as shown in equation (14): （14）； In the formula: The mean squared error of the remaining load; T Total number of time periods; , , These are cascade reservoirs containing mixed pumped storage power stations. t Efforts during the period t Total wind power output during the period t Total photovoltaic power generation output during the period; for t Time-of-day load demand; The power output fluctuation of the system is expressed by the coefficient of variation, as shown in equation (15): （15）； In the formula: The coefficient of variation of system output fluctuation; T Total number of time periods; , , These are cascade reservoirs containing mixed pumped storage power stations. t Efforts during the period t Total wind power output during the period t Total photovoltaic power generation output during the period; This represents the average total output of the complementary system. The minimum value of the square of the total output deviation of the system is shown in equation (16): （16）； In the formula: Punishment for exertion; The total output on day d is obtained from the medium- and long-term scheduling model; The total output on day d is obtained from the short scheduling model; The three objective functions above are transformed into a single-objective integrated objective function by weighting, as shown in equation (17): （17）； In the formula: F It is the short-term scheduling comprehensive objective function; , , The objective functions are respectively , , The weighting coefficients.

6. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 5, the method for solving the medium- and long-term stochastic optimization scheduling strategy based on the reinforcement learning PER-DQN algorithm considers the medium- and long-term stochastic optimization scheduling of cascade reservoirs with mixed pumped storage power stations connected to wind and solar power as a multi-stage decision problem, with each scheduling period as a stage, and the optimal decision sequence, i.e. the optimal scheduling strategy, is solved by the reinforcement learning PER-DQN algorithm. Priority experience replay uses time difference error as a priority indicator and combines it with a summation tree algorithm to sample sample priorities; The summation tree algorithm uses a binary tree structure for sample sampling; each leaf node in the binary tree corresponds to the priority of a sample; the priority of a parent node is the sum of the priorities of its two child nodes, and the priority of the root node is the sum of the priorities of all samples; when training the data, the samples are first divided into multiple intervals, the size of which is the priority of the root node divided by the total number of samples in each batch. A number is randomly selected from each interval, and the leaf nodes are searched from top to bottom starting from the root node to determine the final sampled data. Therefore, the priority definition based on time difference error is as shown in equation (18): （18）； In the formula: Indicates the first i Temporal difference error of each sample; No. i Priority of each sample; It is a very small constant to prevent priority from being 0; for t The reward for the +1 period is the action reward value; In the state action Below, the parameters of the main neural network are The value of the action obtained at that time; In the state action Below, the target network parameters are fixed as follows: The resulting action value is the Q target value. The discount rate describes the impact of future earnings on the present. These are the network parameters of the main neural network; To fix the network parameters of the target network; , , respectively t Time period t The state of the random variable in time interval +1; , They are respectively t Time period t Actions to be taken during the +1 time period; Furthermore, the Q-value and neural network parameter updates of the PER-DQN algorithm are the same as those of the DQN algorithm. By continuously iterating, i.e., the agent trains and updates the neural network parameters and Q-value, the agent's decision-making ability is improved and the optimal scheduling strategy is obtained. As shown in expression (19): （19）； In the formula: This refers to timing difference error; , Representing the first k, k+ 1 training session action Below, the parameters of the main neural network are At that time, the action value obtained is the Q estimate; In the state action Below, the target network parameters are fixed as follows: The resulting action value is the Q target value. The discount rate describes the impact of future earnings on the present. These are the network parameters of the main neural network; To fix the network parameters of the target network; for t +1 period reward is the action reward value; This represents the gradient of the loss function; It is expressed as the learning rate, which determines the extent to which the error is learned; The specific solution steps are as follows: S5.1: Initialize neural network parameters, training iteration limit, and neural network parameter update interval (training iterations); S5.2: Input relevant data on the long-term scheduling environment of the water-wind-solar hybrid system; S5.3: Place time-series data from random scenarios as samples into the experience pool for priority experience replay; S5.4: Take a batch of samples and input them into the neural network to map the Q value; S5.5: The agent interacts with the environment to acquire knowledge and obtain the Q value; S5.6: Update the parameters of the main neural network based on the time difference and gradient descent ideas, reduce the prediction bias of the neural network, update the Q value, and improve the decision-making ability of the agent, see equation (19). S5.7: The interval for updating neural network parameters involves copying the main neural network parameters to the fixed target network parameters after each training iteration. S5.8: Repeat steps S3 to S4 until the agent reaches the final state to complete one training cycle; S5.9: Repeat S5 until the maximum number of training iterations is reached; S5.10: Output the long-term scheduling strategy of the hydro-wind-solar hybrid system.

7. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 6, a method for solving the short-term optimal scheduling strategy in a long-term and short-term coupled optimal scheduling model based on the reinforcement learning Q-learning algorithm is described. The Q-learning algorithm constructs a temporal difference value to update the value estimate at the current moment by calculating the difference between the value at the next moment and the value at the current moment. The Q-value table stores the Q-values ​​of all states at each stage of the multi-stage decision problem during the update process, and always overwrites the old value with a better Q-value during the training process, so that the Q-value is close to the optimal action value, thereby improving the agent's decision-making ability and obtaining the optimal scheduling strategy. The formula for updating the Q-value of the Q-learning algorithm is shown in equation (20): （20）； In the formula: , Representing the first k, k+ 1 training session Take action below The value of the action obtained afterward is the Q value; No. k Secondary training state action The value of the action obtained is the Q value; The Markov transition probability of the runoff state of the cascade reservoirs in adjacent time periods; for t +1 period reward is the action reward value; , They are respectively t Time period t The state of the random variable in time interval +1; , They are respectively t Time period t Actions to be taken during the +1 time period; Set of random variable states; Indicates timing difference error; The learning rate is used to determine the extent to which the error is learned. The specific solution steps are as follows: S6.1: Initialize the Q-value table, setting the maximum number of training iterations to 1. N The state transition probability between adjacent stages is 1, and the number of days is... i =0; S6.2: Input the first... i Short-term forecasts of random variables such as water, wind, and solar power, and the first in medium- and long-term scheduling strategies. i The sun produces strength; S6.3: Construct a deterministic scheduling environment based on the short-term predicted values ​​of water, wind, and solar random variables, i.e., the transition probability at each stage is 1; S6.4: The agent interacts with the environment to make decisions and acquire knowledge; S6.5: Calculate the Q value based on the Q-learning algorithm Q value update formula (20) according to the knowledge sample, and compare it with the Q value in the Q value table. Then, overwrite the corresponding position of the Q value table with the larger Q value to update it, so as to improve the decision-making ability of the agent. S6.6: Repeat steps S3 to S4 until the agent reaches the final state to complete one training cycle; S6.7: Repeat S5 until the maximum number of training iterations is reached; S6.8: Output the first... i Short-term scheduling strategy for the solar-water-wind-solar hybrid system; S6.9: The first i The final state of the short-term scheduling strategy of the solar-water-wind-solar hybrid system is used as the initial state of the medium- and long-term scheduling. The medium- and long-term stochastic optimization scheduling model of the solar-water-wind-solar hybrid system based on the PER-DQN algorithm is called to calculate and update the medium- and long-term stochastic optimization scheduling strategy. S6.10: Output in the order of the first... i+ Long-term stochastic optimization scheduling strategy for a hydro-wind-solar hybrid system with an initial state on day 1; Repeat the above steps to update the short-term dispatch strategy of the hydro-wind-solar hybrid system, so as to compile and update the day-ahead power generation plan for use in the real-time dispatch simulation model.

8. The stochastic optimization scheduling method for cascade reservoirs in a wind-solar hybrid pumped storage power station according to claim 1, characterized in that: In step 7, the real-time scheduling simulation model of the hydro-wind-solar complementary system follows the principle of "water determined by electricity". It strictly monitors the operation status of the hydropower station reservoir in real time according to the day-ahead power generation plan. It uses a model that includes cascade reservoirs of hybrid pumped storage power stations to adjust the prediction deviation of hydro-wind-solar power. Furthermore, it updates the prediction values ​​of hydro-wind-solar power with the observed values ​​of hydro-wind-solar power in each time period to schedule the cascade reservoirs of hybrid pumped storage power stations. The water level after scheduling is used as the initial water level at the beginning of the next time period to update the day-ahead scheduling plan. This process is repeated continuously for rolling updates. The specific steps are as follows: S7.1: Call the long-term and short-term coupled optimization scheduling model of the hydro-wind-solar hybrid system, solve the short-term scheduling strategy through the Q-learning algorithm, and compile the day-ahead power generation plan; S7.2: Based on the day-ahead power generation plan and the measured data of the first dispatch period of cascade reservoir inflow, wind power output, and photovoltaic power output, the deviation between the predicted values ​​and the measured data of cascade reservoir inflow, wind power output, and photovoltaic power output is compensated by cascade reservoirs containing hybrid pumped storage power stations. S7.3: Use the compensated final water level as the initial state of the deterministic medium- and long-term coupled optimization scheduling model to update the day-ahead power generation plan; S7.4: Repeat the above steps to continuously update the cascade reservoir scheduling process, which includes hybrid pumped storage power stations.

Images