Real-time scheduling method for wind-light-water complementary system based on multi-task deep reinforcement learning

Abstract

The invention discloses a wind-light-water complementary system real-time scheduling method based on multi-task deep reinforcement learning. A t-SNE algorithm is adopted to carry out dimension reduction on operation data of a wind-light-water complementary system; clustering the low-dimensional feature data by adopting a DBSCAN algorithm to obtain operation scene data sets of different scene categories; constructing a scene identification model based on a Stacking ensemble learning algorithm; training a scene identification model by adopting the obtained scene data set; taking the minimum wind power and photovoltaic output reduction amount, the minimum power shortage amount and the maximum energy storage amount of the reservoir at the end of the scheduling period as comprehensive optimization targets, and constructing an optimal scheduling model of the wind-light-water complementary system; a wind-light-water complementary system real-time scheduling model based on a multi-task SAC algorithm is constructed and trained; and performing real-time scheduling on the wind-light-water complementary system based on the trained scene identification model and the real-time scheduling model of the wind-light-water complementary system. The method can quickly make decisions, is high in adaptability to wind-light uncertainty, and improves the reservoir energy storage of the wind-light-water complementary system.

Description

Technical Field

[0001] The present invention relates to a wind-solar-water complementary system, and in particular to a real-time scheduling method for a wind-solar-water complementary system based on multi-task deep reinforcement learning. Background Art

[0002] In order to cope with global warming and pursue sustainable development, renewable energy such as wind and solar energy has received widespread attention. However, large-scale wind power and photovoltaic power generation will affect the safe and stable operation of the power system. The development of wind, solar and water multi-energy complementary power generation system provides an effective way to utilize renewable energy. By utilizing the natural complementary characteristics of multiple energy sources, energy utilization efficiency and power supply reliability can be improved.

[0003] At present, in the research on wind-solar-water complementary system, the robust optimization method and the random optimization method are mainly used to deal with uncertainty. However, these two methods have certain defects. The robust optimization method usually considers the worst uncertainty situation and will give an overly conservative scheduling plan; the random optimization method considers a large number of scenarios and takes a long time to calculate. Secondly, for the solution of the optimization scheduling problem of the wind-solar-water complementary system, mathematical programming methods and intelligent optimization methods are mainly used, but the mathematical programming method ignores some nonlinear characteristics and cannot guarantee accuracy; the intelligent optimization method is prone to fall into local optimality and has a slow solution speed.

[0004] In recent years, deep reinforcement learning has achieved certain results in the fields of power system emergency control and integrated energy dispatching, and can effectively handle uncertainty and make quick decisions. However, when deep reinforcement learning is used to solve the real-time dispatch of wind, solar and hydropower complementary systems, the traditional dispatching method based on single-task deep reinforcement learning makes it difficult for deep reinforcement learning agents to find the optimal strategy for all scenarios, and cannot guarantee the optimality of dispatching decisions. Summary of the invention

[0005] Purpose of the invention: The present invention provides a real-time scheduling method for a wind-solar-water complementary system based on multi-task deep reinforcement learning, which improves the real-time decision-making ability of the wind-solar-water complementary system and increases the energy storage of the reservoir.

[0006] Invention content: The real-time scheduling method of the wind-solar-water complementary system based on multi-task deep reinforcement learning described in the present invention includes the following steps:

[0007] () The t-SNE algorithm is used to reduce the dimension of the wind-solar-water complementary system operation data to obtain low-dimensional feature data;

[0008] (2) The DBSCAN algorithm is used to cluster the low-dimensional feature data to obtain the wind-solar-water hybrid system operation scenario data sets of different scenario categories;

[0009] (3) Building a scene recognition model based on the Stacking ensemble learning algorithm;

[0010] (4) using the scene data set obtained in step (2) to train a scene recognition model;

[0011] (5) Taking the minimum reduction in wind and photovoltaic power output, the minimum power shortage, and the maximum reservoir energy storage at the end of the dispatch period as the comprehensive optimization goals, an optimal dispatch model for the wind-solar-water complementary system is constructed;

[0012] (6) Construct a real-time scheduling model for wind-solar-water complementary system based on multi-task SAC algorithm;

[0013] (7) Training the real-time dispatch model of wind, solar and water complementary system;

[0014] (8) Based on the trained scene recognition model and the real-time scheduling model of the wind-solar-water complementary system, the wind-solar-water complementary system is scheduled in real time.

[0015] Furthermore, the operation data of the wind-solar-water complementary system in step (1) includes wind power, photovoltaic power and day-ahead power generation plan.

[0016] Furthermore, the implementation process of step (1) is as follows:

[0017] The operation data of the wind-solar-water complementary system is input into the high-dimensional space of the t-SNE algorithm, and the Gaussian distribution is used to calculate the similarity of the data in the high-dimensional space; the data in the low-dimensional space is initialized; the t-distribution with a degree of freedom of 1 is used to calculate the similarity of the data in the low-dimensional space; the position of the data in the low-dimensional space is optimized by minimizing the Kullback-Leibler divergence; and the low-dimensional feature data in the low-dimensional space is output.

[0018] Furthermore, the implementation process of step (2) is as follows:

[0019] (21) Determine the DBSCAN neighborhood radius ε and density threshold MinPts;

[0020] (22) Find all core objects;

[0021] (23) Extract a core object p without any category from all core objects, find all objects that are density-reachable from object p, and form a new cluster;

[0022] (24) Determine the final clustering clusters based on density connectivity;

[0023] (25) Repeat steps (23) and (24) until all core objects are included in the collection;

[0024] (26) Output all clustering clusters and scene categories.

[0025] Furthermore, the scene recognition model described in step (3) includes a two-layer recognition model, which is as follows:

[0026] The first-level recognition model includes four classifiers: support vector classifier, gradient boosting classifier, multi-layer perceptron classifier and logistic regression classifier;

[0027] The second-layer recognition model includes a classifier, which is a support vector classifier.

[0028] Furthermore, the implementation process of step (4) is as follows:

[0029] The scene data set is divided into a training set and a test set; the four classifiers of the first-layer recognition model are trained using the k-fold cross-validation method; the classification results of the first-layer recognition model are used as input to train the second-layer recognition model; and the trained scene recognition model is saved.

[0030] Furthermore, the objective function of the wind-solar-water hybrid system optimization scheduling model described in step (5) is:

[0031]

[0032] Where c is the penalty coefficient; T is the total dispatch period; M is the total number of hydropower stations; P t rc is the reduction in wind power and photovoltaic power output during period t; P t ps is the power shortage during period t; V m,end V is the water storage capacity of the m hydropower station at the end of the dispatch; m,ini is the initial water storage capacity of the m hydropower station; E m is the set of downstream hydropower stations of hydropower station m; r i is the hydropower conversion rate of hydropower station i.

[0033] Furthermore, the constraints of the wind-solar-water hybrid system optimization scheduling model described in step (5) include:

[0034] Hydropower station output constraints:

[0035]

[0036] Hydropower station power flow constraints:

[0037] Q m,min ≤Q m,t ≤Q m,max

[0038] Constraints on outflow from hydropower stations:

[0039]

[0040] Water balance constraints:

[0041]

[0042] Reservoir water storage constraints:

[0043]

[0044] System output constraints:

[0045]

[0046] In the formula, is the output of hydropower station m during period t; k m is the power generation coefficient of the m hydropower station; H m,t is the generating head of the m hydropower station in period t; Q m,t is the power generation flow of hydropower station m in period t; are the water level in front of the reservoir dam and the tailwater level of the hydropower station m at time t respectively; They are the upper and lower limits of the output of hydropower station m during period t; is the maximum climbing limit of the m hydropower station in a single period; Q m,min , Q m,max are the lower and upper limits of the power generation flow of the m hydropower station respectively; is the outflow of hydropower station m in period t; are the lower and upper limits of the outflow of the hydropower station m; S m,t V is the water discharge of hydropower station m during period t; m,t is the reservoir capacity of hydropower station m at time period t; is the inflow of hydropower station m during period t; U m is the collection of hydropower stations directly upstream of m hydropower station; t k is the water flow delay time from the upstream hydropower station to the downstream hydropower station; q m,t V is the natural water inflow of hydropower station m during period t; m,min 、V m,max are the lower and upper limits of the water storage capacity allowed for the m hydropower station; ΔV m represents the maximum change of water storage capacity of hydropower station m during the dispatching period; P t wind is the wind power output during period t; P t solar is the photovoltaic output during period t; P t plan Plan the total output of the complementary system during period t.

[0047] Furthermore, the implementation process of step (6) is as follows:

[0048] The objective function and constraints of the wind-solar-water hybrid system optimization scheduling model are converted into a Markov decision process, including states, actions and rewards; the state s at time t t Including wind power output, photovoltaic output, day-ahead power generation plan, natural water inflow of cascade hydropower, water volume of cascade hydropower reservoirs and dispatching period; state a at time t t Including the power generation flow of cascade hydropower stations and the abandoned water flow of cascade hydropower stations; the reward r at time t t It includes the objective function and the penalty term for not meeting the constraints, namely:

[0049] r t =-F t -Q t

[0050] In the formula, F t is the objective function value in period t; Q t is the penalty term for not meeting the constraint condition during time period t;

[0051] Construct multiple subtask agents. The number of subtask agents is the same as the number of scene categories. Each subtask agent uses 5 neural networks. The construction of the j-th subtask agent is as follows:

[0052] Build an Actor network with the following network parameters: Construct two Critic networks with the following network parameters: Construct two target Critic networks with the following network parameters:

[0053] Furthermore, the implementation process of step (7) is as follows:

[0054] (71) Set the total number of training iterations N, determine the scheduling cycle length T, and determine the number of subtask agents W;

[0055] (72) Initialize the parameters of the Actor network, two Critic networks, and two target Critic networks, use the full-scenario pre-trained network parameters as the initial network parameters of each subtask learner, and set the current subtask agent j to 1;

[0056] (73) Determine whether j is greater than W. If j is greater than W, end the training. Otherwise, initialize the experience replay pool, set j = j + 1, and set the current number of iterations n to 1;

[0057] (74) Determine whether n is greater than N. If n is greater than N, return to step (73), otherwise initialize state s t , set the current scheduling period t = 1, and let n = n + 1;

[0058] (75) Determine whether t is greater than T. If t is greater than T, return to step (74). Otherwise, set state s t Input to the Actor network and get action a t ;

[0059] (76) Action a t Interact with the environment in scene j and obtain reward r t and the next state s t+1 ;

[0060] (77) The sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool;

[0061] (78) Update the network parameters of the Actor network Update the network parameters of the two Critic networks Update the network parameters of the two target critic networks Update the temperature coefficient, set t=t+1, and return to step (75).

[0062] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0063] The operation data of the wind-solar-water complementary system is high-dimensional data and cannot be directly and effectively clustered. Before clustering, the t-SNE algorithm is used to reduce the dimension of the operation data of the wind-solar-water complementary system to provide high-quality low-dimensional feature data for subsequent operation scene clustering; the DBSCAN algorithm is used to cluster the low-dimensional feature data to obtain the wind-solar-water complementary system operation scene data sets of different scene categories, and clusters of any shape can be found in the sample space to ensure the reliability of scene division; the present invention constructs a scene recognition model based on the Stacking ensemble learning algorithm, and before using the trained multi-task deep learning scheduling model for real-time decision-making, it quickly and accurately provides the scene category to ensure the accurate calling of the corresponding subtask intelligent agent and the accuracy of the identification result; compared with single-task deep reinforcement learning, model predictive control, and particle swarm methods, the present invention can better cope with uncertain environments and improve the reservoir energy storage of the wind-solar-water complementary system; by training the deep reinforcement learning intelligent agent through historical data, the scheduling result can be quickly given during online decision-making, thereby improving the real-time decision-making ability of the wind-solar-water complementary system. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flow chart of the present invention;

[0065] Figure 2 This is the clustering result diagram of the DBSCAN algorithm;

[0066] Figure 3 It is the reward curve during the agent training process;

[0067] Figure 4 This is the output process diagram of the wind-solar-water complementary system under different scenarios;

[0068] Figure 5 It is a diagram of reservoir water level changes under different scenarios. DETAILED DESCRIPTION

[0069] The technical solution of the invention is described in detail below with reference to the accompanying drawings.

[0070] like Figure 1 As shown, a real-time scheduling method for a wind-solar-water complementary system based on multi-task deep reinforcement learning includes the following steps:

[0071] Step 1: Use the t-SNE algorithm to reduce the dimension of the wind-solar-water complementary system operation data to obtain low-dimensional feature data.

[0072] The operation data of the wind-solar-water complementary system are input into the high-dimensional space of the t-SNE algorithm, including wind power, photovoltaic power and day-ahead power generation plan; the Gaussian distribution is used to calculate the similarity of the data in the high-dimensional space; the data in the low-dimensional space is initialized with a dimension of 2; the t-distribution with a degree of freedom of 1 is used to calculate the similarity of the data in the low-dimensional space; the position of the data in the low-dimensional space is optimized by minimizing the Kullback-Leibler divergence; and the low-dimensional feature data in the low-dimensional space is output.

[0073] Step 2: Use the DBSCAN algorithm to cluster the low-dimensional feature data to obtain the wind-solar-water complementary system operation scenario data set of different scenario categories.

[0074] (2.1) Determine the DBSCAN neighborhood radius ε as 3.9 and the density threshold MinPts as 4.

[0075] (2.2) Find all core objects. If the number of objects in the neighborhood radius ε of an object is greater than or equal to MinPts, then the object is marked as a core object.

[0076] (2.3) Extract a core object p without any category from all core objects, find all objects that are density-reachable from object p, and form a new cluster.

[0077] (2.4) Determine the final cluster set based on density connectivity, that is, find all objects in objects without any category that can be density-connected with the objects in the newly formed clusters in step (2.3), and add these objects to the newly formed clusters in step (2.3) to form the final cluster set.

[0078] (2.5) Repeat steps (2.3) and (2.4) until all core objects are included in the collection.

[0079] (2.6) Output all clustering clusters and scene categories.

[0080] Step 3: Build a scene recognition model based on the Stacking ensemble learning algorithm, including two layers of recognition models, as follows:

[0081] The first-layer recognition model includes four classifiers, namely: support vector classifier, gradient boosting classifier, multi-layer perceptron classifier and logistic regression classifier. The second-layer recognition model includes one classifier, using support vector classifier.

[0082] Step 4: Use the scene dataset obtained in step 2 to train the scene recognition model.

[0083] The scene data set is divided into a training set and a test set; the four classifiers of the first-layer recognition model are trained using the k-fold cross-validation method; the classification results of the first-layer recognition model are used as input to train the second-layer recognition model; and the trained scene recognition model is saved.

[0084] Step 5: Construct an optimal dispatching model for the wind-solar-water complementary system with the comprehensive optimization objectives of minimizing wind and photovoltaic output reductions, minimizing power shortages, and maximizing reservoir energy storage at the end of the dispatching period.

[0085] The optimal dispatching model of the wind-solar-water complementary system includes the objective function and constraints. The objective function is:

[0086]

[0087] Where c is the penalty coefficient; T is the total dispatch period; M is the total number of hydropower stations; P t rc is the reduction in wind power and photovoltaic power output during period t; P t ps is the power shortage during period t; V m,end V is the water storage capacity of the m hydropower station at the end of the dispatch; m,ini is the initial water storage capacity of the m hydropower station; E m is the set of downstream hydropower stations of hydropower station m; r i is the hydropower conversion rate of hydropower station i.

[0088] The constraints include:

[0089] Hydropower station output constraints:

[0090]

[0091] Hydropower station power flow constraints:

[0092] Q m,min ≤Q m,t ≤Q m,max

[0093] Constraints on outflow from hydropower stations:

[0094]

[0095] Water balance constraints:

[0096]

[0097] Reservoir water storage constraints:

[0098]

[0099] System output constraints:

[0100]

[0101] In the formula, is the output of hydropower station m during period t; k m is the power generation coefficient of the m hydropower station; H m,t is the generating head of the m hydropower station in period t; Q m,t is the power generation flow of hydropower station m in period t; are the water level in front of the reservoir dam and the tailwater level of the hydropower station m at time t respectively; They are the upper and lower limits of the output of hydropower station m during period t; is the maximum climbing limit of the m hydropower station in a single period; Q m,min , Q m,max are the lower and upper limits of the power generation flow of the m hydropower station respectively; is the outflow of hydropower station m in period t; are the lower and upper limits of the outflow of the hydropower station m; S m,t V is the water discharge of hydropower station m during period t; m,t is the reservoir capacity of hydropower station m at time period t; is the inflow of hydropower station m during period t; U m is the collection of hydropower stations directly upstream of m hydropower station; t k is the water flow delay time from the upstream hydropower station to the downstream hydropower station; q m,t V is the natural water inflow of hydropower station m during period t; m,min 、V m,max are the lower and upper limits of the water storage capacity allowed for the m hydropower station; ΔV m represents the maximum change of water storage capacity of hydropower station m during the dispatching period; P t windis the wind power output during period t; P t solar is the photovoltaic output during period t; P t plan Plan the total output of the complementary system during period t.

[0102] Step 6: Construct a real-time scheduling model for the wind-solar-water complementary system based on the multi-task SAC (Soft Actor Critic, SAC) algorithm.

[0103] The objective function and constraints of the wind-solar-water hybrid system optimization scheduling model are converted into a Markov decision process, including states, actions and rewards; the state s at time t t Including wind power output, photovoltaic output, day-ahead power generation plan, natural water inflow of cascade hydropower, water volume of cascade hydropower reservoirs and dispatching period; state a at time t t Including the power generation flow of cascade hydropower stations and the abandoned water flow of cascade hydropower stations; the reward r at time t t It includes the objective function and the penalty term for not meeting the constraints, namely:

[0104] r t =-F t -Q t

[0105] In the formula, F t is the objective function value in period t; Q t is the penalty term for not meeting the constraints in time period t.

[0106] Construct multiple subtask agents. The number of subtask agents is the same as the number of scene categories. Each subtask agent uses 5 neural networks. The construction of the j-th subtask agent is as follows:

[0107] Build an Actor network with the following network parameters: Construct two Critic networks with the following network parameters: Construct two target Critic networks with the following network parameters:

[0108] Step 7: Train the real-time scheduling model of the wind-solar-water hybrid system.

[0109] (7.1) Set the total number of training iterations N, determine the scheduling cycle length T, and determine the number of subtask agents W.

[0110] (7.2) Initialize the parameters of the Actor network, two Critic networks and two target Critic networks, use the full-scenario pre-trained network parameters as the initial network parameters of each subtask learner, and set the current subtask agent j to 1.

[0111] (7.3) Determine whether j is greater than W. If j is greater than W, end the training. Otherwise, initialize the experience replay pool, set j = j + 1, and set the current number of iterations n to 1.

[0112] (7.4) Determine whether n is greater than N. If n is greater than N, return to step (7.3), otherwise initialize state s t , set the current scheduling period t=1, and let n=n+1.

[0113] (7.5) Determine whether t is greater than T. If t is greater than T, return to step (7.4). Otherwise, set state s t Input to the Actor network and get action a t .

[0114] (7.6) Action a t Interact with the environment in scene j and obtain reward r t and the next state s t+1 .

[0115] (7.7) The sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool.

[0116] (7.8) Update the network parameters of the Actor network Update the network parameters of the two Critic networks Update the network parameters of the two target critic networks Update the temperature coefficient, set t=t+1, and return to step (7.5).

[0117] Step 8: Based on the trained scene recognition model and the real-time scheduling model of the wind-solar-water complementary system, the wind-solar-water complementary system is scheduled in real time.

[0118] The real-time operation data is first input into the trained scene recognition model to obtain the task label, and then the corresponding subtask agent is called according to the task label to make scheduling decisions.

[0119] In this embodiment, a clean energy base in the upper reaches of the Yellow River is taken as an example. The base is composed of wind power, photovoltaic power generation and cascade hydropower, including 2000MW of wind power, 4000MW of photovoltaic power and 4160MW of cascade hydropower. The cascade hydropower stations are composed of Cihaxia (2600MW), Banduo (360MW) and Yangqu (1200MW).

[0120] In order to verify the effectiveness of the scheduling method of the present invention, the 600-day operation data is divided into scenarios. The daily operation data consists of wind power output, photovoltaic output and day-ahead power generation plan. Among them, wind power and photovoltaic data are in 15-minute time intervals, and day-ahead power generation plan is in 1-hour time intervals. First, the t-SNE algorithm is used to reduce the dimension of the high-dimensional operation data to obtain a low-dimensional feature representation, and the 216-dimensional operation data is reduced to a 2-dimensional feature space. Then the DBSCAN algorithm is used to cluster the low-dimensional feature data to obtain a typical task scenario. Figure 2 As shown in the figure, the DBSCAN algorithm divides typical scenes into 4 categories, Class-1, Class-2, Class-3, and Class-4 correspond to scenes 1 to 4. From the clustering visualization results, it can be seen that after the dimensionality reduction by the t-SNE algorithm, the high-dimensional operation data is clearly visualized in 2 dimensions. In the 2D feature space, the data of different scenes are more isolated, and the distribution of scene data can be easily found. The trained scene recognition model was tested using 100 days of operation data, and the accuracy of Stacking ensemble learning was 98.302%.

[0121] The real-time dispatching model of the wind-solar-water hybrid system is trained using 600 days of data. The dispatching task is divided into four subtasks according to the clustering results of the task scenarios. Each subtask agent is trained under the corresponding scenario, that is, subtask 1 to subtask 4 are trained under the corresponding scenarios 1 to 4. For each subtask agent, the same hyperparameter settings and the same network structure are used. The Actor network and the Critic network are set to the same network structure. Each neural network contains 2 hidden layers and the number of neurons in the hidden layer is 256. The Tanh activation function is used for the output layer of the Actor network, and the ReLU activation function is used for the remaining network layers. The Adam optimizer is used to perform parameter updates and temperature coefficient adjustments for each network. The capacity of the experience replay pool is 1,000,000, the training batch is 256, the learning rate of each network is 0.0003, the discount rate is 0.98, and the update factor is 0.005.

[0122] Each subtask agent is trained. During the training process, each agent is trained for 16,000 rounds, and each round goes through 96 time steps. The cumulative reward curves during the training of the four subtask agents are as follows: Figure 3 As shown in the figure, as the agent explores more and more, the agent accumulates more experience, the reward value will also increase, and finally stabilize.

[0123] After training, the real-time operation data is first input into the trained scene recognition model to obtain the task label, and then the corresponding subtask agent is called according to the task label to make a scheduling decision. One day of data is selected from the test data in four task scenarios for testing. The scheduling results under the four task scenarios are as follows: Figure 4 As shown in the figure, the dispatch range is 24 hours and the time interval is 15 minutes. As can be seen from the figure, under different scenario tasks, cascade hydropower can compensate for the fluctuation of wind and solar power output and ensure the real-time dispatch of complementary power generation systems to meet the day-ahead power generation plan. Cascade hydropower shows different operating modes in different scenarios, which shows that the multi-task agent learns different dispatch strategies in different scenarios, and differentiated training is very important for it to learn dispatch strategies in different scenarios.

[0124] Changes of cascade hydropower reservoirs under four mission scenarios Figure 5 As shown. Generally, in the early stage of dispatch, the reservoirs of Cihaxia and Yangqu power stations store water, and Banduo power station slowly releases water. In this stage, the planned output of the complementary system is low. Cihaxia and Yangqu power stations operate at a smaller output, and the power generation head is raised by storing water in the reservoir, thereby reducing the water consumption during subsequent power generation. Banduo power station has a high water consumption rate, and it is used to adjust the fluctuation of wind and solar power output in the early stage of dispatch, so the water volume in the reservoir is reduced. In the middle and late stages of dispatch, the intelligent agent in each scenario adjusts the output of the hydropower station according to the power generation plan and wind and solar power output, effectively increasing the water storage capacity of the leading reservoir during the dispatch period and increasing the energy storage of the complementary system.

[0125] SAC, DDPG (Deep Deterministic Policy Gradient, DDPG), MPC (Model Predictive Control, MPC) and PSO (Particle Swarm Optimization, PSO) methods are introduced for comparison. Among them, the PSO method handles uncertainty based on the scenario method. 20 days of data are randomly selected for comparative testing, and the comparison results are shown in Table 1. Compared with the SAC, DDPG, MPC and PSO methods, the method proposed in the present invention improves the average reservoir energy storage by 3.82%, 6.43%, 6.92% and 30.21% respectively, and obtains the maximum reservoir energy storage at the end of the scheduling period. The method proposed in the present invention can effectively obtain the scheduling knowledge of the corresponding scenarios from different operating scenarios and has good uncertainty adaptability. In addition, the calculation time of the MPC method and the PSO method are 10.739s and 86.479s respectively. In contrast, the calculation time of multi-task SAC, SAC, and DDPG are 0.017s, 0.009s, and 0.008s respectively, which are much less than the calculation time of traditional optimization methods. When making online decisions, the method proposed in the present invention can quickly give scheduling results, thereby improving the real-time decision-making ability of the wind-solar-water complementary system.

[0126] Table 1 Comparison results of various methods

[0127]

[0128] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning, characterized in that: The following steps are involved: (1) The t-SNE algorithm is used to reduce the dimension of the wind-solar-water hybrid system operation data to obtain low-dimensional feature data; (2) The DBSCAN algorithm is used to cluster the low-dimensional feature data to obtain the wind-solar-water hybrid system operation scenario data sets of different scenario categories; (3) Building a scene recognition model based on the Stacking ensemble learning algorithm; (4) using the scene data set obtained in step (2) to train a scene recognition model; (5) Taking the minimum reduction in wind and photovoltaic power output, the minimum power shortage, and the maximum reservoir energy storage at the end of the dispatch period as the comprehensive optimization goals, an optimal dispatch model for the wind-solar-water complementary system is constructed; (6) Construct a real-time scheduling model for wind-solar-water complementary system based on multi-task SAC algorithm; (7) Training the real-time dispatch model of wind, solar and water complementary system; (8) Based on the trained scene recognition model and the real-time scheduling model of the wind-solar-water complementary system, the wind-solar-water complementary system is scheduled in real time.

2. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The operation data of the wind-solar-water complementary system in step (1) includes wind power, photovoltaic power and day-ahead power generation plan.

3. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The implementation process of step (1) is as follows: The operation data of the wind-solar-water complementary system is input into the high-dimensional space of the t-SNE algorithm, and the Gaussian distribution is used to calculate the similarity of the data in the high-dimensional space; the data in the low-dimensional space is initialized; the t-distribution with a degree of freedom of 1 is used to calculate the similarity of the data in the low-dimensional space; the position of the data in the low-dimensional space is optimized by minimizing the Kullback-Leibler divergence; and the low-dimensional feature data in the low-dimensional space is output.

4. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The implementation process of step (2) is as follows: (21) Determine the DBSCAN neighborhood radius ε and density threshold MinPts; (22) Find all core objects; (23) Extract a core object p without any category from all core objects, find all objects that are density-reachable from object p, and form a new cluster; (24) Determine the final clustering clusters based on density connectivity; (25) Repeat steps (23) and (24) until all core objects are included in the collection; (26) Output all clustering clusters and scene categories.

5. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The scene recognition model described in step (3) includes two layers of recognition models, which are as follows: The first-level recognition model includes four classifiers: support vector classifier, gradient boosting classifier, multi-layer perceptron classifier and logistic regression classifier; The second-layer recognition model includes a classifier, which is a support vector classifier.

6. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The implementation process of step (4) is as follows: The scene data set is divided into a training set and a test set; the four classifiers of the first-layer recognition model are trained using the k-fold cross-validation method; the classification results of the first-layer recognition model are used as input to train the second-layer recognition model; and the trained scene recognition model is saved.

7. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The objective function of the optimization scheduling model of the wind-solar-water complementary system described in step (5) is: In the formula, c is the penalty coefficient; T is the total scheduling period; M is the total number of hydropower stations; P t rc is the reduction in wind power and photovoltaic power output during period t; P t ps is the power shortage during period t; V m,end V is the water storage capacity of the m hydropower station at the end of the dispatch; m,ini is the initial water storage capacity of the m hydropower station; E m is the set of downstream hydropower stations of hydropower station m; r i is the hydropower conversion rate of hydropower station i.

8. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The constraints of the wind-solar-water hybrid system optimization scheduling model described in step (5) include: Hydropower station output constraints: Hydropower station power flow constraints: Q m,min ≤Q m,t ≤Q m,max Constraints on outflow from hydropower stations: Water balance constraints: Reservoir water storage constraints: System output constraints: In the formula, is the output of hydropower station m during period t; k m is the power generation coefficient of the m hydropower station; H m,t is the generating head of the m hydropower station in period t; Q m,t is the power generation flow of hydropower station m in period t; are the water level in front of the reservoir dam and the tailwater level of the hydropower station m at time t respectively; They are the upper and lower limits of the output of hydropower station m during period t; is the maximum climbing limit of the m hydropower station in a single period; Q m,min , Q m,max are the lower and upper limits of the power generation flow of the m hydropower station respectively; is the outflow of hydropower station m in period t; are the lower and upper limits of the outflow of the hydropower station m; S m,t V is the water discharge of hydropower station m during period t; m,t is the reservoir capacity of hydropower station m at time period t; is the inflow of hydropower station m during period t; U m is the collection of hydropower stations directly upstream of m hydropower station; t k is the water flow delay time from the upstream hydropower station to the downstream hydropower station; q m,t V is the natural water inflow of hydropower station m during period t; m,min 、V m,max are the lower and upper limits of the water storage capacity allowed for the m hydropower station; ΔV m represents the maximum change of water storage capacity of hydropower station m during the dispatching period; P t wind is the wind power output during period t; P t solar is the photovoltaic output during period t; P t plan Plan the total output of the complementary system during period t.

9. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The implementation process of step (6) is as follows: The objective function and constraints of the wind-solar-water hybrid system optimization scheduling model are converted into a Markov decision process, including states, actions and rewards; the state s at time t t Including wind power output, photovoltaic output, day-ahead power generation plan, natural water inflow of cascade hydropower, water volume of cascade hydropower reservoirs and dispatching period; state a at time t t Including the power generation flow of cascade hydropower stations and the abandoned water flow of cascade hydropower stations; the reward r at time t t It includes the objective function and the penalty term for not meeting the constraints, namely: r t =-F t -Q t In the formula, F t is the objective function value in period t; Q t is the penalty term for not meeting the constraint condition during time period t; Construct multiple subtask agents. The number of subtask agents is the same as the number of scene categories. Each subtask agent uses 5 neural networks. The construction of the j-th subtask agent is as follows: Build an Actor network with the following network parameters: Construct two Critic networks with the following network parameters: Construct two target Critic networks with the following network parameters:

10. The real-time scheduling method for wind-solar-water complementary system based on multi-task deep reinforcement learning according to claim 1 is characterized in that: The implementation process of step (7) is as follows: (71) Set the total number of training iterations N, determine the scheduling cycle length T, and determine the number of subtask agents W; (72) Initialize the parameters of the Actor network, two Critic networks, and two target Critic networks, use the full-scenario pre-trained network parameters as the initial network parameters of each subtask learner, and set the current subtask agent j to 1; (73) Determine whether j is greater than W. If j is greater than W, end the training. Otherwise, initialize the experience replay pool, set j = j + 1, and set the current number of iterations n to 1; (74) Determine whether n is greater than N. If n is greater than N, return to step (73), otherwise initialize state s t , set the current scheduling period t = 1, and let n = n + 1; (75) Determine whether t is greater than T. If t is greater than T, return to step (74). Otherwise, set state s t Input to the Actor network and get action a t ; (76) Action a t Interact with the environment in scene j and obtain reward r t and the next state s t+1 ; (77) The sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool; (78) Update the network parameters of the Actor network Update the network parameters of the two Critic networks Update the network parameters of the two target critic networks Update the temperature coefficient, set t=t+1, and return to step (75).

Images